You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "suxiaogang223 (via GitHub)" <gi...@apache.org> on 2026/10/09 07:52:39 UTC

[PR] [feature](paimon) Support Paimon Rust reader on master [doris]

suxiaogang223 opened a new pull request, #68825:
URL: https://github.com/apache/doris/pull/68825

   ### What problem does this PR solve?
   
   Issue Number: None
   
   Related PR: #66227
   
   Problem Summary:
   
   Forward-port the Paimon Rust reader from branch-4.1 to master. The reader consumes FE-planned Paimon splits in BE through the Arrow C Data Interface and is guarded by the `enable_paimon_rust_reader` session variable, which remains disabled by default.
   
   The forward-port is adapted to the master connector framework instead of copying the branch-4.1 FE scan path. FE selects the Rust reader through the Paimon connector scan provider and checks BE capability advertised by heartbeat. Unsupported table layouts, scan modes, credentials, schemas, projections, predicates, or runtime filters conservatively fall back to the existing JNI reader.
   
   Master-specific compatibility fixes include:
   
   - preserve null-aware predicate and runtime-filter semantics;
   - reject unsupported credential combinations without changing effective catalog credentials in tests;
   - follow master branch and snapshot resolution behavior;
   - carry name-based nested projection state even when a connector has no stable nested field IDs, preventing projected Arrow structs from being decoded as complete Doris structs;
   - explicitly enable TIMESTAMP_LTZ mapping in the time-zone regression fixture.
   
   ### Release note
   
   Add the experimental `enable_paimon_rust_reader` session variable. When enabled and supported by the selected BE and table scan, Doris can read Paimon data through the BE-side Rust reader. Unsupported scans automatically use the existing JNI reader.
   
   ### Check List (For Author)
   
   - Test:
       - FE build and checkstyle
       - FE unit tests: `PaimonRustReaderSelectorTest`, `PluginDrivenScanNodeProjectionTest`
       - Regression tests: `test_paimon_rust_reader_cast_predicates`, `test_paimon_rust_reader_schema_evolution`, `test_paimon_rust_reader_compatibility`, `test_paimon_rust_reader_eq_for_null`
   - Behavior changed: Yes. The new reader can be enabled explicitly; the default remains unchanged.
   - Does this need documentation: Yes. Session-variable documentation should be added with the feature rollout.
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label if needed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6162507196

   I compared this PR at `13e491640aa` with branch-4.1 #66227 while working on bringing branch-4.x changes back to master, mostly by reading the code. The CI results below are from this head; when I wrote this, BE UT had passed, External had two failures and was still running, and FE UT and NonConcurrent were still running. The BE reader, the predicate converter and the BE unit tests match branch-4.1 closely, and every FE gate of `PaimonScanNode` has a counterpart in `PaimonRustReaderSelector` / `PaimonRustReaderCapabilities`. 13e491640aa also fixed the sliced Arrow batches: the row range is now read from the original batch. The points below are where it still differs.
   
   **1. The Performance build does not compile.** GCC stops at `paimon_rust_table_reader.cpp:142` with `-Werror=subobject-linkage`: the members of `PaimonRustTableReader::PaimonHandles` use `table_ptr`, `read_builder_ptr` and the other aliases from the anonymous namespace. GCC does not report this for the main source file, but master compiles `Format` as a unity build, so the file is compiled inside `unity_7_cxx.cxx`. Performance is the only GCC build; it failed the same way at 19a6bab262e. Moving the deleters and aliases into a named namespace should fix it.
   
   **2. With the Rust reader on, COUNT(col), MIN and MAX on append tables no longer use footer aggregates (13e491640aa).** Compatible splits now go to the Rust reader before the native check (`PaimonScanPlanProvider` L991-1002). branch-4.1 keeps raw-file splits on the native reader on purpose: "only the native file reader can use footer aggregates for COUNT(col) and MIN/MAX" (`PaimonScanNode` L1238-1241). Its `test_paimon_rust_reader_compatibility` runs the footer-aggregate queries with the Rust reader both on and off and asserts the native reader (L113-138). Here that block now runs only with the Rust reader off (L119-123), and the forced case that checked that the Rust reader returns real rows to the upper aggregate expects JNI (L147-148), so neither side is covered any more. Either keep raw-file splits ahead of the Rust reader as branch-4.1 does, or at least do so when the scan pushes down an aggregate; in both cases a Rust-enabled run of that block would catch a regression. Also,
  Rust ranges are now planned before the `ignore_split_type` checks, so `IGNORE_JNI` no longer drops them as it does on branch-4.1.
   
   **3. `test_paimon_rust_reader_eq_for_null` fails because the DDL mapping is not ported.** It fails at L222-223: expected `[[1, abc, 1.23], [2, xyz, 2.35]]`, but got `[[1, abcdef, 1.23], [2, xyzuvw, 2.35]]`. The Javadoc of `PaimonTypeMapping.toPaimonType` says CHAR/VARCHAR bounds and timestamp precision are preserved, but the code still maps CHAR(n) and VARCHAR(n) to VARCHAR(2147483647) and DATETIME to TIMESTAMP(6), and `PaimonTypeMappingToPaimonTest` still asserts that. So `MODIFY COLUMN v VARCHAR(3)` (L208) widens the column and Doris reads it back as STRING, which is also why `truncate_char_or_varchar_columns` (L220) does not help. I opened #69045 with the branch-4.1 mapping and its tests. Its Javadoc is the same text as yours, so the two PRs do not conflict. (The profile changes of #66227 are in #69044.)
   
   **4. `test_paimon_rust_reader_compatibility` fails on the new routing.** The check before the ALTER at L156 expects no Rust reader, but `string_to_temporal_0` is an append table, which now goes to the Rust reader; branch-4.1 expects the Rust reader there too (L149). The suite stops at this failure, so the checks after it have not run. They may fail next: their baseline is now the native reader (`force_jni_scanner=false` at L47; branch-4.1 sets it to true, so its baseline is JNI), while the expected values are Java cast results (`1970-02-12`, `1970-01-01 00:00:00.042`, epoch seconds, and `127` for BIGINT to TINYINT).
   
   **5. The session variable is never fuzzed.** `enable_paimon_rust_reader` has `fuzzy = true`, but `setFuzzyForCatalog` does not randomize it; branch-4.1 sets `this.enablePaimonRustReader = random.nextBoolean()` there. On master the annotation only affects printing, so CI never runs the Paimon suites with the Rust reader.
   
   **6. The selector still rejects some scans that branch-4.1 reads with the Rust reader.** These only cost performance, but they would be worth stating in the description:
   - `deletion-vectors.merge-on-read=true` sends every table to JNI. branch-4.1 only does that for partial-update and aggregation tables with deletion vectors, and its tests expect deduplicate + merge-on-read to use the Rust reader.
   - The COUNT(*) range is always JNI.
   - Any access path deeper than one level falls back.
   - A TIMESTAMP(7..9) column anywhere in the current schema sends the whole table to JNI, even when it is not projected.
   
   **7. FE tests.** branch-4.1 adds 73 FE test methods, and about 50 have no counterpart here. In particular, nothing covers the 45 schema-evolution, aggregation and primary-key cases of `PaimonRustReaderCapabilities`. The class is nearly identical to branch-4.1, so `PaimonRustReaderCompatibilityTest`, `PaimonRustReaderEvolutionTest`, `PaimonRustReaderLegacySchemaTest` and the capability cases of `PaimonScanNodeTest` should port with small changes. The Rust range payload (schema JSON without time-travel selectors, relation options, branch) has no FE test either.
   
   One change goes the other way: the `EQ_FOR_NULL` change in `paimon_rust_predicate_converter.cpp` is a real fix. branch-4.1 pushes `IS NULL` for `col <=> non-null literal` and drops matching rows when `NULL_SAFE_EQUAL_TO_EQUAL` is disabled. `EqForNullLiteralIsPushed` only checks that some predicate is pushed, so it also passes with the branch-4.1 code; a test that reads rows would pin the fix.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6096981820

   # BE Regression && UT Coverage Report
   Increment line coverage `100.00% (3/3)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_701da299bb5dc73be36da3cf7e3346905c1250bd_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_701da299bb5dc73be36da3cf7e3346905c1250bd_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 76.71% (35268/45974) |
   | Line Coverage     | 61.89% (398778/644284) |
   | Region Coverage   | 58.36% (336873/577270) |
   | Branch Coverage   | 59.19% (154956/261811) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6076818232

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6080873304

   # BE Regression && UT Coverage Report
   Increment line coverage `100.00% (3/3)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_5bfd0302d26a643cd19a6f418a4186372777b81d_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_5bfd0302d26a643cd19a6f418a4186372777b81d_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 76.68% (35246/45968) |
   | Line Coverage     | 61.87% (398605/644233) |
   | Region Coverage   | 58.30% (336492/577204) |
   | Branch Coverage   | 59.15% (154847/261787) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6189917597

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6187510399

   # BE Regression && UT Coverage Report
   Increment line coverage `100.00% (3/3)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 77.66% (35940/46279) |
   | Line Coverage     | 63.12% (409161/648220) |
   | Region Coverage   | 60.15% (349467/580988) |
   | Branch Coverage   | 60.57% (159782/263812) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6166489204

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 24.73 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 8d52953c3037b078816cdc14d5695e371b5cfacf, data reload: false
   
   query1	0.00	0.00	0.01
   query2	0.10	0.05	0.05
   query3	0.27	0.12	0.11
   query4	1.61	0.12	0.12
   query5	0.25	0.25	0.24
   query6	1.16	0.69	0.69
   query7	0.04	0.01	0.00
   query8	0.06	0.04	0.04
   query9	0.41	0.36	0.36
   query10	0.59	0.61	0.58
   query11	0.21	0.14	0.13
   query12	0.19	0.15	0.15
   query13	0.48	0.50	0.49
   query14	1.02	1.02	1.02
   query15	0.64	0.61	0.63
   query16	0.34	0.34	0.38
   query17	1.13	1.12	1.14
   query18	0.24	0.25	0.22
   query19	2.05	1.99	2.02
   query20	0.02	0.02	0.01
   query21	15.49	0.27	0.14
   query22	4.40	0.06	0.05
   query23	16.24	0.32	0.12
   query24	3.21	0.73	0.41
   query25	0.10	0.08	0.05
   query26	1.55	0.25	0.15
   query27	0.05	0.04	0.03
   query28	1.96	0.60	0.42
   query29	12.47	4.22	3.35
   query30	0.30	0.14	0.13
   query31	2.77	0.57	0.34
   query32	3.23	0.61	0.52
   query33	3.16	3.25	3.25
   query34	15.64	4.01	3.40
   query35	3.37	3.32	3.34
   query36	0.57	0.43	0.43
   query37	0.10	0.07	0.06
   query38	0.05	0.04	0.04
   query39	0.04	0.03	0.03
   query40	0.17	0.14	0.13
   query41	0.09	0.04	0.03
   query42	0.04	0.03	0.03
   query43	0.04	0.03	0.04
   Total cold run time: 95.85 s
   Total hot run time: 24.73 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6182963352

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6169982135

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6166236500

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29742 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 8d52953c3037b078816cdc14d5695e371b5cfacf, data reload: false
   
   ------ Round 1 ----------------------------------
   ============================================
   q1	17585	5648	5558	5558
   q2	2146	314	265	265
   q3	10199	1435	859	859
   q4	4680	475	347	347
   q5	7443	800	547	547
   q6	191	189	153	153
   q7	775	804	625	625
   q8	9257	1391	1115	1115
   q9	5536	4287	4277	4277
   q10	6832	1303	1006	1006
   q11	431	260	242	242
   q12	638	402	290	290
   q13	18098	3654	3041	3041
   q14	284	260	250	250
   q15	q16	752	725	657	657
   q17	1208	761	577	577
   q18	7173	6249	6177	6177
   q19	1212	964	645	645
   q20	375	338	221	221
   q21	6315	2582	2646	2582
   q22	414	368	308	308
   Total cold run time: 101544 ms
   Total hot run time: 29742 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   ============================================
   q1	6724	6658	6753	6658
   q2	688	602	575	575
   q3	5542	5794	5320	5320
   q4	2065	2206	1547	1547
   q5	5831	5540	5879	5540
   q6	241	184	144	144
   q7	2278	1945	1816	1816
   q8	3212	2799	2790	2790
   q9	8135	8055	7816	7816
   q10	3626	3614	3248	3248
   q11	599	401	378	378
   q12	664	691	503	503
   q13	3283	3658	3010	3010
   q14	286	285	257	257
   q15	q16	651	685	617	617
   q17	7899	6999	6864	6864
   q18	13101	12245	13190	12245
   q19	926	809	789	789
   q20	2180	2159	1917	1917
   q21	5816	4735	5010	4735
   q22	508	448	402	402
   Total cold run time: 74255 ms
   Total hot run time: 67171 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6183560696

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29548 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit b3e30fa9240264b19d185ee444c6531d4106a8ff, data reload: false
   
   ------ Round 1 ----------------------------------
   ============================================
   q1	17630	5688	5635	5635
   q2	2098	311	252	252
   q3	10262	1409	868	868
   q4	4678	468	355	355
   q5	7432	807	556	556
   q6	198	195	158	158
   q7	783	817	626	626
   q8	9277	1355	1198	1198
   q9	5659	4267	4307	4267
   q10	6823	1297	1009	1009
   q11	439	260	236	236
   q12	633	404	283	283
   q13	18104	3557	3035	3035
   q14	281	278	254	254
   q15	q16	729	729	679	679
   q17	1205	757	582	582
   q18	7096	6246	6197	6197
   q19	1115	962	607	607
   q20	400	352	224	224
   q21	5418	2256	2437	2256
   q22	370	315	271	271
   Total cold run time: 100630 ms
   Total hot run time: 29548 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   ============================================
   q1	6186	6121	6154	6121
   q2	672	590	529	529
   q3	5216	5381	4953	4953
   q4	2003	2130	1556	1556
   q5	5266	5467	5763	5467
   q6	266	194	160	160
   q7	2265	2094	1875	1875
   q8	3255	2849	2786	2786
   q9	8096	7947	8250	7947
   q10	3850	3696	3360	3360
   q11	621	452	424	424
   q12	720	725	516	516
   q13	3533	3921	3257	3257
   q14	312	303	291	291
   q15	q16	696	711	657	657
   q17	8412	7505	6936	6936
   q18	13196	12247	13078	12247
   q19	930	812	803	803
   q20	2201	2175	1943	1943
   q21	5912	4802	4868	4802
   q22	501	492	408	408
   Total cold run time: 74109 ms
   Total hot run time: 67038 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6081574159

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6184154264

   # FE UT Coverage Report
   Increment line coverage `  75.86% (22/29)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6163261344

   # BE Regression && UT Coverage Report
   Increment line coverage `100.00% (3/3)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_13e491640aa8d4bb03224c0190f1c40ec26b9db1_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_13e491640aa8d4bb03224c0190f1c40ec26b9db1_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 76.98% (35573/46208) |
   | Line Coverage     | 62.25% (402988/647392) |
   | Region Coverage   | 58.71% (340612/580130) |
   | Branch Coverage   | 59.56% (156854/263346) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6186486279

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6077634195

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6080541300

   # FE UT Coverage Report
   Increment line coverage `  81.48% (22/27)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_5bfd0302d26a643cd19a6f418a4186372777b81d/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_5bfd0302d26a643cd19a6f418a4186372777b81d/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6095464494

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6163492087

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6166403825

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 153018 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 8d52953c3037b078816cdc14d5695e371b5cfacf, data reload: false
   
   query5	4309	610	468	468
   query6	422	202	184	184
   query7	4812	439	225	225
   query8	319	177	173	173
   query9	8763	4109	4085	4085
   query10	460	312	265	265
   query11	5954	3755	3530	3530
   query12	151	92	86	86
   query13	1276	459	343	343
   query14	6157	4024	3775	3775
   query14_1	3583	3586	3556	3556
   query15	208	196	181	181
   query16	947	432	420	420
   query17	849	658	539	539
   query18	2419	425	334	334
   query19	194	171	137	137
   query20	87	86	85	85
   query21	210	138	112	112
   query22	13023	12970	12887	12887
   query23	13481	12918	12626	12626
   query23_1	12590	12551	12549	12549
   query24	7107	843	486	486
   query24_1	484	492	474	474
   query25	539	421	365	365
   query26	1271	260	140	140
   query27	2758	437	270	270
   query28	4631	1988	1977	1977
   query29	1569	594	465	465
   query30	307	217	189	189
   query31	998	927	806	806
   query32	137	95	94	94
   query33	537	308	249	249
   query34	918	845	497	497
   query35	746	752	644	644
   query36	798	824	746	746
   query37	137	100	92	92
   query38	1782	1782	1701	1701
   query39	786	801	740	740
   query39_1	756	725	755	725
   query40	221	113	97	97
   query41	74	68	71	68
   query42	89	86	87	86
   query43	361	372	324	324
   query44	1286	711	703	703
   query45	181	184	172	172
   query46	875	1008	569	569
   query47	2807	3008	2931	2931
   query48	301	305	216	216
   query49	576	393	294	294
   query50	669	275	212	212
   query51	10505	10712	10540	10540
   query52	75	82	65	65
   query53	184	208	150	150
   query54	240	189	179	179
   query55	70	69	61	61
   query56	223	257	223	223
   query57	1614	1564	1552	1552
   query58	284	266	256	256
   query59	2058	2103	1930	1930
   query60	274	241	225	225
   query61	154	144	157	144
   query62	389	338	286	286
   query63	192	161	169	161
   query64	2730	927	777	777
   query65	3298	3227	3229	3227
   query66	1786	408	311	311
   query67	20164	20157	20082	20082
   query68	3275	986	616	616
   query69	406	282	244	244
   query70	839	803	792	792
   query71	306	228	204	204
   query72	2793	2608	2273	2273
   query73	530	524	295	295
   query74	4614	4470	4325	4325
   query75	2314	2283	1961	1961
   query76	2382	1003	583	583
   query77	330	368	270	270
   query78	9740	9681	9175	9175
   query79	956	843	524	524
   query80	1214	414	358	358
   query81	584	321	273	273
   query82	611	131	105	105
   query83	294	193	180	180
   query84	310	116	101	101
   query85	847	410	374	374
   query86	390	253	235	235
   query87	2026	1960	1833	1833
   query88	3643	2728	2725	2725
   query89	347	301	256	256
   query90	1876	185	178	178
   query91	164	147	126	126
   query92	98	88	90	88
   query93	1025	1016	590	590
   query94	636	330	296	296
   query95	574	391	317	317
   query96	671	519	231	231
   query97	2469	2430	2352	2352
   query98	164	152	145	145
   query99	786	755	643	643
   Total cold run time: 235153 ms
   Total hot run time: 153018 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6175021011

   Thanks for the update. I re-read 446fd368cc1 against my earlier comment: points 1, 2, 4 and 5 are addressed, and most of the 4.x FE tests are now ported.
   
   What remains:
   
   - **Merge conflict with master.** #69045 has merged the same CHAR/VARCHAR length and DATETIME precision mapping (e771daa9b01), so `PaimonTypeMapping.java` and `PaimonTypeMappingToPaimonTest.java` conflict. Every TeamCity job on 446fd368cc1 stopped at `merge refs/pull/68825/head into master failed`, before compiling anything. Rebasing onto master and dropping this branch's copy of the mapping should unblock them.
   - **Routing.** With an aggregate pushed down a split now stays native, as in 4.x. Without one, a split that can be read natively still goes to Rust first, while 4.x lets Rust replace only JNI.
   - **Description.** The fallbacks that differ from 4.x (COUNT(*) ranges, access paths deeper than one level, TIMESTAMP(7..9)) are explained in the selector's Javadoc but not in the PR description; listing them there, with the routing difference above, would help reviewers.
   - **Tests still without a counterpart** (by name, against 4.x): `testPrimaryKeyMergeEnginesWithOneInsertOnlyFile`, the query-auth, REST token and environment credential provider cases, the mixed-format ORC LTZ case, and the primary-key deletion-vector level-0 / delete matrix.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6169990569

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6163410656

   # FE Regression Coverage Report
   Increment line coverage `  12.08% (18/149)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_13e491640aa8d4bb03224c0190f1c40ec26b9db1_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_13e491640aa8d4bb03224c0190f1c40ec26b9db1_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6187658819

   # BE Regression && UT Coverage Report
   Increment line coverage `100.00% (3/3)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 77.66% (35940/46279) |
   | Line Coverage     | 63.12% (409161/648220) |
   | Region Coverage   | 60.15% (349467/580988) |
   | Branch Coverage   | 60.57% (159782/263812) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6143849699

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6157792705

   # FE Regression Coverage Report
   Increment line coverage `  66.67% (18/27)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_19a6bab262e26ece93dc9bc5510af5c4827aa6fd_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_19a6bab262e26ece93dc9bc5510af5c4827aa6fd_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6185576017

   # BE Regression && UT Coverage Report
   Increment line coverage `100.00% (3/3)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 77.66% (35940/46279) |
   | Line Coverage     | 63.12% (409161/648220) |
   | Region Coverage   | 60.15% (349467/580988) |
   | Branch Coverage   | 60.57% (159782/263812) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 merged PR #68825:
URL: https://github.com/apache/doris/pull/68825


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6187885515

   # BE Regression && UT Coverage Report
   Increment line coverage `100.00% (3/3)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 76.97% (35623/46279) |
   | Line Coverage     | 62.25% (403501/648220) |
   | Region Coverage   | 58.64% (340693/580988) |
   | Branch Coverage   | 59.56% (157121/263812) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6183674250

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 152887 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit b3e30fa9240264b19d185ee444c6531d4106a8ff, data reload: false
   
   query5	4316	598	460	460
   query6	441	205	195	195
   query7	4797	454	231	231
   query8	324	168	170	168
   query9	8737	4046	4033	4033
   query10	464	320	266	266
   query11	6057	3835	3529	3529
   query12	150	105	89	89
   query13	1270	467	325	325
   query14	6094	4029	3731	3731
   query14_1	3549	3558	3587	3558
   query15	218	207	182	182
   query16	945	448	405	405
   query17	856	658	539	539
   query18	2408	430	311	311
   query19	196	168	139	139
   query20	87	82	82	82
   query21	212	138	124	124
   query22	12946	12988	12878	12878
   query23	13608	12917	12614	12614
   query23_1	12617	12761	12673	12673
   query24	7082	840	471	471
   query24_1	470	471	479	471
   query25	537	422	362	362
   query26	1258	258	144	144
   query27	2760	437	273	273
   query28	4607	1962	1946	1946
   query29	1579	603	467	467
   query30	310	228	186	186
   query31	1021	930	829	829
   query32	140	97	96	96
   query33	509	310	251	251
   query34	894	857	504	504
   query35	756	741	658	658
   query36	834	825	716	716
   query37	144	103	96	96
   query38	1806	1751	1721	1721
   query39	789	802	753	753
   query39_1	751	737	741	737
   query40	221	120	99	99
   query41	78	69	69	69
   query42	88	85	85	85
   query43	364	382	326	326
   query44	1304	702	712	702
   query45	185	181	168	168
   query46	846	964	570	570
   query47	2921	3061	2908	2908
   query48	300	297	216	216
   query49	565	403	297	297
   query50	692	274	215	215
   query51	10438	10553	10493	10493
   query52	77	80	68	68
   query53	184	208	156	156
   query54	237	211	175	175
   query55	71	64	60	60
   query56	241	208	214	208
   query57	1616	1595	1556	1556
   query58	290	269	252	252
   query59	2113	2079	1906	1906
   query60	294	245	220	220
   query61	148	140	148	140
   query62	396	345	293	293
   query63	215	161	161	161
   query64	2778	953	811	811
   query65	3292	3263	3249	3249
   query66	1789	413	311	311
   query67	20300	20137	20196	20137
   query68	2902	999	604	604
   query69	385	288	255	255
   query70	881	768	771	768
   query71	300	239	206	206
   query72	2729	2602	2274	2274
   query73	510	528	296	296
   query74	4606	4442	4309	4309
   query75	2357	2294	1941	1941
   query76	2303	1009	598	598
   query77	337	393	303	303
   query78	9695	9671	9102	9102
   query79	989	891	510	510
   query80	547	432	346	346
   query81	544	317	271	271
   query82	572	136	108	108
   query83	303	205	182	182
   query84	322	118	103	103
   query85	823	423	379	379
   query86	325	238	228	228
   query87	2012	1943	1822	1822
   query88	3608	2712	2693	2693
   query89	342	293	259	259
   query90	1895	184	178	178
   query91	167	149	128	128
   query92	103	105	91	91
   query93	1014	1003	579	579
   query94	465	309	288	288
   query95	593	395	318	318
   query96	658	520	235	235
   query97	2464	2425	2359	2359
   query98	164	159	144	144
   query99	781	774	652	652
   Total cold run time: 234011 ms
   Total hot run time: 152887 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6159981959

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6157575732

   # FE UT Coverage Report
   Increment line coverage `  81.48% (22/27)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_19a6bab262e26ece93dc9bc5510af5c4827aa6fd/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_19a6bab262e26ece93dc9bc5510af5c4827aa6fd/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6168629502

   # BE Regression && UT Coverage Report
   Increment line coverage `100.00% (3/3)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_8d52953c3037b078816cdc14d5695e371b5cfacf_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_8d52953c3037b078816cdc14d5695e371b5cfacf_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 76.88% (35571/46271) |
   | Line Coverage     | 62.16% (402851/648106) |
   | Region Coverage   | 58.41% (339323/580884) |
   | Branch Coverage   | 59.43% (156770/263769) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6183731801

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 24.62 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit b3e30fa9240264b19d185ee444c6531d4106a8ff, data reload: false
   
   query1	0.01	0.01	0.01
   query2	0.09	0.05	0.05
   query3	0.27	0.12	0.12
   query4	1.61	0.13	0.12
   query5	0.28	0.25	0.22
   query6	1.18	0.67	0.68
   query7	0.04	0.01	0.01
   query8	0.05	0.04	0.04
   query9	0.40	0.37	0.33
   query10	0.57	0.60	0.58
   query11	0.21	0.14	0.14
   query12	0.18	0.14	0.14
   query13	0.49	0.50	0.50
   query14	1.02	1.02	1.01
   query15	0.63	0.63	0.61
   query16	0.33	0.34	0.35
   query17	1.14	1.16	1.08
   query18	0.22	0.21	0.21
   query19	2.16	2.03	1.99
   query20	0.02	0.02	0.01
   query21	15.50	0.27	0.14
   query22	4.55	0.06	0.05
   query23	16.27	0.32	0.13
   query24	2.93	0.62	0.41
   query25	0.11	0.07	0.05
   query26	1.12	0.25	0.14
   query27	0.04	0.03	0.03
   query28	1.61	0.62	0.41
   query29	12.54	4.22	3.33
   query30	0.29	0.14	0.13
   query31	2.77	0.57	0.34
   query32	3.22	0.60	0.52
   query33	3.26	3.21	3.25
   query34	15.46	4.03	3.42
   query35	3.35	3.36	3.35
   query36	0.56	0.44	0.42
   query37	0.10	0.07	0.07
   query38	0.05	0.04	0.03
   query39	0.04	0.03	0.03
   query40	0.17	0.14	0.15
   query41	0.08	0.03	0.03
   query42	0.04	0.03	0.03
   query43	0.04	0.04	0.03
   Total cold run time: 95 s
   Total hot run time: 24.62 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68825:
URL: https://github.com/apache/doris/pull/68825#discussion_r4280224754


##########
be/src/format_v2/table/paimon_rust_predicate_converter.cpp:
##########
@@ -0,0 +1,837 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+
+#include <algorithm>
+#include <cctype>
+#include <limits>
+#include <memory>
+#include <utility>
+
+#include "common/logging.h"
+#include "core/column/column_const.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "core/types.h"
+#include "core/value/decimalv2_value.h"
+#include "core/value/timestamptz_value.h"
+#include "core/value/vdatetime_value.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vcompound_pred.h"
+#include "exprs/vdirect_in_predicate.h"
+#include "exprs/vectorized_fn_call.h"
+#include "exprs/vexpr.h"
+#include "exprs/vin_predicate.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+
+namespace doris {
+
+namespace {
+// paimon_datum tags (see paimon.h / bindings/c/src/table.rs::datum_from_c).
+constexpr int32_t kTagBool = 0;
+constexpr int32_t kTagTinyInt = 1;
+constexpr int32_t kTagSmallInt = 2;
+constexpr int32_t kTagInt = 3;
+constexpr int32_t kTagLong = 4;
+constexpr int32_t kTagDouble = 6;
+constexpr int32_t kTagString = 7;
+constexpr int32_t kTagDate = 8;
+constexpr int32_t kTagTimestamp = 10;
+constexpr int32_t kTagDecimal = 12;
+constexpr int32_t kTagBytes = 13;
+
+// paimon decimal precision ceiling (paimon::Decimal::MAX_PRECISION).
+constexpr int32_t kPaimonDecimalMaxPrecision = 38;
+
+// RAII for an owned paimon_predicate*. and/or/not consume their inputs, so we
+// release() before handing pointers to them.
+struct predicate_deleter {
+    void operator()(paimon_predicate* p) const {
+        if (p) {
+            paimon_predicate_free(p);
+        }
+    }
+};
+using predicate_ptr = std::unique_ptr<paimon_predicate, predicate_deleter>;
+
+// RAII for an owned paimon_error*.
+struct error_deleter {
+    void operator()(paimon_error* p) const {
+        if (p) {
+            paimon_error_free(p);
+        }
+    }
+};
+using error_ptr = std::unique_ptr<paimon_error, error_deleter>;
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_predicate_error(paimon_error* err) {
+    error_ptr owned(err);
+    if (!owned) {
+        return "unknown error";
+    }
+    std::string msg;
+    if (owned->message.data != nullptr && owned->message.len > 0) {
+        msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+    }
+    return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+} // namespace
+
+PaimonRustPredicateConverter::PaimonRustPredicateConverter(
+        const std::vector<std::string>& column_names, const std::vector<DataTypePtr>& column_types,
+        const paimon_table* table)
+        : _table(table) {
+    DORIS_CHECK(column_names.size() == column_types.size());
+    _columns_by_name.reserve(column_names.size());
+    for (size_t i = 0; i < column_names.size(); ++i) {
+        _columns_by_name.emplace(_normalize_name(column_names[i]),
+                                 std::make_pair(column_names[i], column_types[i]));
+    }
+    // Paimon TIMESTAMP (wall clock) is stored as epoch-millis-of-the-wall-time
+    // and the DateTimeV2 serde decodes timezone-naive arrow values in UTC, so
+    // timestamp literals convert wall->epoch in UTC. utc_time_zone() needs no
+    // tzdata lookup, so the conversion cannot silently fall back to a
+    // machine-local zone.
+    _utc_tz = cctz::utc_time_zone();
+}
+
+paimon_predicate* PaimonRustPredicateConverter::build(const VExprContextSPtrs& conjuncts) {
+    _converted_conjuncts = 0;
+    _converted_runtime_filters = 0;
+    if (_table == nullptr) {
+        return nullptr;
+    }
+    predicate_ptr result;
+    for (const auto& conjunct : conjuncts) {
+        if (!conjunct || !conjunct->root()) {
+            continue;
+        }
+        auto root = conjunct->root();
+        const bool is_runtime_filter = root->is_rf_wrapper();
+        if (root->is_rf_wrapper()) {
+            if (auto impl = root->get_impl()) {
+                // A null-aware runtime filter (an EQ_FOR_NULL join) must stay
+                // residual: its wrapper execution restores NULL probe rows to
+                // true (RuntimeFilterExpr::change_null_to_true), while the
+                // unwrapped impl — rebuilt as an ordinary IN predicate through
+                // VDirectInPredicate::get_slot_in_expr — treats NULL as
+                // not-in-set and would prune the NULL probes permanently
+                // before the join sees them. Keep the wrapper itself: it fails
+                // every dispatch below, so the conjunct stays in the residual,
+                // and it is safe to execute on selected rows, so later
+                // conjuncts keep pushing. The lance pushdown declines
+                // is_null_aware() filters for the same reason.
+                // is_null_aware() is concrete on RuntimeFilterExpr (the only
+                // class whose is_rf_wrapper() is true), so the dynamic_cast
+                // never fails in practice; the null guard keeps the unwrap for
+                // any future wrapper shape.
+                auto* rf_wrapper = dynamic_cast<RuntimeFilterExpr*>(root.get());
+                if (rf_wrapper == nullptr || !rf_wrapper->is_null_aware()) {
+                    root = impl;
+                }
+            }
+        }
+        // Preserve a safe prefix of the conjunct order: a later pushed
+        // predicate (e.g. an arrived IN runtime filter) could otherwise prune
+        // rows on which an earlier error-preserving conjunct —
+        // assert_true(...), a failing cast, ... — must still raise. The v1
+        // partition-pruning path (FileScanner::_init_runtime_filter_partition_
+        // prune_ctxs) stops at is_safe_to_execute_on_selected_rows() for the
+        // same reason, so a convertible predicate after an unsafe conjunct
+        // must not be pushed. Safe conjuncts that cannot be converted keep
+        // the old skip: they cannot raise, so pruning rows before they are
+        // evaluated as the residual never loses an error.
+        if (!_is_safe_to_push(root)) {
+            break;
+        }
+        predicate_ptr pred(_convert_expr(root));
+        if (!pred) {
+            continue;
+        }
+        ++_converted_conjuncts;
+        _converted_runtime_filters += is_runtime_filter;
+        if (!result) {
+            result = std::move(pred);
+        } else {
+            // and consumes both inputs regardless of success.
+            result.reset(paimon_predicate_and(result.release(), pred.release()));
+            if (!result) {
+                _converted_conjuncts = 0;
+                _converted_runtime_filters = 0;
+                return nullptr;
+            }
+        }
+    }
+    return result.release();
+}
+
+bool PaimonRustPredicateConverter::_is_safe_to_push(const VExprSPtr& expr) {
+    if (expr->is_safe_to_execute_on_selected_rows()) {
+        return true;
+    }
+    // VectorizedFnCall::is_safe_to_execute_on_selected_rows() admits a fixed
+    // whitelist that does not include `like`, so a like conjunct would always
+    // stop the safe prefix here and never reach _convert_like. A like call
+    // whose children are themselves safe is just as total as the whitelisted
+    // comparisons: it only compares strings, and its regex is built by escaping
+    // the pattern operand (FunctionLike::convert_like_pattern), so no scanned
+    // value can make it raise — a malformed pattern or escape operand fails
+    // at open() on every execution, regardless of which rows survive. Admit
+    // it here rather than widening the generic whitelist, which gates shared
+    // BE pushdown paths outside this reader's scope.
+    auto* fn = dynamic_cast<VectorizedFnCall*>(expr.get());
+    if (fn == nullptr || _normalize_name(fn->function_name()) != "like") {
+        return false;
+    }
+    for (uint16_t i = 0; i < fn->get_num_children(); ++i) {
+        if (!fn->get_child(i)->is_safe_to_execute_on_selected_rows()) {
+            return false;
+        }
+    }
+    return true;
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_convert_expr(const VExprSPtr& expr) {
+    if (!expr) {
+        return nullptr;
+    }
+
+    // Casts are not unwrapped anywhere (predicate root included): a cast node
+    // fails every dispatch below and the conjunct stays in the Doris residual,
+    // mirroring the FE converter, which keeps casted expressions unconverted.
+    if (auto* direct_in = dynamic_cast<VDirectInPredicate*>(expr.get())) {
+        VExprSPtr in_expr;
+        if (direct_in->get_slot_in_expr(in_expr)) {

Review Comment:
   [P2] Reject residual-only IN filters before expanding the set. An arrived IN runtime filter reaches this branch as VDirectInPredicate, and get_slot_in_expr walks its entire HybridSet and allocates one VLiteral per key. Only afterward does _convert_in reject oversized lists or unsupported slot types. With runtime_filter_max_in_num=70000, the 65535-key case added in this PR therefore creates and discards 65535 literals for every logical split while pushing nothing; 10000 splits cause over 655 million unnecessary literal allocations. Check the original set size and slot support before expanding it (the native ORC path already does the size check), and cover the actual VDirectInPredicate path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6188122753

   # FE Regression Coverage Report
   Increment line coverage `  20.00% (19/95)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6150399285

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6163166502

   # FE UT Coverage Report
   Increment line coverage `  81.48% (22/27)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_13e491640aa8d4bb03224c0190f1c40ec26b9db1/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_13e491640aa8d4bb03224c0190f1c40ec26b9db1/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6168591763

   # FE Regression Coverage Report
   Increment line coverage `  41.30% (19/46)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_8d52953c3037b078816cdc14d5695e371b5cfacf_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_8d52953c3037b078816cdc14d5695e371b5cfacf_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68825:
URL: https://github.com/apache/doris/pull/68825#discussion_r4272204393


##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonRustReaderSelector.java:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.paimon;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.handle.ConnectorColumnHandle;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.rest.RESTTokenFileIO;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.types.ArrayType;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.DataTypeRoot;
+import org.apache.paimon.types.MapType;
+import org.apache.paimon.types.MultisetType;
+import org.apache.paimon.types.RowType;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Chooses the Rust reader only for table, split, storage and backend shapes it can read safely.
+ * Nested pruning and precision 7-9 timestamps still require JNI because the pinned Arrow ABI
+ * cannot preserve their full semantics. COUNT(*) metadata ranges bypass this selector entirely.
+ */
+final class PaimonRustReaderSelector {
+    static final String BACKEND_CAPABILITY = "paimon-rust-reader";
+    private static final String ENABLE_PAIMON_RUST_READER = "enable_paimon_rust_reader";
+    private static final String ENABLE_FILE_SCANNER_V2 = "enable_file_scanner_v2";
+    private static final String FORCE_JNI_SCANNER = "force_jni_scanner";
+    private static final String DELETION_VECTORS_MERGE_ON_READ = "deletion-vectors.merge-on-read";
+    private static final Set<String> VERIFIED_LOCATION_SCHEMES =
+            new HashSet<>(Arrays.asList("s3", "s3a", "oss", "hdfs", "file"));
+    private static final Set<String> VERIFIED_HDFS_OPTION_KEYS = new HashSet<>(Arrays.asList(
+            "fs.defaultFS",
+            "hadoop.security.authentication",
+            "hdfs.security.authentication",
+            "ipc.client.fallback-to-simple-auth-allowed"));
+
+    private final FileStoreTable table;
+    private final PaimonRustReaderCapabilities capabilities;
+    private final Map<String, String> backendStorageProperties;
+    private final boolean scanCompatible;
+    private final boolean partialUpdateOrAggregateWithDeletionVectors;
+    private final boolean deletionVectorsMergeOnRead;
+
+    PaimonRustReaderSelector(ConnectorSession session, boolean backendsSupportRust,
+            FileStoreTable table, List<ConnectorColumnHandle> columns,
+            Map<String, String> backendStorageProperties, boolean usesFallbackRead,
+            boolean incrementalRead, boolean projectedVariant) {
+        this.table = table;
+        this.backendStorageProperties = backendStorageProperties;
+        CoreOptions options = table.coreOptions();
+        CoreOptions.MergeEngine mergeEngine = options.mergeEngine();
+        partialUpdateOrAggregateWithDeletionVectors = options.deletionVectorsEnabled()
+                && (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+                        || mergeEngine == CoreOptions.MergeEngine.AGGREGATE);
+        TableSchema schema = table.schema();
+        String mergeOnRead = schema.options().get(DELETION_VECTORS_MERGE_ON_READ);
+        deletionVectorsMergeOnRead = "true".equalsIgnoreCase(mergeOnRead);
+
+        String location = table.location().toString();
+        boolean enabled = session != null && "true".equalsIgnoreCase(
+                session.getSessionProperties().get(ENABLE_PAIMON_RUST_READER));
+        boolean fileScannerV2 = session != null && "true".equalsIgnoreCase(
+                session.getSessionProperties().get(ENABLE_FILE_SCANNER_V2));
+        boolean forceJniScanner = session != null && "true".equalsIgnoreCase(
+                session.getSessionProperties().get(FORCE_JNI_SCANNER));
+        boolean restTokenTable = table.fileIO() instanceof RESTTokenFileIO;
+        boolean unsupportedMergeOption = (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+                || mergeEngine == CoreOptions.MergeEngine.AGGREGATE)
+                && hasRustUnsupportedMergeOption(schema.options(), mergeEngine);
+        boolean deduplicateIgnoreDelete = mergeEngine == CoreOptions.MergeEngine.DEDUPLICATE
+                && options.ignoreDelete();
+        String scanMode = PaimonScanParams.withoutTimeTravelSelectors(schema)
+                .options().get(CoreOptions.SCAN_MODE.key());
+        boolean scanModeSupported = scanMode == null || "default".equalsIgnoreCase(scanMode);
+
+        scanCompatible = enabled && fileScannerV2 && !forceJniScanner
+                && backendsSupportRust
+                && !usesFallbackRead && !incrementalRead && !projectedVariant
+                && !options.queryAuthEnabled() && !restTokenTable
+                && !(partialUpdateOrAggregateWithDeletionVectors && deletionVectorsMergeOnRead)
+                && !deduplicateIgnoreDelete
+                && !unsupportedMergeOption && scanModeSupported
+                && isRustVerifiedLocationScheme(location)
+                && (!isHdfsLocationScheme(location)
+                        || isRustVerifiedHdfsBackend(backendStorageProperties))
+                && isCredentialProviderTranslatable(location, backendStorageProperties);
+        capabilities = new PaimonRustReaderCapabilities(table, columns);
+    }
+
+    boolean canRead(DataSplit split) {
+        if (!scanCompatible || splitHasExternalFiles(split)) {
+            return false;
+        }
+        if (splitHasOrcFile(split) && table.schema().fields().stream()
+                .anyMatch(field -> containsTimestampLtz(field.type()))) {
+            return false;
+        }
+        if (partialUpdateOrAggregateWithDeletionVectors && !deletionVectorsMergeOnRead
+                && !isFullyMaterializedPkDvSplit(split)) {
+            return false;
+        }
+        return capabilities.canRead(split);
+    }
+
+    static boolean isRustVerifiedLocationScheme(String location) {
+        if (location == null) {
+            return false;
+        }
+        int separator = location.indexOf("://");

Review Comment:
   [P2] Reject or qualify authority-less HDFS table locations. A filesystem catalog with warehouse=hdfs:///user/paimon and fs.defaultFS=hdfs://namenode:8020 is valid for Java, but Paimon Path.toString emits hdfs:/user/paimon/db.db/t. Because this helper only looks for ://, it admits that path as scheme-less and isHdfsLocationScheme also skips its checks. RustScanMetadata sends the path unchanged; the pinned Rust HDFS parser requires a host (and hdfs:// for file paths) and does not translate fs.defaultFS, so otherwise eligible scans fail when Rust is enabled. Parse schemes independently of authority, then fully qualify HDFS paths or keep these scans on JNI, and cover this catalog form in selector/differential tests.



##########
be/src/format_v2/table/paimon_rust_table_reader.cpp:
##########
@@ -0,0 +1,944 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace rust_handles {
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn)                \
+    struct type##_deleter {                       \
+        void operator()(paimon_##type* p) const { \
+            if (p) {                              \
+                freefn(p);                        \
+            }                                     \
+        }                                         \
+    };                                            \
+    using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+} // namespace rust_handles
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+    explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+    ~ArrowBatch() {
+        auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+        auto* array = static_cast<ArrowArray*>(batch_.array);
+        if (array && array->release) {
+            array->release(array);
+        }
+        if (schema && schema->release) {
+            schema->release(schema);
+        }
+        paimon_arrow_batch_free(batch_);
+    }
+
+    ArrowBatch(const ArrowBatch&) = delete;
+    ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+    ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+    ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+    paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+    rust_handles::error_ptr owned(err);
+    if (!owned) {
+        return "unknown error";
+    }
+    std::string msg;
+    if (owned->message.data != nullptr && owned->message.len > 0) {
+        msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+    }
+    return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+    std::string out;
+    for (const auto& kv : options) {
+        if (!out.empty()) {
+            out += ", ";
+        }
+        out += kv.first;
+    }
+    return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+    rust_handles::table_ptr table;
+    rust_handles::read_builder_ptr read_builder;
+    rust_handles::plan_ptr plan;
+    rust_handles::table_read_ptr table_read;
+    rust_handles::record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+    RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+    {
+        // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+        // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.init_timer);
+        // Materialize TIMESTAMP_LTZ in the session timezone — the same
+        // convention as the JNI reader (PaimonJniScanner reads time_zone from
+        // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+        // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+        // of _ctz, so NTZ wall-clock semantics are preserved.
+        DORIS_CHECK(_runtime_state != nullptr);
+        _ctz = _runtime_state->timezone_obj();
+        if (_scanner_profile != nullptr) {
+            file_scan_profile::ensure_hierarchy(_scanner_profile);
+            _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+                                               file_scan_profile::TABLE_READER);
+            _rust_open_split_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+            _rust_read_batch_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+            _rust_arrow_to_block_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+            _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+                                                       TUnit::UNIT, "PaimonRustReader");
+            _rust_predicates_converted = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+            _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+                                                         TUnit::UNIT, "PaimonRustReader");
+            _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+            _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+        }
+        // Projected column name -> fixed output position, registered with both the exact and
+        // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+        // semantics: exact match first, lower-case fallback on lookup).
+        _output_name_to_idx.reserve(_projected_columns.size() * 2);
+        for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+            _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+            _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+        }
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+    // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+    // are idempotent, and clear it only when a new split is explicitly prepared.
+    _close_split_reader();
+    _split_eof = false;
+    _current_range = options.current_range;
+    RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+    if (current_split_pruned()) {
+        return Status::OK();
+    }
+    if (_is_table_level_count_active()) {
+        // No rust pipeline is opened; get_block emits the synthetic count rows.
+        return Status::OK();
+    }
+    RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+    {
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.prepare_split_timer);
+        // Open may dominate the scan or fail before get_block; include it in the parent timer.
+        SCOPED_TIMER(_rust_total_time);
+        SCOPED_TIMER(_rust_open_split_time);
+        RETURN_IF_ERROR(_open_split_reader(options.current_range));
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+    SCOPED_TIMER(_profile.total_timer);
+    SCOPED_TIMER(_profile.exec_timer);
+    SCOPED_TIMER(_rust_total_time);
+    DORIS_CHECK(block != nullptr);
+    DORIS_CHECK(eos != nullptr);
+    DORIS_CHECK(block->columns() == _projected_columns.size());
+    block->clear_column_data(_projected_columns.size());
+    *eos = false;
+
+    if (_is_table_level_count_active()) {
+        return _read_table_level_count(block, eos);
+    }
+
+    // num_splits == 0 yields an empty (but valid) stream: report EOF.
+    if (_split_eof) {
+        *eos = true;
+        return Status::OK();
+    }
+    if (!_handles || !_handles->reader) {
+        return Status::InternalError("paimon-rust reader is not initialized");
+    }
+
+    while (true) {
+        // Mirror the base TableReader cancellation contract so a cancelled query does not
+        // drain the whole split.
+        if (_io_ctx != nullptr && _io_ctx->should_stop) {
+            _split_eof = true;
+            _close_split_reader();
+            *eos = true;
+            return Status::OK();
+        }
+
+        if (!_pending_record_batch) {
+            paimon_result_next_batch next;
+            {
+                SCOPED_TIMER(_rust_read_batch_time);
+                next = paimon_record_batch_reader_next(_handles->reader.get());

Review Comment:
   [P1] Charge Rust read buffers to the query memory tracker. The buffers produced by this call remain Rust-owned, and the zero-copy import retains them in _pending_record_batch. In this build malloc/posix_memalign only forward to jemalloc; query tracking is explicit in Doris Allocator, which the Rust system allocator bypasses. Neither this reader nor the C binding reserves or charges those bytes, including Parquet row-group/decode buffers and Tokio worker allocations. Even the added wide-string fixture holds roughly 4 MiB in the Rust batch alongside its tracked Doris copy at a 32-row cap. Large row groups or concurrent scanners can therefore exceed query/workload-group limits while only output columns are counted. Add a query-owned accounting/reservation bridge with balanced release on EOF, error, abort and close; output slicing alone does not address this separate issue.



##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonRustReaderSelector.java:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.paimon;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.handle.ConnectorColumnHandle;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.rest.RESTTokenFileIO;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.types.ArrayType;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.DataTypeRoot;
+import org.apache.paimon.types.MapType;
+import org.apache.paimon.types.MultisetType;
+import org.apache.paimon.types.RowType;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Chooses the Rust reader only for table, split, storage and backend shapes it can read safely.
+ * Nested pruning and precision 7-9 timestamps still require JNI because the pinned Arrow ABI
+ * cannot preserve their full semantics. COUNT(*) metadata ranges bypass this selector entirely.
+ */
+final class PaimonRustReaderSelector {
+    static final String BACKEND_CAPABILITY = "paimon-rust-reader";
+    private static final String ENABLE_PAIMON_RUST_READER = "enable_paimon_rust_reader";
+    private static final String ENABLE_FILE_SCANNER_V2 = "enable_file_scanner_v2";
+    private static final String FORCE_JNI_SCANNER = "force_jni_scanner";
+    private static final String DELETION_VECTORS_MERGE_ON_READ = "deletion-vectors.merge-on-read";
+    private static final Set<String> VERIFIED_LOCATION_SCHEMES =
+            new HashSet<>(Arrays.asList("s3", "s3a", "oss", "hdfs", "file"));
+    private static final Set<String> VERIFIED_HDFS_OPTION_KEYS = new HashSet<>(Arrays.asList(
+            "fs.defaultFS",
+            "hadoop.security.authentication",
+            "hdfs.security.authentication",
+            "ipc.client.fallback-to-simple-auth-allowed"));
+
+    private final FileStoreTable table;
+    private final PaimonRustReaderCapabilities capabilities;
+    private final Map<String, String> backendStorageProperties;
+    private final boolean scanCompatible;
+    private final boolean partialUpdateOrAggregateWithDeletionVectors;
+    private final boolean deletionVectorsMergeOnRead;
+
+    PaimonRustReaderSelector(ConnectorSession session, boolean backendsSupportRust,
+            FileStoreTable table, List<ConnectorColumnHandle> columns,
+            Map<String, String> backendStorageProperties, boolean usesFallbackRead,
+            boolean incrementalRead, boolean projectedVariant) {
+        this.table = table;
+        this.backendStorageProperties = backendStorageProperties;
+        CoreOptions options = table.coreOptions();
+        CoreOptions.MergeEngine mergeEngine = options.mergeEngine();
+        partialUpdateOrAggregateWithDeletionVectors = options.deletionVectorsEnabled()
+                && (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+                        || mergeEngine == CoreOptions.MergeEngine.AGGREGATE);
+        TableSchema schema = table.schema();
+        String mergeOnRead = schema.options().get(DELETION_VECTORS_MERGE_ON_READ);
+        deletionVectorsMergeOnRead = "true".equalsIgnoreCase(mergeOnRead);
+
+        String location = table.location().toString();
+        boolean enabled = session != null && "true".equalsIgnoreCase(
+                session.getSessionProperties().get(ENABLE_PAIMON_RUST_READER));
+        boolean fileScannerV2 = session != null && "true".equalsIgnoreCase(
+                session.getSessionProperties().get(ENABLE_FILE_SCANNER_V2));
+        boolean forceJniScanner = session != null && "true".equalsIgnoreCase(
+                session.getSessionProperties().get(FORCE_JNI_SCANNER));
+        boolean restTokenTable = table.fileIO() instanceof RESTTokenFileIO;
+        boolean unsupportedMergeOption = (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+                || mergeEngine == CoreOptions.MergeEngine.AGGREGATE)
+                && hasRustUnsupportedMergeOption(schema.options(), mergeEngine);
+        boolean deduplicateIgnoreDelete = mergeEngine == CoreOptions.MergeEngine.DEDUPLICATE
+                && options.ignoreDelete();
+        String scanMode = PaimonScanParams.withoutTimeTravelSelectors(schema)
+                .options().get(CoreOptions.SCAN_MODE.key());
+        boolean scanModeSupported = scanMode == null || "default".equalsIgnoreCase(scanMode);
+
+        scanCompatible = enabled && fileScannerV2 && !forceJniScanner
+                && backendsSupportRust
+                && !usesFallbackRead && !incrementalRead && !projectedVariant
+                && !options.queryAuthEnabled() && !restTokenTable
+                && !(partialUpdateOrAggregateWithDeletionVectors && deletionVectorsMergeOnRead)
+                && !deduplicateIgnoreDelete
+                && !unsupportedMergeOption && scanModeSupported
+                && isRustVerifiedLocationScheme(location)
+                && (!isHdfsLocationScheme(location)
+                        || isRustVerifiedHdfsBackend(backendStorageProperties))
+                && isCredentialProviderTranslatable(location, backendStorageProperties);
+        capabilities = new PaimonRustReaderCapabilities(table, columns);
+    }
+
+    boolean canRead(DataSplit split) {
+        if (!scanCompatible || splitHasExternalFiles(split)) {
+            return false;
+        }
+        if (splitHasOrcFile(split) && table.schema().fields().stream()

Review Comment:
   [P2] Preserve the full range of low-precision ORC timestamps. This guard rejects only LTZ, so an append-only ORC table with current-schema TIMESTAMP(3) or TIMESTAMP(6) and a value such as 3000-01-01 still selects Rust. The pinned Paimon OrcFormatReader leaves orc-rust 0.8.0 at its default nanosecond precision; that decoder converts epoch nanoseconds to i64 and errors outside roughly 1677-2262, before any cast to the table's millisecond/microsecond unit. Enabling Rust therefore makes a valid scan fail although JNI can read it. This is a distinct remaining instance from the existing TIMESTAMP(7-9)/Parquet INT96 thread: the <=6 schema guard cannot cover it. Decode ORC at a suitable unit or conservatively route these timestamp schemas to JNI, including nested leaves, and add an out-of-range ORC parity case. The pinned [ORC builder contract](https://docs.rs/orc-rust/0.8.0/orc_rust/arrow_reader/struct.ArrowReaderBuilder.html#method.with_timestamp_precision) documents this default.



##########
be/src/format_v2/table/paimon_rust_table_reader.cpp:
##########
@@ -0,0 +1,944 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace rust_handles {
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn)                \
+    struct type##_deleter {                       \
+        void operator()(paimon_##type* p) const { \
+            if (p) {                              \
+                freefn(p);                        \
+            }                                     \
+        }                                         \
+    };                                            \
+    using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+} // namespace rust_handles
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+    explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+    ~ArrowBatch() {
+        auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+        auto* array = static_cast<ArrowArray*>(batch_.array);
+        if (array && array->release) {
+            array->release(array);
+        }
+        if (schema && schema->release) {
+            schema->release(schema);
+        }
+        paimon_arrow_batch_free(batch_);
+    }
+
+    ArrowBatch(const ArrowBatch&) = delete;
+    ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+    ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+    ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+    paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+    rust_handles::error_ptr owned(err);
+    if (!owned) {
+        return "unknown error";
+    }
+    std::string msg;
+    if (owned->message.data != nullptr && owned->message.len > 0) {
+        msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+    }
+    return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+    std::string out;
+    for (const auto& kv : options) {
+        if (!out.empty()) {
+            out += ", ";
+        }
+        out += kv.first;
+    }
+    return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+    rust_handles::table_ptr table;
+    rust_handles::read_builder_ptr read_builder;
+    rust_handles::plan_ptr plan;
+    rust_handles::table_read_ptr table_read;
+    rust_handles::record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+    RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+    {
+        // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+        // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.init_timer);
+        // Materialize TIMESTAMP_LTZ in the session timezone — the same
+        // convention as the JNI reader (PaimonJniScanner reads time_zone from
+        // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+        // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+        // of _ctz, so NTZ wall-clock semantics are preserved.
+        DORIS_CHECK(_runtime_state != nullptr);
+        _ctz = _runtime_state->timezone_obj();
+        if (_scanner_profile != nullptr) {
+            file_scan_profile::ensure_hierarchy(_scanner_profile);
+            _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+                                               file_scan_profile::TABLE_READER);
+            _rust_open_split_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+            _rust_read_batch_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+            _rust_arrow_to_block_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+            _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+                                                       TUnit::UNIT, "PaimonRustReader");
+            _rust_predicates_converted = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+            _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+                                                         TUnit::UNIT, "PaimonRustReader");
+            _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+            _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+        }
+        // Projected column name -> fixed output position, registered with both the exact and
+        // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+        // semantics: exact match first, lower-case fallback on lookup).
+        _output_name_to_idx.reserve(_projected_columns.size() * 2);
+        for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+            _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+            _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+        }
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+    // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+    // are idempotent, and clear it only when a new split is explicitly prepared.
+    _close_split_reader();
+    _split_eof = false;
+    _current_range = options.current_range;
+    RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+    if (current_split_pruned()) {
+        return Status::OK();
+    }
+    if (_is_table_level_count_active()) {
+        // No rust pipeline is opened; get_block emits the synthetic count rows.
+        return Status::OK();
+    }
+    RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+    {
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.prepare_split_timer);
+        // Open may dominate the scan or fail before get_block; include it in the parent timer.
+        SCOPED_TIMER(_rust_total_time);
+        SCOPED_TIMER(_rust_open_split_time);
+        RETURN_IF_ERROR(_open_split_reader(options.current_range));
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+    SCOPED_TIMER(_profile.total_timer);
+    SCOPED_TIMER(_profile.exec_timer);
+    SCOPED_TIMER(_rust_total_time);
+    DORIS_CHECK(block != nullptr);
+    DORIS_CHECK(eos != nullptr);
+    DORIS_CHECK(block->columns() == _projected_columns.size());
+    block->clear_column_data(_projected_columns.size());
+    *eos = false;
+
+    if (_is_table_level_count_active()) {
+        return _read_table_level_count(block, eos);
+    }
+
+    // num_splits == 0 yields an empty (but valid) stream: report EOF.
+    if (_split_eof) {
+        *eos = true;
+        return Status::OK();
+    }
+    if (!_handles || !_handles->reader) {
+        return Status::InternalError("paimon-rust reader is not initialized");
+    }
+
+    while (true) {
+        // Mirror the base TableReader cancellation contract so a cancelled query does not
+        // drain the whole split.
+        if (_io_ctx != nullptr && _io_ctx->should_stop) {
+            _split_eof = true;
+            _close_split_reader();
+            *eos = true;
+            return Status::OK();
+        }
+
+        if (!_pending_record_batch) {
+            paimon_result_next_batch next;
+            {
+                SCOPED_TIMER(_rust_read_batch_time);
+                next = paimon_record_batch_reader_next(_handles->reader.get());
+            }
+            if (next.error != nullptr) {

Review Comment:
   [P2] Preserve missing-data-file status on the Rust path. Rust now precedes the native reader for eligible ordinary append-only splits. If a single-file Parquet split's data file disappears after planning, Rust's storage error is returned here as INTERNAL_ERROR. FileScannerV2 skips only NOT_FOUND, so ignore_not_found_file_in_external_table (true by default, and applicable to Paimon) stops working; the same split on the native route preserves ENOENT as NOT_FOUND and follows that policy. Preserve a distinguishable missing-data-file error through the Rust binding and map it to the scanner's expected status, without treating other IO/schema errors as missing files. Add native/Rust differential coverage with the file removed after planning and both ignore settings.



##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -897,6 +911,16 @@ && hasTrustworthyLimitAccounting(table)
             }
         }
 
+        boolean incrementalRead = pinnedOptions.containsKey("incremental-between")
+                || pinnedOptions.containsKey("incremental-between-timestamp");
+        PaimonRustReaderSelector rustReaderSelector = table instanceof FileStoreTable
+                && !requiresMetadataColumns
+                ? new PaimonRustReaderSelector(session, backendsSupportRust,
+                        (FileStoreTable) table, columns, backendStorageProperties(),

Review Comment:
   [P2] Check and transport the effective Paimon storage configuration. This selector sees only the typed backend map, but PaimonCatalogFactory.applyStorageConfig applies raw fs.s3a.* and paimon.s3.* overrides after that map when constructing Java FileIO. For example, canonical s3.access_key/secret_key can name pair A while fs.s3a.access.key/secret.key override it with readable pair B. A non-raw append-only data-evolution split with INT columns previously uses JNI and retains B in its serialized FileIO; it now passes Rust's static-key gate using A. The Rust payload never carries the raw overlays and its schema-json FileIO constructor does not recover them, so the scan uses A and fails access where JNI succeeds. Preserve the effective option precedence in the Rust transport/eligibility check, or fall back for untranslatable overrides; add a differential catalog case.



##########
be/src/format_v2/table/paimon_rust_table_reader.cpp:
##########
@@ -0,0 +1,944 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace rust_handles {
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn)                \
+    struct type##_deleter {                       \
+        void operator()(paimon_##type* p) const { \
+            if (p) {                              \
+                freefn(p);                        \
+            }                                     \
+        }                                         \
+    };                                            \
+    using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+} // namespace rust_handles
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+    explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+    ~ArrowBatch() {
+        auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+        auto* array = static_cast<ArrowArray*>(batch_.array);
+        if (array && array->release) {
+            array->release(array);
+        }
+        if (schema && schema->release) {
+            schema->release(schema);
+        }
+        paimon_arrow_batch_free(batch_);
+    }
+
+    ArrowBatch(const ArrowBatch&) = delete;
+    ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+    ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+    ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+    paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+    rust_handles::error_ptr owned(err);
+    if (!owned) {
+        return "unknown error";
+    }
+    std::string msg;
+    if (owned->message.data != nullptr && owned->message.len > 0) {
+        msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+    }
+    return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+    std::string out;
+    for (const auto& kv : options) {
+        if (!out.empty()) {
+            out += ", ";
+        }
+        out += kv.first;
+    }
+    return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+    rust_handles::table_ptr table;
+    rust_handles::read_builder_ptr read_builder;
+    rust_handles::plan_ptr plan;
+    rust_handles::table_read_ptr table_read;
+    rust_handles::record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+    RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+    {
+        // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+        // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.init_timer);
+        // Materialize TIMESTAMP_LTZ in the session timezone — the same
+        // convention as the JNI reader (PaimonJniScanner reads time_zone from
+        // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+        // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+        // of _ctz, so NTZ wall-clock semantics are preserved.
+        DORIS_CHECK(_runtime_state != nullptr);
+        _ctz = _runtime_state->timezone_obj();
+        if (_scanner_profile != nullptr) {
+            file_scan_profile::ensure_hierarchy(_scanner_profile);
+            _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+                                               file_scan_profile::TABLE_READER);
+            _rust_open_split_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+            _rust_read_batch_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+            _rust_arrow_to_block_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+            _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+                                                       TUnit::UNIT, "PaimonRustReader");
+            _rust_predicates_converted = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+            _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+                                                         TUnit::UNIT, "PaimonRustReader");
+            _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+            _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+        }
+        // Projected column name -> fixed output position, registered with both the exact and
+        // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+        // semantics: exact match first, lower-case fallback on lookup).
+        _output_name_to_idx.reserve(_projected_columns.size() * 2);
+        for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+            _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+            _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+        }
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+    // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+    // are idempotent, and clear it only when a new split is explicitly prepared.
+    _close_split_reader();
+    _split_eof = false;
+    _current_range = options.current_range;
+    RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+    if (current_split_pruned()) {
+        return Status::OK();
+    }
+    if (_is_table_level_count_active()) {
+        // No rust pipeline is opened; get_block emits the synthetic count rows.
+        return Status::OK();
+    }
+    RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+    {
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.prepare_split_timer);
+        // Open may dominate the scan or fail before get_block; include it in the parent timer.
+        SCOPED_TIMER(_rust_total_time);
+        SCOPED_TIMER(_rust_open_split_time);
+        RETURN_IF_ERROR(_open_split_reader(options.current_range));
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+    SCOPED_TIMER(_profile.total_timer);
+    SCOPED_TIMER(_profile.exec_timer);
+    SCOPED_TIMER(_rust_total_time);
+    DORIS_CHECK(block != nullptr);
+    DORIS_CHECK(eos != nullptr);
+    DORIS_CHECK(block->columns() == _projected_columns.size());
+    block->clear_column_data(_projected_columns.size());
+    *eos = false;
+
+    if (_is_table_level_count_active()) {
+        return _read_table_level_count(block, eos);
+    }
+
+    // num_splits == 0 yields an empty (but valid) stream: report EOF.
+    if (_split_eof) {
+        *eos = true;
+        return Status::OK();
+    }
+    if (!_handles || !_handles->reader) {
+        return Status::InternalError("paimon-rust reader is not initialized");
+    }
+
+    while (true) {
+        // Mirror the base TableReader cancellation contract so a cancelled query does not
+        // drain the whole split.
+        if (_io_ctx != nullptr && _io_ctx->should_stop) {
+            _split_eof = true;
+            _close_split_reader();
+            *eos = true;
+            return Status::OK();
+        }
+
+        if (!_pending_record_batch) {
+            paimon_result_next_batch next;
+            {
+                SCOPED_TIMER(_rust_read_batch_time);
+                next = paimon_record_batch_reader_next(_handles->reader.get());
+            }
+            if (next.error != nullptr) {
+                return Status::InternalError("paimon-rust read batch failed: {}",
+                                             consume_error(next.error));
+            }
+            // End of stream: both pointers are null.
+            if (next.batch.array == nullptr && next.batch.schema == nullptr) {
+                _split_eof = true;
+                _close_split_reader();
+                *eos = true;
+                return Status::OK();
+            }
+
+            // RAII: the batch's Arrow release callbacks + container free run when
+            // `batch` leaves this scope, including on any early return.
+            ArrowBatch batch(next.batch);
+            arrow::Result<std::shared_ptr<arrow::RecordBatch>> import_result =
+                    arrow::ImportRecordBatch(batch.array(), batch.schema());
+            if (!import_result.ok()) {
+                return Status::InternalError("failed to import paimon-rust arrow batch: {}",
+                                             import_result.status().message());
+            }
+            _pending_record_batch = std::move(import_result).ValueUnsafe();
+            _pending_record_batch_offset = 0;
+            if (_pending_record_batch->num_rows() == 0) {
+                _pending_record_batch.reset();
+                continue;
+            }
+        }
+
+        const size_t batch_size = static_cast<size_t>(
+                _batch_size > 0 ? _batch_size : std::max(1, _runtime_state->batch_size()));
+        const size_t remaining = static_cast<size_t>(_pending_record_batch->num_rows()) -
+                                 _pending_record_batch_offset;
+        const size_t rows = std::min(batch_size, remaining);
+        RETURN_IF_ERROR(_fill_block_from_record_batch(_pending_record_batch, block,
+                                                      _pending_record_batch_offset, rows));
+        _pending_record_batch_offset += rows;
+        if (_pending_record_batch_offset ==
+            static_cast<size_t>(_pending_record_batch->num_rows())) {
+            _pending_record_batch.reset();
+            _pending_record_batch_offset = 0;
+        }
+        _record_scan_rows(rows);
+        *eos = false;
+        return Status::OK();
+    }
+}
+
+Status PaimonRustTableReader::abort_split() {
+    {
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.close_timer);
+        _close_split_reader();
+        _split_eof = false;
+    }
+    return format::TableReader::abort_split();
+}
+
+#ifdef BE_TEST
+std::string PaimonRustTableReader::TEST_format_options(
+        const std::map<std::string, std::string>& options) {
+    return format_options(options);
+}
+
+std::map<std::string, std::string> PaimonRustTableReader::TEST_build_options(
+        TFileScanRangeParams* scan_params, const TFileRangeDesc& range) {
+    TFileScanRangeParams* previous_params = _scan_params;
+    TFileRangeDesc previous_range = _current_range;
+    _scan_params = scan_params;
+    _current_range = range;
+    std::map<std::string, std::string> options = _build_options();
+    _scan_params = previous_params;
+    _current_range = std::move(previous_range);
+    return options;
+}
+#endif
+
+Status PaimonRustTableReader::close() {
+    {
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.close_timer);
+        _close_split_reader();
+        _close_table();
+    }
+    return format::TableReader::close();
+}
+
+Status PaimonRustTableReader::_validate_rust_split(const TFileRangeDesc& range) const {
+    if (!range.__isset.table_format_params || !range.table_format_params.__isset.paimon_params) {
+        return Status::InternalError(
+                "missing paimon_params for paimon rust reader, possibly caused by FE/BE protocol "
+                "mismatch");
+    }
+    const auto& params = range.table_format_params.paimon_params;
+    if (!params.__isset.paimon_split || params.paimon_split.empty()) {
+        return Status::InternalError(
+                "missing paimon_split for paimon rust reader, possibly caused by FE/BE protocol "
+                "mismatch");
+    }
+    if (params.__isset.reader_type && params.reader_type != TPaimonReaderType::PAIMON_RUST) {
+        return Status::InternalError(
+                "invalid reader_type for paimon rust reader, possibly caused by FE/BE protocol "
+                "mismatch");
+    }
+    if (!_resolve_table_path(range).has_value()) {
+        return Status::InternalError(
+                "paimon-rust missing paimon_table; cannot resolve paimon table location");
+    }
+    if (!_resolve_db_name(range).has_value()) {
+        return Status::InternalError(
+                "paimon-rust missing db_name; cannot open paimon table via schema json");
+    }
+    if (!_resolve_table_name(range).has_value()) {
+        return Status::InternalError(
+                "paimon-rust missing table_name; cannot open paimon table via schema json");
+    }
+    if (!_resolve_table_schema_json(range).has_value()) {
+        return Status::InternalError(
+                "paimon-rust missing paimon_table_schema_json; cannot open paimon table via "
+                "schema json");
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::_open_split_reader(const TFileRangeDesc& range) {
+    // 1. Decode the FE-planned split first so we fail fast (and without any
+    // filesystem IO) when it is missing or malformed.
+    std::string split_bytes;
+    RETURN_IF_ERROR(_decode_split_bytes(&split_bytes));
+
+    // 2. Resolve identifier + table_path + FE-supplied TableSchema JSON.
+    auto table_path = _resolve_table_path(range).value();
+    auto db_name = _resolve_db_name(range).value();
+    auto table_name = _resolve_table_name(range).value();
+    auto schema_json = _resolve_table_schema_json(range).value();
+    auto branch_opt = _resolve_branch(range);
+
+    // 3. Assemble storage options: FE-supplied paimon options + hadoop_conf +
+    // OSS/S3 → AWS_* translations. These feed FileIO only (per
+    // paimon_table_from_schema_json contract); they are NOT merged into the
+    // supplied table schema.
+    auto options = _build_options();
+
+    auto opened_table_key =
+            std::make_tuple(table_path, schema_json, db_name, table_name, branch_opt, options);
+    if (!_handles || !_handles->table || _opened_table_key != opened_table_key) {
+        // A paimon scan reads one table, so the handle is opened at most once per
+        // distinct identity (e.g. re-created after a close); splits of the same
+        // table reuse it and only rebuild the read pipeline below.
+        _close_table();
+        _handles = std::make_unique<PaimonHandles>();
+
+        std::vector<paimon_option> c_options;
+        c_options.reserve(options.size());
+        for (const auto& kv : options) {
+            c_options.push_back(paimon_option {kv.first.c_str(), kv.second.c_str()});
+        }
+
+        LOG(INFO) << "paimon-rust opening table via schema json: db=" << db_name
+                  << " table=" << table_name << " path=" << table_path
+                  << " branch=" << (branch_opt.has_value() ? branch_opt.value() : "main")
+                  << " storage_options=[" << format_options(options) << "]";
+
+        // Build the table directly from the FE-supplied schema JSON. The Rust
+        // side rejects null / empty branch, so we default to paimon's canonical
+        // "main" sentinel when FE did not set paimon_branch (i.e. the table is
+        // on the main branch — matches upstream Identifier.DEFAULT_MAIN_BRANCH).
+        const std::string& branch_str = branch_opt.has_value() ? branch_opt.value() : "main";
+        paimon_result_get_table tbl_res = paimon_table_from_schema_json(
+                table_path.c_str(), schema_json.c_str(), db_name.c_str(), table_name.c_str(),
+                branch_str.c_str(), c_options.empty() ? nullptr : c_options.data(),
+                c_options.size());
+        if (tbl_res.error != nullptr) {
+            return Status::InternalError(
+                    "paimon-rust table_from_schema_json failed: db={} table={} err={}", db_name,
+                    table_name, consume_error(tbl_res.error));
+        }
+        _handles->table.reset(tbl_res.table);
+        _opened_table_key = std::move(opened_table_key);
+    }
+
+    // 4. Build the read pipeline: read_builder -> case-insensitive -> projection.
+    // Bound the Rust Arrow allocation itself: wide rows must respect the scanner's
+    // adaptive probe size before they are materialized into a Doris block.
+    const auto batch_size = std::to_string(
+            _batch_size > 0 ? _batch_size : std::max(1, _runtime_state->batch_size()));
+    const paimon_option batch_option {"read.batch-size", batch_size.c_str()};
+    paimon_result_read_builder rb_res =
+            paimon_table_new_read_builder_with_options(_handles->table.get(), &batch_option, 1);
+    if (rb_res.error != nullptr) {
+        return Status::InternalError("paimon-rust new read builder failed: {}",
+                                     consume_error(rb_res.error));
+    }
+    _handles->read_builder.reset(rb_res.read_builder);
+
+    // Fold column casing on the Rust side so FE-normalized lowercase names
+    // resolve against tables with mixed-case column definitions.
+    if (paimon_error* case_err =
+                paimon_read_builder_with_case_sensitive(_handles->read_builder.get(), false)) {
+        return Status::InternalError("paimon-rust set case_sensitive failed: {}",
+                                     consume_error(case_err));
+    }
+
+    // Partition keys are excluded: they are materialized from split metadata
+    // (see _fill_non_arrow_columns), and paimon-rust does not emit them.
+    auto read_columns = _build_read_columns();
+    std::vector<const char*> projection;
+    projection.reserve(read_columns.size() + 1);
+    for (const auto& col : read_columns) {
+        projection.push_back(col.c_str());
+    }
+    projection.push_back(nullptr);
+    if (paimon_error* proj_err = paimon_read_builder_with_projection(_handles->read_builder.get(),
+                                                                     projection.data())) {
+        return Status::InternalError("paimon-rust set projection failed: {}",
+                                     consume_error(proj_err));
+    }
+
+    // Convert the scanner conjuncts into a paimon-rust filter and apply it.
+    RETURN_IF_ERROR(_apply_predicate());
+
+    // 5. Deserialize the FE-planned split into a one-split plan, so this
+    // scanner reads exactly the split it was assigned rather than replanning
+    // the whole table. The wire form is identical to what paimon-cpp consumes
+    // (`paimon::table::DataSplit::serialize`).
+    paimon_result_plan plan_res = paimon_plan_from_split_bytes(
+            reinterpret_cast<const uint8_t*>(split_bytes.data()), split_bytes.size());
+    if (plan_res.error != nullptr) {
+        return Status::InternalError("paimon-rust build plan failed: {}",
+                                     consume_error(plan_res.error));
+    }
+    _handles->plan.reset(plan_res.plan);
+
+    size_t num_splits = paimon_plan_num_splits(_handles->plan.get());
+    if (num_splits == 0) {
+        _split_eof = true;
+        return Status::OK();
+    }
+
+    // 6. Open the arrow stream over the plan.
+    paimon_result_new_read read_res = paimon_read_builder_new_read(_handles->read_builder.get());
+    if (read_res.error != nullptr) {
+        return Status::InternalError("paimon-rust new read failed: {}",
+                                     consume_error(read_res.error));
+    }
+    _handles->table_read.reset(read_res.read);
+
+    paimon_result_record_batch_reader rdr_res = paimon_table_read_to_arrow(
+            _handles->table_read.get(), _handles->plan.get(), /*offset=*/0, /*length=*/num_splits);
+    if (rdr_res.error != nullptr) {
+        return Status::InternalError("paimon-rust open arrow reader failed: {}",
+                                     consume_error(rdr_res.error));
+    }
+    _handles->reader.reset(rdr_res.reader);
+    return Status::OK();
+}
+
+void PaimonRustTableReader::_close_split_reader() {
+    _pending_record_batch.reset();
+    _pending_record_batch_offset = 0;
+    if (!_handles) {
+        return;
+    }
+    // Reverse of the declaration order in PaimonHandles.
+    _handles->reader.reset();
+    _handles->table_read.reset();
+    _handles->plan.reset();
+    _handles->read_builder.reset();
+}
+
+void PaimonRustTableReader::_close_table() {
+    if (!_handles) {
+        return;
+    }
+    _close_split_reader();
+    _handles->table.reset();
+    _opened_table_key.reset();
+}
+
+Status PaimonRustTableReader::_apply_predicate() {
+    if (_conjuncts.empty() || !_handles || !_handles->table || !_handles->read_builder) {
+        return Status::OK();
+    }
+    if (_scanner_profile != nullptr) {
+        COUNTER_UPDATE(_rust_predicates_input, _conjuncts.size());
+        for (const auto& conjunct : _conjuncts) {
+            if (conjunct && conjunct->root() && conjunct->root()->is_rf_wrapper()) {
+                COUNTER_UPDATE(_rust_runtime_filters_input, 1);
+            }
+        }
+    }
+    VLOG_DEBUG << "paimon-rust predicate pushdown: " << _conjuncts.size() << " conjunct(s) input";
+    // The conjunct VSlotRefs carry table global indices (positions), so the v2
+    // converter mode resolves fields by the projected column names; partition
+    // keys are excluded because the rust reader does not read them.
+    std::vector<std::string> names;
+    std::vector<DataTypePtr> types;
+    names.reserve(_projected_columns.size());
+    types.reserve(_projected_columns.size());
+    for (const auto& col : _projected_columns) {
+        if (col.is_partition_key) {
+            continue;
+        }
+        names.push_back(col.name);
+        types.push_back(col.type);
+    }
+    PaimonRustPredicateConverter converter(names, types, _handles->table.get());
+    paimon_predicate* predicate = converter.build(_conjuncts);
+    if (_scanner_profile != nullptr) {
+        COUNTER_UPDATE(_rust_predicates_converted, converter.converted_conjuncts());
+    }
+    if (predicate == nullptr) {
+        VLOG_DEBUG << "paimon-rust predicate pushdown: nothing convertible, no filter applied";
+        return Status::OK();
+    }
+    // paimon_read_builder_with_filter consumes the predicate (ownership moves to
+    // the builder) on every path, so we must not free it here.
+    if (paimon_error* err =
+                paimon_read_builder_with_filter(_handles->read_builder.get(), predicate)) {
+        return Status::InternalError("paimon-rust apply filter failed: {}", consume_error(err));
+    }
+    // Count application only after the C API accepts the filter; reader timers
+    // alone cannot distinguish pushdown from Doris residual-only execution.
+    if (_scanner_profile != nullptr) {
+        COUNTER_UPDATE(_rust_predicates_applied, converter.converted_conjuncts());
+        COUNTER_UPDATE(_rust_runtime_filters_applied, converter.converted_runtime_filters());
+    }
+    VLOG_DEBUG << "paimon-rust predicate pushdown: applied";
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::_fill_block_from_record_batch(
+        const std::shared_ptr<arrow::RecordBatch>& batch, Block* block, size_t start, size_t rows) {
+    SCOPED_TIMER(_rust_arrow_to_block_time);
+    DORIS_CHECK(batch != nullptr);
+    DORIS_CHECK(block != nullptr);
+    DORIS_CHECK_LE(start + rows, static_cast<size_t>(batch->num_rows()));
+    const auto arrow_start = static_cast<int64_t>(start);
+    const auto arrow_end = static_cast<int64_t>(start + rows);
+    std::unordered_set<size_t> materialized_indices;
+    materialized_indices.reserve(_projected_columns.size());
+    {
+        auto columns_guard = block->mutate_columns_scoped();
+        auto& columns = columns_guard.mutable_columns();
+        for (int c = 0; c < batch->num_columns(); ++c) {
+            const auto& field = batch->schema()->field(c);
+            if (field->name() == VALUE_KIND_FIELD) {
+                continue;
+            }
+            // Projected column names are FE-normalized to lowercase.
+            // paimon-rust's case_sensitive=false setting also case-folds column
+            // names in the schema output, so exact match works — but tolerate
+            // mixed-case Rust output by folding here as well.
+            auto it = _output_name_to_idx.find(field->name());
+            if (it == _output_name_to_idx.end()) {
+                it = _output_name_to_idx.find(to_lower(field->name()));
+            }
+            if (it == _output_name_to_idx.end()) {
+                // Skip columns that are not in the block (e.g. columns dropped by
+                // slot pruning).
+                continue;
+            }
+            const auto output_idx = it->second;
+            if (!materialized_indices.emplace(output_idx).second) {
+                return Status::InternalError("paimon-rust returned duplicate column '{}'",
+                                             field->name());
+            }
+            try {
+                RETURN_IF_ERROR(columns_guard.get_datatype_by_position(output_idx)
+                                        ->get_serde()
+                                        ->read_column_from_arrow(*columns[output_idx],

Review Comment:
   [P2] Handle sliced BOOLEAN arrays produced by data evolution. Eligible data-evolution.enabled append-only tables can produce sliced arrays inside Rust's column-wise merge. For overlapping partial-column files whose Parquet row groups have different boundaries (for example source A first emits 17 rows while a BOOLEAN source B emits 32), DataEvolutionReader emits 17 rows and then B.slice(17,15). Arrow 58.4 preserves the BOOLEAN bit offset in ArrayData and the C Data Interface, so this direct SerDe call fails with non-zero array offset is not supported under the default enable_arrow_input_validation=true. Keeping the original imported batch only avoids slices made by this C++ reader; it does not normalize Rust's incoming bit buffers. Normalize these BOOLEAN slices or keep affected data-evolution scans on JNI, and add a differential fixture with mismatched source batch boundaries. See [BOOLEAN export offset](https://github.com/apache/arrow-rs/blob/58.4.0/arrow-array/src/array/boolean_
 array.rs#L745-L752).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68825:
URL: https://github.com/apache/doris/pull/68825#discussion_r4230741120


##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -575,7 +581,8 @@ public List<ConnectorScanRange> planScan(ConnectorSession session, ConnectorScan
         return scanReuse.computeIfAbsent(reuseKey,
                 key -> Collections.unmodifiableList(planScanInternal(session,
                         request.getTableHandle(), request.getColumns(), request.getFilter(),

Review Comment:
   [P1] Include Rust eligibility in the scan reuse key. The key records only root column names, so with nested pruning enabled two aliases reading the same struct root, one full and one `s.b`, collide even though only the full projection may use Rust. If it plans first, this cache hands its Rust ranges to the nested scan, bypassing the selector's required JNI fallback. The common backend capability set is also absent, so independently selected aliases in a mixed-version BE pool can reuse ranges assigned to an incapable backend. Key the projection shape and backend capabilities before reusing these ranges.



##########
fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanRequest.java:
##########
@@ -147,10 +151,20 @@ public boolean isExplainOnly() {
         return explainOnly;
     }
 
+    /** Capabilities shared by every backend eligible to execute this scan. */
+    public Set<String> getBackendCapabilities() {
+        return backendCapabilities;

Review Comment:
   [P1] Bump the connector SPI major for the new request method. The Paimon plugin now invokes `ConnectorScanRequest.allBackendsSupport`, but the plugin API stays at 12.0. An updated directory plugin can pass the old FE's major-version gate and fail its first scan with `NoSuchMethodError`. Increment the API major and update the pinned surface/version baselines so that older FEs reject the incompatible plugin at load time.



##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -1650,6 +1668,25 @@ private PaimonScanRange buildJniScanRange(Split split, String defaultFileFormat,
         return builder.build();
     }
 
+    private PaimonScanRange buildRustScanRange(DataSplit split, FileStoreTable table,
+            PaimonTableHandle handle, String defaultFileFormat,

Review Comment:
   [P2] Serialize the table schema once per scan. `buildRustScanRange` runs for every logical split and calls `table.schema()`, strips options, and encodes a fresh JSON string each time; every range retains that identical string. A split-heavy table can consume hundreds of MB or more of FE heap and repeat substantial JSON work during planning. Compute the schema JSON and branch once outside the split loop and share them among range builders.



##########
be/src/format_v2/table/paimon_rust_predicate_converter.cpp:
##########
@@ -0,0 +1,840 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+
+#include <algorithm>
+#include <cctype>
+#include <limits>
+#include <memory>
+#include <utility>
+
+#include "common/logging.h"
+#include "core/column/column_const.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "core/types.h"
+#include "core/value/decimalv2_value.h"
+#include "core/value/timestamptz_value.h"
+#include "core/value/vdatetime_value.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vcompound_pred.h"
+#include "exprs/vdirect_in_predicate.h"
+#include "exprs/vectorized_fn_call.h"
+#include "exprs/vexpr.h"
+#include "exprs/vin_predicate.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+
+namespace doris {
+
+namespace {
+// paimon_datum tags (see paimon.h / bindings/c/src/table.rs::datum_from_c).
+constexpr int32_t kTagBool = 0;
+constexpr int32_t kTagTinyInt = 1;
+constexpr int32_t kTagSmallInt = 2;
+constexpr int32_t kTagInt = 3;
+constexpr int32_t kTagLong = 4;
+constexpr int32_t kTagDouble = 6;
+constexpr int32_t kTagString = 7;
+constexpr int32_t kTagDate = 8;
+constexpr int32_t kTagTimestamp = 10;
+constexpr int32_t kTagDecimal = 12;
+constexpr int32_t kTagBytes = 13;
+
+// paimon decimal precision ceiling (paimon::Decimal::MAX_PRECISION).
+constexpr int32_t kPaimonDecimalMaxPrecision = 38;
+
+// RAII for an owned paimon_predicate*. and/or/not consume their inputs, so we
+// release() before handing pointers to them.
+struct predicate_deleter {
+    void operator()(paimon_predicate* p) const {
+        if (p) {
+            paimon_predicate_free(p);
+        }
+    }
+};
+using predicate_ptr = std::unique_ptr<paimon_predicate, predicate_deleter>;
+
+// RAII for an owned paimon_error*.
+struct error_deleter {
+    void operator()(paimon_error* p) const {
+        if (p) {
+            paimon_error_free(p);
+        }
+    }
+};
+using error_ptr = std::unique_ptr<paimon_error, error_deleter>;
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_predicate_error(paimon_error* err) {
+    error_ptr owned(err);
+    if (!owned) {
+        return "unknown error";
+    }
+    std::string msg;
+    if (owned->message.data != nullptr && owned->message.len > 0) {
+        msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+    }
+    return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+} // namespace
+
+PaimonRustPredicateConverter::PaimonRustPredicateConverter(
+        const std::vector<std::string>& column_names, const std::vector<DataTypePtr>& column_types,
+        const paimon_table* table)
+        : _table(table) {
+    DORIS_CHECK(column_names.size() == column_types.size());
+    _columns_by_name.reserve(column_names.size());
+    for (size_t i = 0; i < column_names.size(); ++i) {
+        _columns_by_name.emplace(_normalize_name(column_names[i]),
+                                 std::make_pair(column_names[i], column_types[i]));
+    }
+    // Paimon TIMESTAMP (wall clock) is stored as epoch-millis-of-the-wall-time
+    // and the DateTimeV2 serde decodes timezone-naive arrow values in UTC, so
+    // timestamp literals convert wall->epoch in UTC. utc_time_zone() needs no
+    // tzdata lookup, so the conversion cannot silently fall back to a
+    // machine-local zone.
+    _utc_tz = cctz::utc_time_zone();
+}
+
+paimon_predicate* PaimonRustPredicateConverter::build(const VExprContextSPtrs& conjuncts) {
+    _converted_conjuncts = 0;
+    _converted_runtime_filters = 0;
+    if (_table == nullptr) {
+        return nullptr;
+    }
+    predicate_ptr result;
+    for (const auto& conjunct : conjuncts) {
+        if (!conjunct || !conjunct->root()) {
+            continue;
+        }
+        auto root = conjunct->root();
+        const bool is_runtime_filter = root->is_rf_wrapper();
+        if (root->is_rf_wrapper()) {
+            if (auto impl = root->get_impl()) {
+                // A null-aware runtime filter (an EQ_FOR_NULL join) must stay
+                // residual: its wrapper execution restores NULL probe rows to
+                // true (RuntimeFilterExpr::change_null_to_true), while the
+                // unwrapped impl — rebuilt as an ordinary IN predicate through
+                // VDirectInPredicate::get_slot_in_expr — treats NULL as
+                // not-in-set and would prune the NULL probes permanently
+                // before the join sees them. Keep the wrapper itself: it fails
+                // every dispatch below, so the conjunct stays in the residual,
+                // and it is safe to execute on selected rows, so later
+                // conjuncts keep pushing. The lance pushdown declines
+                // is_null_aware() filters for the same reason.
+                // is_null_aware() is concrete on RuntimeFilterExpr (the only
+                // class whose is_rf_wrapper() is true), so the dynamic_cast
+                // never fails in practice; the null guard keeps the unwrap for
+                // any future wrapper shape.
+                auto* rf_wrapper = dynamic_cast<RuntimeFilterExpr*>(root.get());
+                if (rf_wrapper == nullptr || !rf_wrapper->is_null_aware()) {
+                    root = impl;
+                }
+            }
+        }
+        // Preserve a safe prefix of the conjunct order: a later pushed
+        // predicate (e.g. an arrived IN runtime filter) could otherwise prune
+        // rows on which an earlier error-preserving conjunct —
+        // assert_true(...), a failing cast, ... — must still raise. The v1
+        // partition-pruning path (FileScanner::_init_runtime_filter_partition_
+        // prune_ctxs) stops at is_safe_to_execute_on_selected_rows() for the
+        // same reason, so a convertible predicate after an unsafe conjunct
+        // must not be pushed. Safe conjuncts that cannot be converted keep
+        // the old skip: they cannot raise, so pruning rows before they are
+        // evaluated as the residual never loses an error.
+        if (!_is_safe_to_push(root)) {
+            break;
+        }
+        predicate_ptr pred(_convert_expr(root));
+        if (!pred) {
+            continue;
+        }
+        ++_converted_conjuncts;
+        _converted_runtime_filters += is_runtime_filter;
+        if (!result) {
+            result = std::move(pred);
+        } else {
+            // and consumes both inputs regardless of success.
+            result.reset(paimon_predicate_and(result.release(), pred.release()));
+            if (!result) {
+                _converted_conjuncts = 0;
+                _converted_runtime_filters = 0;
+                return nullptr;
+            }
+        }
+    }
+    return result.release();
+}
+
+bool PaimonRustPredicateConverter::_is_safe_to_push(const VExprSPtr& expr) {
+    if (expr->is_safe_to_execute_on_selected_rows()) {
+        return true;
+    }
+    // VectorizedFnCall::is_safe_to_execute_on_selected_rows() admits a fixed
+    // whitelist that does not include `like`, so a like conjunct would always
+    // stop the safe prefix here and never reach _convert_like. A like call
+    // whose children are themselves safe is just as total as the whitelisted
+    // comparisons: it only compares strings, and its regex is built by escaping
+    // the pattern operand (FunctionLike::convert_like_pattern), so no scanned
+    // value can make it raise — a malformed pattern or escape operand fails
+    // at open() on every execution, regardless of which rows survive. Admit
+    // it here rather than widening the generic whitelist, which gates shared
+    // BE pushdown paths outside this reader's scope.
+    auto* fn = dynamic_cast<VectorizedFnCall*>(expr.get());
+    if (fn == nullptr || _normalize_name(fn->function_name()) != "like") {
+        return false;
+    }
+    for (uint16_t i = 0; i < fn->get_num_children(); ++i) {
+        if (!fn->get_child(i)->is_safe_to_execute_on_selected_rows()) {
+            return false;
+        }
+    }
+    return true;
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_convert_expr(const VExprSPtr& expr) {
+    if (!expr) {
+        return nullptr;
+    }
+
+    // Casts are not unwrapped anywhere (predicate root included): a cast node
+    // fails every dispatch below and the conjunct stays in the Doris residual,
+    // mirroring the FE converter, which keeps casted expressions unconverted.
+    if (auto* direct_in = dynamic_cast<VDirectInPredicate*>(expr.get())) {
+        VExprSPtr in_expr;
+        if (direct_in->get_slot_in_expr(in_expr)) {
+            return _convert_in(in_expr);
+        }
+        return nullptr;
+    }
+
+    if (dynamic_cast<VInPredicate*>(expr.get()) != nullptr) {
+        return _convert_in(expr);
+    }
+
+    switch (expr->op()) {
+    case TExprOpcode::COMPOUND_AND:
+    case TExprOpcode::COMPOUND_OR:
+        return _convert_compound(expr);
+    case TExprOpcode::COMPOUND_NOT:
+        return nullptr;
+    case TExprOpcode::EQ:
+    case TExprOpcode::EQ_FOR_NULL:
+    case TExprOpcode::NE:
+    case TExprOpcode::GE:
+    case TExprOpcode::GT:
+    case TExprOpcode::LE:
+    case TExprOpcode::LT:
+        return _convert_binary(expr);
+    default:
+        break;
+    }
+
+    if (auto* fn = dynamic_cast<VectorizedFnCall*>(expr.get())) {
+        auto fn_name = _normalize_name(fn->function_name());
+        if (fn_name == "is_null_pred" || fn_name == "is_not_null_pred") {
+            return _convert_is_null(expr, fn_name);
+        }
+        if (fn_name == "like") {
+            return _convert_like(expr);
+        }
+    }
+
+    return nullptr;
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_convert_compound(const VExprSPtr& expr) {
+    if (!expr || expr->get_num_children() != 2) {
+        return nullptr;
+    }
+    predicate_ptr left(_convert_expr(expr->get_child(0)));
+    if (!left) {
+        return nullptr;
+    }
+    predicate_ptr right(_convert_expr(expr->get_child(1)));
+    if (!right) {
+        return nullptr;
+    }
+
+    if (expr->op() == TExprOpcode::COMPOUND_AND) {
+        return paimon_predicate_and(left.release(), right.release());
+    }
+    if (expr->op() == TExprOpcode::COMPOUND_OR) {
+        return paimon_predicate_or(left.release(), right.release());
+    }
+    return nullptr;
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_convert_in(const VExprSPtr& expr) {
+    auto* in_pred = dynamic_cast<VInPredicate*>(expr.get());
+    // Runtime-generated IN sets can exceed VExpr's 16-bit child count. A wrapped
+    // count would push only a prefix and permanently discard valid probe rows.
+    if (!in_pred || expr->children().size() < 2 ||
+        expr->children().size() > std::numeric_limits<uint16_t>::max()) {
+        return nullptr;
+    }
+    auto field_meta = _resolve_field(expr->get_child(0));
+    if (!field_meta) {
+        return nullptr;
+    }
+
+    const auto num_values = expr->get_num_children() - 1;
+    // Reserve up front so the backing strings never reallocate: each datum's
+    // str_data points into storages[i], which must stay stable.
+    std::vector<std::string> storages;
+    std::vector<paimon_datum> datums;
+    storages.reserve(num_values);
+    datums.reserve(num_values);
+    for (uint16_t i = 1; i < expr->get_num_children(); ++i) {
+        // Casted list values are rejected by _convert_literal (the same rule
+        // as the binary RHS): with debug_skip_fold_constant the cast reaches
+        // the BE un-folded, and unwrapping it would filter on the pre-cast
+        // value — in `amount IN (CAST(1.24 AS DECIMAL(10,1)))` Doris keeps
+        // the 1.2 rows while the unwrapped 1.24 push removes them. Rejecting
+        // the value rejects the whole predicate; the residual applies the
+        // cast correctly.
+        auto holder = _convert_literal(expr->get_child(i), field_meta->type);
+        if (!holder) {
+            return nullptr;
+        }
+        storages.emplace_back(std::move(holder->storage));
+        paimon_datum datum = holder->datum;
+        _bind_datum_storage(&datum, storages.back());
+        datums.emplace_back(datum);
+    }
+
+    if (datums.empty()) {
+        return nullptr;
+    }
+    if (in_pred->is_not_in()) {
+        return _take(paimon_predicate_is_not_in(_table, field_meta->column.c_str(), datums.data(),
+                                                datums.size()));
+    }
+    return _take(paimon_predicate_is_in(_table, field_meta->column.c_str(), datums.data(),
+                                        datums.size()));
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_convert_binary(const VExprSPtr& expr) {
+    if (!expr || expr->get_num_children() != 2) {
+        return nullptr;
+    }
+    auto field_meta = _resolve_field(expr->get_child(0));
+    if (!field_meta) {
+        return nullptr;
+    }
+    const char* column = field_meta->column.c_str();
+
+    // Convert the RHS first so EQ_FOR_NULL (<=>) only converts when the RHS is
+    // a convertible literal, mirroring the FE converter, which rejects a
+    // non-literal RHS. A column-to-column `a <=> b` must therefore stay in the
+    // Doris residual: it has no single-column rust predicate, and pushing
+    // `a IS NULL` would wrongly discard rows like (1, 1) — rows dropped by the
+    // pushed filter cannot be recovered by the residual conjunct.
+    auto holder = _convert_literal(expr->get_child(1), field_meta->type);
+    if (!holder) {
+        return nullptr;
+    }
+
+    if (expr->op() == TExprOpcode::EQ_FOR_NULL) {
+        return _take(paimon_predicate_is_null(_table, column));

Review Comment:
   [P1] Preserve non-null null-safe equality when pushing this predicate. With `disable_nereids_expression_rules='NULL_SAFE_EQUAL_TO_EQUAL'`, `WHERE id <=> 1` reaches this converter with a non-null literal. It currently pushes `id IS NULL`, so Rust removes every matching `id=1` row before Doris applies the residual and the query returns no rows. Push equality for a non-null literal, reserving `IS NULL` for a null literal, or leave the expression as a residual.



##########
be/src/format_v2/table/paimon_rust_table_reader.cpp:
##########
@@ -0,0 +1,924 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn)                \
+    struct type##_deleter {                       \
+        void operator()(paimon_##type* p) const { \
+            if (p) {                              \
+                freefn(p);                        \
+            }                                     \
+        }                                         \
+    };                                            \
+    using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+    explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+    ~ArrowBatch() {
+        auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+        auto* array = static_cast<ArrowArray*>(batch_.array);
+        if (array && array->release) {
+            array->release(array);
+        }
+        if (schema && schema->release) {
+            schema->release(schema);
+        }
+        paimon_arrow_batch_free(batch_);
+    }
+
+    ArrowBatch(const ArrowBatch&) = delete;
+    ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+    ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+    ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+    paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+    error_ptr owned(err);
+    if (!owned) {
+        return "unknown error";
+    }
+    std::string msg;
+    if (owned->message.data != nullptr && owned->message.len > 0) {
+        msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+    }
+    return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+    std::string out;
+    for (const auto& kv : options) {
+        if (!out.empty()) {
+            out += ", ";
+        }
+        out += kv.first;
+    }
+    return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+    table_ptr table;
+    read_builder_ptr read_builder;
+    plan_ptr plan;
+    table_read_ptr table_read;
+    record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+    RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+    {
+        // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+        // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.init_timer);
+        // Materialize TIMESTAMP_LTZ in the session timezone — the same
+        // convention as the JNI reader (PaimonJniScanner reads time_zone from
+        // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+        // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+        // of _ctz, so NTZ wall-clock semantics are preserved.
+        DORIS_CHECK(_runtime_state != nullptr);
+        _ctz = _runtime_state->timezone_obj();
+        if (_scanner_profile != nullptr) {
+            file_scan_profile::ensure_hierarchy(_scanner_profile);
+            _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+                                               file_scan_profile::TABLE_READER);
+            _rust_open_split_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+            _rust_read_batch_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+            _rust_arrow_to_block_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+            _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+                                                       TUnit::UNIT, "PaimonRustReader");
+            _rust_predicates_converted = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+            _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+                                                         TUnit::UNIT, "PaimonRustReader");
+            _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+            _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+        }
+        // Projected column name -> fixed output position, registered with both the exact and
+        // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+        // semantics: exact match first, lower-case fallback on lookup).
+        _output_name_to_idx.reserve(_projected_columns.size() * 2);
+        for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+            _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+            _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+        }
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+    // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+    // are idempotent, and clear it only when a new split is explicitly prepared.
+    _close_split_reader();
+    _split_eof = false;
+    _current_range = options.current_range;
+    RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+    if (current_split_pruned()) {
+        return Status::OK();
+    }
+    if (_is_table_level_count_active()) {
+        // No rust pipeline is opened; get_block emits the synthetic count rows.
+        return Status::OK();
+    }
+    RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+    {
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.prepare_split_timer);
+        // Open may dominate the scan or fail before get_block; include it in the parent timer.
+        SCOPED_TIMER(_rust_total_time);
+        SCOPED_TIMER(_rust_open_split_time);
+        RETURN_IF_ERROR(_open_split_reader(options.current_range));
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+    SCOPED_TIMER(_profile.total_timer);
+    SCOPED_TIMER(_profile.exec_timer);
+    SCOPED_TIMER(_rust_total_time);
+    DORIS_CHECK(block != nullptr);
+    DORIS_CHECK(eos != nullptr);
+    DORIS_CHECK(block->columns() == _projected_columns.size());
+    block->clear_column_data(_projected_columns.size());
+    *eos = false;
+
+    if (_is_table_level_count_active()) {
+        return _read_table_level_count(block, eos);
+    }
+
+    // num_splits == 0 yields an empty (but valid) stream: report EOF.
+    if (_split_eof) {
+        *eos = true;
+        return Status::OK();
+    }
+    if (!_handles || !_handles->reader) {
+        return Status::InternalError("paimon-rust reader is not initialized");
+    }
+
+    while (true) {
+        // Mirror the base TableReader cancellation contract so a cancelled query does not
+        // drain the whole split.
+        if (_io_ctx != nullptr && _io_ctx->should_stop) {
+            _split_eof = true;
+            _close_split_reader();
+            *eos = true;
+            return Status::OK();
+        }
+
+        paimon_result_next_batch next;
+        {
+            SCOPED_TIMER(_rust_read_batch_time);
+            next = paimon_record_batch_reader_next(_handles->reader.get());
+        }
+        if (next.error != nullptr) {
+            return Status::InternalError("paimon-rust read batch failed: {}",
+                                         consume_error(next.error));
+        }
+        // End of stream: both pointers are null.
+        if (next.batch.array == nullptr && next.batch.schema == nullptr) {
+            _split_eof = true;
+            _close_split_reader();
+            *eos = true;
+            return Status::OK();
+        }
+
+        // RAII: the batch's Arrow release callbacks + container free run when
+        // `batch` leaves this scope, including on any early return.
+        ArrowBatch batch(next.batch);
+
+        auto* c_array = batch.array();
+        auto* c_schema = batch.schema();
+        arrow::Result<std::shared_ptr<arrow::RecordBatch>> import_result =
+                arrow::ImportRecordBatch(c_array, c_schema);
+        if (!import_result.ok()) {
+            return Status::InternalError("failed to import paimon-rust arrow batch: {}",
+                                         import_result.status().message());
+        }
+
+        auto record_batch = std::move(import_result).ValueUnsafe();
+        const auto rows = static_cast<size_t>(record_batch->num_rows());
+        if (rows == 0) {
+            // Skip empty batches and keep draining the stream.
+            continue;
+        }
+        RETURN_IF_ERROR(_fill_block_from_record_batch(record_batch, block, rows));
+        _record_scan_rows(rows);
+        *eos = false;
+        return Status::OK();
+    }
+}
+
+Status PaimonRustTableReader::abort_split() {
+    {
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.close_timer);
+        _close_split_reader();
+        _split_eof = false;
+    }
+    return format::TableReader::abort_split();
+}
+
+#ifdef BE_TEST
+std::string PaimonRustTableReader::TEST_format_options(
+        const std::map<std::string, std::string>& options) {
+    return format_options(options);
+}
+
+std::map<std::string, std::string> PaimonRustTableReader::TEST_build_options(
+        TFileScanRangeParams* scan_params, const TFileRangeDesc& range) {
+    TFileScanRangeParams* previous_params = _scan_params;
+    TFileRangeDesc previous_range = _current_range;
+    _scan_params = scan_params;
+    _current_range = range;
+    std::map<std::string, std::string> options = _build_options();
+    _scan_params = previous_params;
+    _current_range = std::move(previous_range);
+    return options;
+}
+#endif
+
+Status PaimonRustTableReader::close() {
+    {
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.close_timer);
+        _close_split_reader();
+        _close_table();
+    }
+    return format::TableReader::close();
+}
+
+Status PaimonRustTableReader::_validate_rust_split(const TFileRangeDesc& range) const {
+    if (!range.__isset.table_format_params || !range.table_format_params.__isset.paimon_params) {
+        return Status::InternalError(
+                "missing paimon_params for paimon rust reader, possibly caused by FE/BE protocol "
+                "mismatch");
+    }
+    const auto& params = range.table_format_params.paimon_params;
+    if (!params.__isset.paimon_split || params.paimon_split.empty()) {
+        return Status::InternalError(
+                "missing paimon_split for paimon rust reader, possibly caused by FE/BE protocol "
+                "mismatch");
+    }
+    if (params.__isset.reader_type && params.reader_type != TPaimonReaderType::PAIMON_RUST) {
+        return Status::InternalError(
+                "invalid reader_type for paimon rust reader, possibly caused by FE/BE protocol "
+                "mismatch");
+    }
+    if (!_resolve_table_path(range).has_value()) {
+        return Status::InternalError(
+                "paimon-rust missing paimon_table; cannot resolve paimon table location");
+    }
+    if (!_resolve_db_name(range).has_value()) {
+        return Status::InternalError(
+                "paimon-rust missing db_name; cannot open paimon table via schema json");
+    }
+    if (!_resolve_table_name(range).has_value()) {
+        return Status::InternalError(
+                "paimon-rust missing table_name; cannot open paimon table via schema json");
+    }
+    if (!_resolve_table_schema_json(range).has_value()) {
+        return Status::InternalError(
+                "paimon-rust missing paimon_table_schema_json; cannot open paimon table via "
+                "schema json");
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::_open_split_reader(const TFileRangeDesc& range) {
+    // 1. Decode the FE-planned split first so we fail fast (and without any
+    // filesystem IO) when it is missing or malformed.
+    std::string split_bytes;
+    RETURN_IF_ERROR(_decode_split_bytes(&split_bytes));
+
+    // 2. Resolve identifier + table_path + FE-supplied TableSchema JSON.
+    auto table_path = _resolve_table_path(range).value();
+    auto db_name = _resolve_db_name(range).value();
+    auto table_name = _resolve_table_name(range).value();
+    auto schema_json = _resolve_table_schema_json(range).value();
+    auto branch_opt = _resolve_branch(range);
+
+    // 3. Assemble storage options: FE-supplied paimon options + hadoop_conf +
+    // OSS/S3 → AWS_* translations. These feed FileIO only (per
+    // paimon_table_from_schema_json contract); they are NOT merged into the
+    // supplied table schema.
+    auto options = _build_options();
+
+    auto opened_table_key =
+            std::make_tuple(table_path, schema_json, db_name, table_name, branch_opt, options);
+    if (!_handles || !_handles->table || _opened_table_key != opened_table_key) {
+        // A paimon scan reads one table, so the handle is opened at most once per
+        // distinct identity (e.g. re-created after a close); splits of the same
+        // table reuse it and only rebuild the read pipeline below.
+        _close_table();
+        _handles = std::make_unique<PaimonHandles>();
+
+        std::vector<paimon_option> c_options;
+        c_options.reserve(options.size());
+        for (const auto& kv : options) {
+            c_options.push_back(paimon_option {kv.first.c_str(), kv.second.c_str()});
+        }
+
+        LOG(INFO) << "paimon-rust opening table via schema json: db=" << db_name
+                  << " table=" << table_name << " path=" << table_path
+                  << " branch=" << (branch_opt.has_value() ? branch_opt.value() : "main")
+                  << " storage_options=[" << format_options(options) << "]";
+
+        // Build the table directly from the FE-supplied schema JSON. The Rust
+        // side rejects null / empty branch, so we default to paimon's canonical
+        // "main" sentinel when FE did not set paimon_branch (i.e. the table is
+        // on the main branch — matches upstream Identifier.DEFAULT_MAIN_BRANCH).
+        const std::string& branch_str = branch_opt.has_value() ? branch_opt.value() : "main";
+        paimon_result_get_table tbl_res = paimon_table_from_schema_json(
+                table_path.c_str(), schema_json.c_str(), db_name.c_str(), table_name.c_str(),
+                branch_str.c_str(), c_options.empty() ? nullptr : c_options.data(),
+                c_options.size());
+        if (tbl_res.error != nullptr) {
+            return Status::InternalError(
+                    "paimon-rust table_from_schema_json failed: db={} table={} err={}", db_name,
+                    table_name, consume_error(tbl_res.error));
+        }
+        _handles->table.reset(tbl_res.table);
+        _opened_table_key = std::move(opened_table_key);
+    }
+
+    // 4. Build the read pipeline: read_builder -> case-insensitive -> projection.
+    // Bound the Rust Arrow allocation itself: wide rows must respect the scanner's
+    // adaptive probe size before they are materialized into a Doris block.
+    const auto batch_size = std::to_string(
+            _batch_size > 0 ? _batch_size : std::max(1, _runtime_state->batch_size()));
+    const paimon_option batch_option {"read.batch-size", batch_size.c_str()};
+    paimon_result_read_builder rb_res =
+            paimon_table_new_read_builder_with_options(_handles->table.get(), &batch_option, 1);
+    if (rb_res.error != nullptr) {
+        return Status::InternalError("paimon-rust new read builder failed: {}",
+                                     consume_error(rb_res.error));
+    }
+    _handles->read_builder.reset(rb_res.read_builder);
+
+    // Fold column casing on the Rust side so FE-normalized lowercase names
+    // resolve against tables with mixed-case column definitions.
+    if (paimon_error* case_err =
+                paimon_read_builder_with_case_sensitive(_handles->read_builder.get(), false)) {
+        return Status::InternalError("paimon-rust set case_sensitive failed: {}",
+                                     consume_error(case_err));
+    }
+
+    // Partition keys are excluded: they are materialized from split metadata
+    // (see _fill_non_arrow_columns), and paimon-rust does not emit them.
+    auto read_columns = _build_read_columns();
+    std::vector<const char*> projection;
+    projection.reserve(read_columns.size() + 1);
+    for (const auto& col : read_columns) {
+        projection.push_back(col.c_str());
+    }
+    projection.push_back(nullptr);
+    if (paimon_error* proj_err = paimon_read_builder_with_projection(_handles->read_builder.get(),
+                                                                     projection.data())) {
+        return Status::InternalError("paimon-rust set projection failed: {}",
+                                     consume_error(proj_err));
+    }
+
+    // Convert the scanner conjuncts into a paimon-rust filter and apply it.
+    RETURN_IF_ERROR(_apply_predicate());
+
+    // 5. Deserialize the FE-planned split into a one-split plan, so this
+    // scanner reads exactly the split it was assigned rather than replanning
+    // the whole table. The wire form is identical to what paimon-cpp consumes
+    // (`paimon::table::DataSplit::serialize`).
+    paimon_result_plan plan_res = paimon_plan_from_split_bytes(
+            reinterpret_cast<const uint8_t*>(split_bytes.data()), split_bytes.size());
+    if (plan_res.error != nullptr) {
+        return Status::InternalError("paimon-rust build plan failed: {}",
+                                     consume_error(plan_res.error));
+    }
+    _handles->plan.reset(plan_res.plan);
+
+    size_t num_splits = paimon_plan_num_splits(_handles->plan.get());
+    if (num_splits == 0) {
+        _split_eof = true;
+        return Status::OK();
+    }
+
+    // 6. Open the arrow stream over the plan.
+    paimon_result_new_read read_res = paimon_read_builder_new_read(_handles->read_builder.get());
+    if (read_res.error != nullptr) {
+        return Status::InternalError("paimon-rust new read failed: {}",
+                                     consume_error(read_res.error));
+    }
+    _handles->table_read.reset(read_res.read);
+
+    paimon_result_record_batch_reader rdr_res = paimon_table_read_to_arrow(
+            _handles->table_read.get(), _handles->plan.get(), /*offset=*/0, /*length=*/num_splits);
+    if (rdr_res.error != nullptr) {
+        return Status::InternalError("paimon-rust open arrow reader failed: {}",
+                                     consume_error(rdr_res.error));
+    }
+    _handles->reader.reset(rdr_res.reader);
+    return Status::OK();
+}
+
+void PaimonRustTableReader::_close_split_reader() {
+    if (!_handles) {
+        return;
+    }
+    // Reverse of the declaration order in PaimonHandles.
+    _handles->reader.reset();
+    _handles->table_read.reset();
+    _handles->plan.reset();
+    _handles->read_builder.reset();
+}
+
+void PaimonRustTableReader::_close_table() {
+    if (!_handles) {
+        return;
+    }
+    _close_split_reader();
+    _handles->table.reset();
+    _opened_table_key.reset();
+}
+
+Status PaimonRustTableReader::_apply_predicate() {
+    if (_conjuncts.empty() || !_handles || !_handles->table || !_handles->read_builder) {
+        return Status::OK();
+    }
+    if (_scanner_profile != nullptr) {
+        COUNTER_UPDATE(_rust_predicates_input, _conjuncts.size());
+        for (const auto& conjunct : _conjuncts) {
+            if (conjunct && conjunct->root() && conjunct->root()->is_rf_wrapper()) {
+                COUNTER_UPDATE(_rust_runtime_filters_input, 1);
+            }
+        }
+    }
+    LOG(INFO) << "paimon-rust predicate pushdown: " << _conjuncts.size() << " conjunct(s) input";

Review Comment:
   [P2] Avoid INFO logging for every filtered split. `_apply_predicate` runs once per Rust logical split and emits this line plus an applied/none line whenever conjuncts exist. A 50,000-split scan therefore writes about 100,000 INFO records, adding avoidable log I/O and rotation pressure to a hot scan path. Put per-split details behind verbose logging or use the existing predicate profile counters for the aggregate result.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68825:
URL: https://github.com/apache/doris/pull/68825#discussion_r4237034237


##########
gensrc/thrift/PlanNodes.thrift:
##########
@@ -398,6 +399,11 @@ struct TPaimonFileDesc {
     // Original Paimon RawFile.path() before Doris storage path normalization. Native readers use this
     // to materialize the public file-location metadata column.
     18: optional string original_file_path;
+    // paimon-rust: TableSchema serialized via JsonSerdeUtil.toJson(table.schema()).
+    // Consumed by paimon_table_from_schema_json on BE; skips catalog/warehouse resolution.
+    19: optional string paimon_table_schema_json;

Review Comment:
   [P1] Keep Paimon Rust payload IDs compatible across release branches. Branches 4.1 and 4.2 use field 18 for schema JSON and field 19 for branch, whereas this new field 19 is schema JSON and field 20 is branch. Those branches already advertise Rust capability, so a mixed-version FE/BE pair can select Rust and decode the schema as a branch, or find no schema at all, causing scans to fail. Master already used field 18 for `original_file_path` before this PR, so a compatible versioned payload/capability or equivalent migration is needed; merely renumbering that existing field would break master-side compatibility.



##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -991,16 +1009,24 @@ && hasTrustworthyLimitAccounting(table)
                             dataSplit.bucket()));
                 }
             } else {
-                // JNI reader path
+                // Logical reader path: use paimon-rust only for the compatibility-checked subset.
                 if (ignoreJni) {
                     // FIX-L14: ignore_split_type=IGNORE_JNI drops JNI splits (legacy getSplits:483).
                     continue;
                 }
                 if (requiresMetadataColumns) {
                     validateMetadataColumnReader(true, false);
                 }
-                ranges.add(buildJniScanRange(dataSplit, defaultFileFormat,
-                        partitionValues, true, weightDenominator));
+                if (rustReaderSelector != null && rustReaderSelector.canRead(dataSplit)) {

Review Comment:
   [P2] Preserve the `force_jni_scanner` escape hatch when Rust is enabled. That session variable is documented as forcing JNI for external tables. It disables the native arm, but this logical arm still selects Rust when `enable_paimon_rust_reader=true` and the split is eligible; the added differential test uses exactly that combination. A user setting the force flag to bypass a failing Rust read will still run Rust. Make the override reach the Rust selector, and use a separate logical-split test mechanism if needed.



##########
be/src/format_v2/table/paimon_rust_table_reader.cpp:
##########
@@ -0,0 +1,924 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn)                \
+    struct type##_deleter {                       \
+        void operator()(paimon_##type* p) const { \
+            if (p) {                              \
+                freefn(p);                        \
+            }                                     \
+        }                                         \
+    };                                            \
+    using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+    explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+    ~ArrowBatch() {
+        auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+        auto* array = static_cast<ArrowArray*>(batch_.array);
+        if (array && array->release) {
+            array->release(array);
+        }
+        if (schema && schema->release) {
+            schema->release(schema);
+        }
+        paimon_arrow_batch_free(batch_);
+    }
+
+    ArrowBatch(const ArrowBatch&) = delete;
+    ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+    ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+    ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+    paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+    error_ptr owned(err);
+    if (!owned) {
+        return "unknown error";
+    }
+    std::string msg;
+    if (owned->message.data != nullptr && owned->message.len > 0) {
+        msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+    }
+    return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+    std::string out;
+    for (const auto& kv : options) {
+        if (!out.empty()) {
+            out += ", ";
+        }
+        out += kv.first;
+    }
+    return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+    table_ptr table;
+    read_builder_ptr read_builder;
+    plan_ptr plan;
+    table_read_ptr table_read;
+    record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+    RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+    {
+        // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+        // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.init_timer);
+        // Materialize TIMESTAMP_LTZ in the session timezone — the same
+        // convention as the JNI reader (PaimonJniScanner reads time_zone from
+        // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+        // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+        // of _ctz, so NTZ wall-clock semantics are preserved.
+        DORIS_CHECK(_runtime_state != nullptr);
+        _ctz = _runtime_state->timezone_obj();
+        if (_scanner_profile != nullptr) {
+            file_scan_profile::ensure_hierarchy(_scanner_profile);
+            _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+                                               file_scan_profile::TABLE_READER);
+            _rust_open_split_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+            _rust_read_batch_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+            _rust_arrow_to_block_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+            _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+                                                       TUnit::UNIT, "PaimonRustReader");
+            _rust_predicates_converted = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+            _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+                                                         TUnit::UNIT, "PaimonRustReader");
+            _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+            _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+        }
+        // Projected column name -> fixed output position, registered with both the exact and
+        // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+        // semantics: exact match first, lower-case fallback on lookup).
+        _output_name_to_idx.reserve(_projected_columns.size() * 2);
+        for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+            _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+            _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+        }
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+    // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+    // are idempotent, and clear it only when a new split is explicitly prepared.
+    _close_split_reader();
+    _split_eof = false;
+    _current_range = options.current_range;
+    RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+    if (current_split_pruned()) {
+        return Status::OK();
+    }
+    if (_is_table_level_count_active()) {
+        // No rust pipeline is opened; get_block emits the synthetic count rows.
+        return Status::OK();
+    }
+    RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+    {
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.prepare_split_timer);
+        // Open may dominate the scan or fail before get_block; include it in the parent timer.
+        SCOPED_TIMER(_rust_total_time);
+        SCOPED_TIMER(_rust_open_split_time);
+        RETURN_IF_ERROR(_open_split_reader(options.current_range));
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+    SCOPED_TIMER(_profile.total_timer);
+    SCOPED_TIMER(_profile.exec_timer);
+    SCOPED_TIMER(_rust_total_time);
+    DORIS_CHECK(block != nullptr);
+    DORIS_CHECK(eos != nullptr);
+    DORIS_CHECK(block->columns() == _projected_columns.size());
+    block->clear_column_data(_projected_columns.size());
+    *eos = false;
+
+    if (_is_table_level_count_active()) {
+        return _read_table_level_count(block, eos);
+    }
+
+    // num_splits == 0 yields an empty (but valid) stream: report EOF.
+    if (_split_eof) {
+        *eos = true;
+        return Status::OK();
+    }
+    if (!_handles || !_handles->reader) {
+        return Status::InternalError("paimon-rust reader is not initialized");
+    }
+
+    while (true) {
+        // Mirror the base TableReader cancellation contract so a cancelled query does not
+        // drain the whole split.
+        if (_io_ctx != nullptr && _io_ctx->should_stop) {
+            _split_eof = true;
+            _close_split_reader();
+            *eos = true;
+            return Status::OK();
+        }
+
+        paimon_result_next_batch next;
+        {
+            SCOPED_TIMER(_rust_read_batch_time);
+            next = paimon_record_batch_reader_next(_handles->reader.get());
+        }
+        if (next.error != nullptr) {
+            return Status::InternalError("paimon-rust read batch failed: {}",
+                                         consume_error(next.error));
+        }
+        // End of stream: both pointers are null.
+        if (next.batch.array == nullptr && next.batch.schema == nullptr) {
+            _split_eof = true;
+            _close_split_reader();
+            *eos = true;
+            return Status::OK();
+        }
+
+        // RAII: the batch's Arrow release callbacks + container free run when
+        // `batch` leaves this scope, including on any early return.
+        ArrowBatch batch(next.batch);
+
+        auto* c_array = batch.array();
+        auto* c_schema = batch.schema();
+        arrow::Result<std::shared_ptr<arrow::RecordBatch>> import_result =
+                arrow::ImportRecordBatch(c_array, c_schema);
+        if (!import_result.ok()) {
+            return Status::InternalError("failed to import paimon-rust arrow batch: {}",
+                                         import_result.status().message());
+        }
+
+        auto record_batch = std::move(import_result).ValueUnsafe();
+        const auto rows = static_cast<size_t>(record_batch->num_rows());
+        if (rows == 0) {
+            // Skip empty batches and keep draining the stream.
+            continue;
+        }
+        RETURN_IF_ERROR(_fill_block_from_record_batch(record_batch, block, rows));
+        _record_scan_rows(rows);
+        *eos = false;
+        return Status::OK();
+    }
+}
+
+Status PaimonRustTableReader::abort_split() {
+    {
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.close_timer);
+        _close_split_reader();
+        _split_eof = false;
+    }
+    return format::TableReader::abort_split();
+}
+
+#ifdef BE_TEST
+std::string PaimonRustTableReader::TEST_format_options(
+        const std::map<std::string, std::string>& options) {
+    return format_options(options);
+}
+
+std::map<std::string, std::string> PaimonRustTableReader::TEST_build_options(
+        TFileScanRangeParams* scan_params, const TFileRangeDesc& range) {
+    TFileScanRangeParams* previous_params = _scan_params;
+    TFileRangeDesc previous_range = _current_range;
+    _scan_params = scan_params;
+    _current_range = range;
+    std::map<std::string, std::string> options = _build_options();
+    _scan_params = previous_params;
+    _current_range = std::move(previous_range);
+    return options;
+}
+#endif
+
+Status PaimonRustTableReader::close() {
+    {
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.close_timer);
+        _close_split_reader();
+        _close_table();
+    }
+    return format::TableReader::close();
+}
+
+Status PaimonRustTableReader::_validate_rust_split(const TFileRangeDesc& range) const {
+    if (!range.__isset.table_format_params || !range.table_format_params.__isset.paimon_params) {
+        return Status::InternalError(
+                "missing paimon_params for paimon rust reader, possibly caused by FE/BE protocol "
+                "mismatch");
+    }
+    const auto& params = range.table_format_params.paimon_params;
+    if (!params.__isset.paimon_split || params.paimon_split.empty()) {
+        return Status::InternalError(
+                "missing paimon_split for paimon rust reader, possibly caused by FE/BE protocol "
+                "mismatch");
+    }
+    if (params.__isset.reader_type && params.reader_type != TPaimonReaderType::PAIMON_RUST) {
+        return Status::InternalError(
+                "invalid reader_type for paimon rust reader, possibly caused by FE/BE protocol "
+                "mismatch");
+    }
+    if (!_resolve_table_path(range).has_value()) {
+        return Status::InternalError(
+                "paimon-rust missing paimon_table; cannot resolve paimon table location");
+    }
+    if (!_resolve_db_name(range).has_value()) {
+        return Status::InternalError(
+                "paimon-rust missing db_name; cannot open paimon table via schema json");
+    }
+    if (!_resolve_table_name(range).has_value()) {
+        return Status::InternalError(
+                "paimon-rust missing table_name; cannot open paimon table via schema json");
+    }
+    if (!_resolve_table_schema_json(range).has_value()) {
+        return Status::InternalError(
+                "paimon-rust missing paimon_table_schema_json; cannot open paimon table via "
+                "schema json");
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::_open_split_reader(const TFileRangeDesc& range) {
+    // 1. Decode the FE-planned split first so we fail fast (and without any
+    // filesystem IO) when it is missing or malformed.
+    std::string split_bytes;
+    RETURN_IF_ERROR(_decode_split_bytes(&split_bytes));
+
+    // 2. Resolve identifier + table_path + FE-supplied TableSchema JSON.
+    auto table_path = _resolve_table_path(range).value();
+    auto db_name = _resolve_db_name(range).value();
+    auto table_name = _resolve_table_name(range).value();
+    auto schema_json = _resolve_table_schema_json(range).value();
+    auto branch_opt = _resolve_branch(range);
+
+    // 3. Assemble storage options: FE-supplied paimon options + hadoop_conf +
+    // OSS/S3 → AWS_* translations. These feed FileIO only (per
+    // paimon_table_from_schema_json contract); they are NOT merged into the
+    // supplied table schema.
+    auto options = _build_options();
+
+    auto opened_table_key =
+            std::make_tuple(table_path, schema_json, db_name, table_name, branch_opt, options);
+    if (!_handles || !_handles->table || _opened_table_key != opened_table_key) {
+        // A paimon scan reads one table, so the handle is opened at most once per
+        // distinct identity (e.g. re-created after a close); splits of the same
+        // table reuse it and only rebuild the read pipeline below.
+        _close_table();
+        _handles = std::make_unique<PaimonHandles>();
+
+        std::vector<paimon_option> c_options;
+        c_options.reserve(options.size());
+        for (const auto& kv : options) {
+            c_options.push_back(paimon_option {kv.first.c_str(), kv.second.c_str()});
+        }
+
+        LOG(INFO) << "paimon-rust opening table via schema json: db=" << db_name
+                  << " table=" << table_name << " path=" << table_path
+                  << " branch=" << (branch_opt.has_value() ? branch_opt.value() : "main")
+                  << " storage_options=[" << format_options(options) << "]";
+
+        // Build the table directly from the FE-supplied schema JSON. The Rust
+        // side rejects null / empty branch, so we default to paimon's canonical
+        // "main" sentinel when FE did not set paimon_branch (i.e. the table is
+        // on the main branch — matches upstream Identifier.DEFAULT_MAIN_BRANCH).
+        const std::string& branch_str = branch_opt.has_value() ? branch_opt.value() : "main";
+        paimon_result_get_table tbl_res = paimon_table_from_schema_json(
+                table_path.c_str(), schema_json.c_str(), db_name.c_str(), table_name.c_str(),
+                branch_str.c_str(), c_options.empty() ? nullptr : c_options.data(),
+                c_options.size());
+        if (tbl_res.error != nullptr) {
+            return Status::InternalError(
+                    "paimon-rust table_from_schema_json failed: db={} table={} err={}", db_name,
+                    table_name, consume_error(tbl_res.error));
+        }
+        _handles->table.reset(tbl_res.table);
+        _opened_table_key = std::move(opened_table_key);
+    }
+
+    // 4. Build the read pipeline: read_builder -> case-insensitive -> projection.
+    // Bound the Rust Arrow allocation itself: wide rows must respect the scanner's
+    // adaptive probe size before they are materialized into a Doris block.
+    const auto batch_size = std::to_string(
+            _batch_size > 0 ? _batch_size : std::max(1, _runtime_state->batch_size()));
+    const paimon_option batch_option {"read.batch-size", batch_size.c_str()};

Review Comment:
   [P2] Apply adaptive row caps to the active Rust stream. FileScannerV2 seeds 32 rows before `prepare_split`, then learns bytes per row and calls `set_batch_size` before later `get_block` calls. Here `read.batch-size` is fixed when the Rust builder opens, while the inherited setter only changes `_batch_size`. A split with 1 MiB rows therefore keeps emitting roughly 32 MiB Arrow batches after the scanner asks for about eight rows to meet its default 8 MiB target. The added test changes the cap only between splits; please cover and honor a cap change after the first batch.



##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonRustReaderCapabilities.java:
##########
@@ -0,0 +1,281 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.paimon;
+
+import org.apache.doris.connector.spi.handle.ConnectorColumnHandle;
+
+import com.google.common.collect.ImmutableSet;
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.types.ArrayType;
+import org.apache.paimon.types.DataField;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.DataTypeChecks;
+import org.apache.paimon.types.DataTypeRoot;
+import org.apache.paimon.types.DecimalType;
+import org.apache.paimon.types.MapType;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.types.VarCharType;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/** Compatibility checks for the pinned paimon-rust reader, beyond storage capabilities. */
+final class PaimonRustReaderCapabilities {
+    private static final Set<String> SUPPORTED_AGGREGATE_NAMES = ImmutableSet.of(
+            "sum", "product", "min", "max", "last_value", "first_value", "last_non_null_value",
+            "first_non_null_value", "first_not_null_value", "bool_and", "bool_or", "listagg");
+    private final FileStoreTable table;
+    private final TableSchema schema;
+    private final boolean tableCompatible;
+    private final Map<Long, Boolean> compatibleFileSchemas = new ConcurrentHashMap<>();
+
+    PaimonRustReaderCapabilities(FileStoreTable table, List<ConnectorColumnHandle> columns) {
+        this.table = table;
+        this.schema = table.schema();
+        this.tableCompatible = schema != null && hasCompatibleSchemaVersion(schema)
+                && hasFullNestedProjection(columns) && hasCompatibleMergeEngine(schema)
+                && hasCompatibleAggregates(schema);
+    }
+
+    boolean canRead(DataSplit split) {
+        if (!tableCompatible) {
+            return false;
+        }
+        // The pinned merge reader retains losing input batches until an output batch fills.
+        // Neither zero deletes nor a small read.batch-size bounds this across multiple files.
+        if (!schema.primaryKeys().isEmpty() && split.dataFiles().size() > 1) {
+            return false;
+        }
+        for (DataFileMeta file : split.dataFiles()) {
+            // Sort-merge retains consumed batches until it emits enough rows. Retracts can
+            // produce an unbounded zero-output prefix; aggregation also rejects retracts.
+            // Unknown counts must stay on JNI, including old files without this statistic.
+            if (!schema.primaryKeys().isEmpty() && file.deleteRowCount().orElse(-1L) != 0L) {
+                return false;
+            }
+            if (file.schemaId() != schema.id() && !compatibleFileSchemas.computeIfAbsent(

Review Comment:
   [P1] Guard current-schema nanosecond timestamps before Rust selection. Paimon permits `TIMESTAMP(9)` year-3000 values and stores precision >6 as Parquet INT96 ([type range](https://paimon.apache.org/docs/master/concepts/data-types/); [Parquet mapping](https://paimon.apache.org/docs/master/concepts/spec/fileformat/)). The pinned Rust/Parquet reader maps INT96 to signed i64 Arrow nanoseconds, and [`Int96::to_nanos` wraps on overflow](https://github.com/apache/arrow-rs/blob/58.4.0/parquet/src/data_type.rs#L110-L119) beyond year 2262. This `schemaId() != schema.id()` condition skips the only timestamp precision check for current-schema files, so an eligible logical split can return a different date from JNI without error. Gate precision 7–9, including nested leaves, until decoding preserves the full range; add a year-3000 parity case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6185258505

   # BE UT Coverage Report
   Increment line coverage `100.00% (31/31)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/b3e30fa9240264b19d185ee444c6531d4106a8ff_b3e30fa9240264b19d185ee444c6531d4106a8ff/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/b3e30fa9240264b19d185ee444c6531d4106a8ff_b3e30fa9240264b19d185ee444c6531d4106a8ff/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 66.18% (31645/47817) |
   | Line Coverage     | 50.99% (334292/655632) |
   | Region Coverage   | 46.30% (270121/583450) |
   | Branch Coverage   | 47.91% (126796/264643) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6187819730

   # FE Regression Coverage Report
   Increment line coverage `  20.00% (19/95)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6095464254

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6095738470

   # Cloud UT Coverage Report
   Increment line coverage ` ` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/701da299bb5dc73be36da3cf7e3346905c1250bd_701da299bb5dc73be36da3cf7e3346905c1250bd_cloud/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/701da299bb5dc73be36da3cf7e3346905c1250bd_701da299bb5dc73be36da3cf7e3346905c1250bd_cloud/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 77.84% (2079/2671) |
   | Line Coverage     | 66.04% (38197/57840) |
   | Region Coverage   | 53.49% (35798/66920) |
   | Branch Coverage   | 56.95% (11517/20223) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6161889666

   # BE UT Coverage Report
   Increment line coverage `100.00% (31/31)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/13e491640aa8d4bb03224c0190f1c40ec26b9db1_13e491640aa8d4bb03224c0190f1c40ec26b9db1/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/13e491640aa8d4bb03224c0190f1c40ec26b9db1_13e491640aa8d4bb03224c0190f1c40ec26b9db1/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 65.91% (31469/47746) |
   | Line Coverage     | 50.65% (331626/654805) |
   | Region Coverage   | 46.02% (268095/582553) |
   | Branch Coverage   | 47.68% (125945/264159) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6168561428

   # BE UT Coverage Report
   Increment line coverage `100.00% (31/31)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/8d52953c3037b078816cdc14d5695e371b5cfacf_8d52953c3037b078816cdc14d5695e371b5cfacf/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/8d52953c3037b078816cdc14d5695e371b5cfacf_8d52953c3037b078816cdc14d5695e371b5cfacf/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 66.16% (31631/47809) |
   | Line Coverage     | 50.97% (334128/655519) |
   | Region Coverage   | 46.29% (270014/583346) |
   | Branch Coverage   | 47.90% (126753/264600) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on code in PR #68825:
URL: https://github.com/apache/doris/pull/68825#discussion_r4278368880


##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -897,6 +911,16 @@ && hasTrustworthyLimitAccounting(table)
             }
         }
 
+        boolean incrementalRead = pinnedOptions.containsKey("incremental-between")
+                || pinnedOptions.containsKey("incremental-between-timestamp");
+        PaimonRustReaderSelector rustReaderSelector = table instanceof FileStoreTable
+                && !requiresMetadataColumns
+                ? new PaimonRustReaderSelector(session, backendsSupportRust,
+                        (FileStoreTable) table, columns, backendStorageProperties(),

Review Comment:
   Thanks. The original #66227 selector likewise derives Rust eligibility and transport from typed backend storage properties. The raw-option precedence gap is valid, but changing the effective-option transport in this forward-port would diverge from the picked implementation. We will address the effective Paimon FileIO configuration in a focused follow-up and keep this PR aligned with the source. Resolving as deferred, not fixed.



##########
be/src/format_v2/table/paimon_rust_table_reader.cpp:
##########
@@ -0,0 +1,944 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace rust_handles {
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn)                \
+    struct type##_deleter {                       \
+        void operator()(paimon_##type* p) const { \
+            if (p) {                              \
+                freefn(p);                        \
+            }                                     \
+        }                                         \
+    };                                            \
+    using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+} // namespace rust_handles
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+    explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+    ~ArrowBatch() {
+        auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+        auto* array = static_cast<ArrowArray*>(batch_.array);
+        if (array && array->release) {
+            array->release(array);
+        }
+        if (schema && schema->release) {
+            schema->release(schema);
+        }
+        paimon_arrow_batch_free(batch_);
+    }
+
+    ArrowBatch(const ArrowBatch&) = delete;
+    ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+    ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+    ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+    paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+    rust_handles::error_ptr owned(err);
+    if (!owned) {
+        return "unknown error";
+    }
+    std::string msg;
+    if (owned->message.data != nullptr && owned->message.len > 0) {
+        msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+    }
+    return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+    std::string out;
+    for (const auto& kv : options) {
+        if (!out.empty()) {
+            out += ", ";
+        }
+        out += kv.first;
+    }
+    return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+    rust_handles::table_ptr table;
+    rust_handles::read_builder_ptr read_builder;
+    rust_handles::plan_ptr plan;
+    rust_handles::table_read_ptr table_read;
+    rust_handles::record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+    RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+    {
+        // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+        // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.init_timer);
+        // Materialize TIMESTAMP_LTZ in the session timezone — the same
+        // convention as the JNI reader (PaimonJniScanner reads time_zone from
+        // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+        // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+        // of _ctz, so NTZ wall-clock semantics are preserved.
+        DORIS_CHECK(_runtime_state != nullptr);
+        _ctz = _runtime_state->timezone_obj();
+        if (_scanner_profile != nullptr) {
+            file_scan_profile::ensure_hierarchy(_scanner_profile);
+            _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+                                               file_scan_profile::TABLE_READER);
+            _rust_open_split_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+            _rust_read_batch_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+            _rust_arrow_to_block_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+            _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+                                                       TUnit::UNIT, "PaimonRustReader");
+            _rust_predicates_converted = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+            _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+                                                         TUnit::UNIT, "PaimonRustReader");
+            _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+            _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+        }
+        // Projected column name -> fixed output position, registered with both the exact and
+        // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+        // semantics: exact match first, lower-case fallback on lookup).
+        _output_name_to_idx.reserve(_projected_columns.size() * 2);
+        for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+            _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+            _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+        }
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+    // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+    // are idempotent, and clear it only when a new split is explicitly prepared.
+    _close_split_reader();
+    _split_eof = false;
+    _current_range = options.current_range;
+    RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+    if (current_split_pruned()) {
+        return Status::OK();
+    }
+    if (_is_table_level_count_active()) {
+        // No rust pipeline is opened; get_block emits the synthetic count rows.
+        return Status::OK();
+    }
+    RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+    {
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.prepare_split_timer);
+        // Open may dominate the scan or fail before get_block; include it in the parent timer.
+        SCOPED_TIMER(_rust_total_time);
+        SCOPED_TIMER(_rust_open_split_time);
+        RETURN_IF_ERROR(_open_split_reader(options.current_range));
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+    SCOPED_TIMER(_profile.total_timer);
+    SCOPED_TIMER(_profile.exec_timer);
+    SCOPED_TIMER(_rust_total_time);
+    DORIS_CHECK(block != nullptr);
+    DORIS_CHECK(eos != nullptr);
+    DORIS_CHECK(block->columns() == _projected_columns.size());
+    block->clear_column_data(_projected_columns.size());
+    *eos = false;
+
+    if (_is_table_level_count_active()) {
+        return _read_table_level_count(block, eos);
+    }
+
+    // num_splits == 0 yields an empty (but valid) stream: report EOF.
+    if (_split_eof) {
+        *eos = true;
+        return Status::OK();
+    }
+    if (!_handles || !_handles->reader) {
+        return Status::InternalError("paimon-rust reader is not initialized");
+    }
+
+    while (true) {
+        // Mirror the base TableReader cancellation contract so a cancelled query does not
+        // drain the whole split.
+        if (_io_ctx != nullptr && _io_ctx->should_stop) {
+            _split_eof = true;
+            _close_split_reader();
+            *eos = true;
+            return Status::OK();
+        }
+
+        if (!_pending_record_batch) {
+            paimon_result_next_batch next;
+            {
+                SCOPED_TIMER(_rust_read_batch_time);
+                next = paimon_record_batch_reader_next(_handles->reader.get());
+            }
+            if (next.error != nullptr) {
+                return Status::InternalError("paimon-rust read batch failed: {}",
+                                             consume_error(next.error));
+            }
+            // End of stream: both pointers are null.
+            if (next.batch.array == nullptr && next.batch.schema == nullptr) {
+                _split_eof = true;
+                _close_split_reader();
+                *eos = true;
+                return Status::OK();
+            }
+
+            // RAII: the batch's Arrow release callbacks + container free run when
+            // `batch` leaves this scope, including on any early return.
+            ArrowBatch batch(next.batch);
+            arrow::Result<std::shared_ptr<arrow::RecordBatch>> import_result =
+                    arrow::ImportRecordBatch(batch.array(), batch.schema());
+            if (!import_result.ok()) {
+                return Status::InternalError("failed to import paimon-rust arrow batch: {}",
+                                             import_result.status().message());
+            }
+            _pending_record_batch = std::move(import_result).ValueUnsafe();
+            _pending_record_batch_offset = 0;
+            if (_pending_record_batch->num_rows() == 0) {
+                _pending_record_batch.reset();
+                continue;
+            }
+        }
+
+        const size_t batch_size = static_cast<size_t>(
+                _batch_size > 0 ? _batch_size : std::max(1, _runtime_state->batch_size()));
+        const size_t remaining = static_cast<size_t>(_pending_record_batch->num_rows()) -
+                                 _pending_record_batch_offset;
+        const size_t rows = std::min(batch_size, remaining);
+        RETURN_IF_ERROR(_fill_block_from_record_batch(_pending_record_batch, block,
+                                                      _pending_record_batch_offset, rows));
+        _pending_record_batch_offset += rows;
+        if (_pending_record_batch_offset ==
+            static_cast<size_t>(_pending_record_batch->num_rows())) {
+            _pending_record_batch.reset();
+            _pending_record_batch_offset = 0;
+        }
+        _record_scan_rows(rows);
+        *eos = false;
+        return Status::OK();
+    }
+}
+
+Status PaimonRustTableReader::abort_split() {
+    {
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.close_timer);
+        _close_split_reader();
+        _split_eof = false;
+    }
+    return format::TableReader::abort_split();
+}
+
+#ifdef BE_TEST
+std::string PaimonRustTableReader::TEST_format_options(
+        const std::map<std::string, std::string>& options) {
+    return format_options(options);
+}
+
+std::map<std::string, std::string> PaimonRustTableReader::TEST_build_options(
+        TFileScanRangeParams* scan_params, const TFileRangeDesc& range) {
+    TFileScanRangeParams* previous_params = _scan_params;
+    TFileRangeDesc previous_range = _current_range;
+    _scan_params = scan_params;
+    _current_range = range;
+    std::map<std::string, std::string> options = _build_options();
+    _scan_params = previous_params;
+    _current_range = std::move(previous_range);
+    return options;
+}
+#endif
+
+Status PaimonRustTableReader::close() {
+    {
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.close_timer);
+        _close_split_reader();
+        _close_table();
+    }
+    return format::TableReader::close();
+}
+
+Status PaimonRustTableReader::_validate_rust_split(const TFileRangeDesc& range) const {
+    if (!range.__isset.table_format_params || !range.table_format_params.__isset.paimon_params) {
+        return Status::InternalError(
+                "missing paimon_params for paimon rust reader, possibly caused by FE/BE protocol "
+                "mismatch");
+    }
+    const auto& params = range.table_format_params.paimon_params;
+    if (!params.__isset.paimon_split || params.paimon_split.empty()) {
+        return Status::InternalError(
+                "missing paimon_split for paimon rust reader, possibly caused by FE/BE protocol "
+                "mismatch");
+    }
+    if (params.__isset.reader_type && params.reader_type != TPaimonReaderType::PAIMON_RUST) {
+        return Status::InternalError(
+                "invalid reader_type for paimon rust reader, possibly caused by FE/BE protocol "
+                "mismatch");
+    }
+    if (!_resolve_table_path(range).has_value()) {
+        return Status::InternalError(
+                "paimon-rust missing paimon_table; cannot resolve paimon table location");
+    }
+    if (!_resolve_db_name(range).has_value()) {
+        return Status::InternalError(
+                "paimon-rust missing db_name; cannot open paimon table via schema json");
+    }
+    if (!_resolve_table_name(range).has_value()) {
+        return Status::InternalError(
+                "paimon-rust missing table_name; cannot open paimon table via schema json");
+    }
+    if (!_resolve_table_schema_json(range).has_value()) {
+        return Status::InternalError(
+                "paimon-rust missing paimon_table_schema_json; cannot open paimon table via "
+                "schema json");
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::_open_split_reader(const TFileRangeDesc& range) {
+    // 1. Decode the FE-planned split first so we fail fast (and without any
+    // filesystem IO) when it is missing or malformed.
+    std::string split_bytes;
+    RETURN_IF_ERROR(_decode_split_bytes(&split_bytes));
+
+    // 2. Resolve identifier + table_path + FE-supplied TableSchema JSON.
+    auto table_path = _resolve_table_path(range).value();
+    auto db_name = _resolve_db_name(range).value();
+    auto table_name = _resolve_table_name(range).value();
+    auto schema_json = _resolve_table_schema_json(range).value();
+    auto branch_opt = _resolve_branch(range);
+
+    // 3. Assemble storage options: FE-supplied paimon options + hadoop_conf +
+    // OSS/S3 → AWS_* translations. These feed FileIO only (per
+    // paimon_table_from_schema_json contract); they are NOT merged into the
+    // supplied table schema.
+    auto options = _build_options();
+
+    auto opened_table_key =
+            std::make_tuple(table_path, schema_json, db_name, table_name, branch_opt, options);
+    if (!_handles || !_handles->table || _opened_table_key != opened_table_key) {
+        // A paimon scan reads one table, so the handle is opened at most once per
+        // distinct identity (e.g. re-created after a close); splits of the same
+        // table reuse it and only rebuild the read pipeline below.
+        _close_table();
+        _handles = std::make_unique<PaimonHandles>();
+
+        std::vector<paimon_option> c_options;
+        c_options.reserve(options.size());
+        for (const auto& kv : options) {
+            c_options.push_back(paimon_option {kv.first.c_str(), kv.second.c_str()});
+        }
+
+        LOG(INFO) << "paimon-rust opening table via schema json: db=" << db_name
+                  << " table=" << table_name << " path=" << table_path
+                  << " branch=" << (branch_opt.has_value() ? branch_opt.value() : "main")
+                  << " storage_options=[" << format_options(options) << "]";
+
+        // Build the table directly from the FE-supplied schema JSON. The Rust
+        // side rejects null / empty branch, so we default to paimon's canonical
+        // "main" sentinel when FE did not set paimon_branch (i.e. the table is
+        // on the main branch — matches upstream Identifier.DEFAULT_MAIN_BRANCH).
+        const std::string& branch_str = branch_opt.has_value() ? branch_opt.value() : "main";
+        paimon_result_get_table tbl_res = paimon_table_from_schema_json(
+                table_path.c_str(), schema_json.c_str(), db_name.c_str(), table_name.c_str(),
+                branch_str.c_str(), c_options.empty() ? nullptr : c_options.data(),
+                c_options.size());
+        if (tbl_res.error != nullptr) {
+            return Status::InternalError(
+                    "paimon-rust table_from_schema_json failed: db={} table={} err={}", db_name,
+                    table_name, consume_error(tbl_res.error));
+        }
+        _handles->table.reset(tbl_res.table);
+        _opened_table_key = std::move(opened_table_key);
+    }
+
+    // 4. Build the read pipeline: read_builder -> case-insensitive -> projection.
+    // Bound the Rust Arrow allocation itself: wide rows must respect the scanner's
+    // adaptive probe size before they are materialized into a Doris block.
+    const auto batch_size = std::to_string(
+            _batch_size > 0 ? _batch_size : std::max(1, _runtime_state->batch_size()));
+    const paimon_option batch_option {"read.batch-size", batch_size.c_str()};
+    paimon_result_read_builder rb_res =
+            paimon_table_new_read_builder_with_options(_handles->table.get(), &batch_option, 1);
+    if (rb_res.error != nullptr) {
+        return Status::InternalError("paimon-rust new read builder failed: {}",
+                                     consume_error(rb_res.error));
+    }
+    _handles->read_builder.reset(rb_res.read_builder);
+
+    // Fold column casing on the Rust side so FE-normalized lowercase names
+    // resolve against tables with mixed-case column definitions.
+    if (paimon_error* case_err =
+                paimon_read_builder_with_case_sensitive(_handles->read_builder.get(), false)) {
+        return Status::InternalError("paimon-rust set case_sensitive failed: {}",
+                                     consume_error(case_err));
+    }
+
+    // Partition keys are excluded: they are materialized from split metadata
+    // (see _fill_non_arrow_columns), and paimon-rust does not emit them.
+    auto read_columns = _build_read_columns();
+    std::vector<const char*> projection;
+    projection.reserve(read_columns.size() + 1);
+    for (const auto& col : read_columns) {
+        projection.push_back(col.c_str());
+    }
+    projection.push_back(nullptr);
+    if (paimon_error* proj_err = paimon_read_builder_with_projection(_handles->read_builder.get(),
+                                                                     projection.data())) {
+        return Status::InternalError("paimon-rust set projection failed: {}",
+                                     consume_error(proj_err));
+    }
+
+    // Convert the scanner conjuncts into a paimon-rust filter and apply it.
+    RETURN_IF_ERROR(_apply_predicate());
+
+    // 5. Deserialize the FE-planned split into a one-split plan, so this
+    // scanner reads exactly the split it was assigned rather than replanning
+    // the whole table. The wire form is identical to what paimon-cpp consumes
+    // (`paimon::table::DataSplit::serialize`).
+    paimon_result_plan plan_res = paimon_plan_from_split_bytes(
+            reinterpret_cast<const uint8_t*>(split_bytes.data()), split_bytes.size());
+    if (plan_res.error != nullptr) {
+        return Status::InternalError("paimon-rust build plan failed: {}",
+                                     consume_error(plan_res.error));
+    }
+    _handles->plan.reset(plan_res.plan);
+
+    size_t num_splits = paimon_plan_num_splits(_handles->plan.get());
+    if (num_splits == 0) {
+        _split_eof = true;
+        return Status::OK();
+    }
+
+    // 6. Open the arrow stream over the plan.
+    paimon_result_new_read read_res = paimon_read_builder_new_read(_handles->read_builder.get());
+    if (read_res.error != nullptr) {
+        return Status::InternalError("paimon-rust new read failed: {}",
+                                     consume_error(read_res.error));
+    }
+    _handles->table_read.reset(read_res.read);
+
+    paimon_result_record_batch_reader rdr_res = paimon_table_read_to_arrow(
+            _handles->table_read.get(), _handles->plan.get(), /*offset=*/0, /*length=*/num_splits);
+    if (rdr_res.error != nullptr) {
+        return Status::InternalError("paimon-rust open arrow reader failed: {}",
+                                     consume_error(rdr_res.error));
+    }
+    _handles->reader.reset(rdr_res.reader);
+    return Status::OK();
+}
+
+void PaimonRustTableReader::_close_split_reader() {
+    _pending_record_batch.reset();
+    _pending_record_batch_offset = 0;
+    if (!_handles) {
+        return;
+    }
+    // Reverse of the declaration order in PaimonHandles.
+    _handles->reader.reset();
+    _handles->table_read.reset();
+    _handles->plan.reset();
+    _handles->read_builder.reset();
+}
+
+void PaimonRustTableReader::_close_table() {
+    if (!_handles) {
+        return;
+    }
+    _close_split_reader();
+    _handles->table.reset();
+    _opened_table_key.reset();
+}
+
+Status PaimonRustTableReader::_apply_predicate() {
+    if (_conjuncts.empty() || !_handles || !_handles->table || !_handles->read_builder) {
+        return Status::OK();
+    }
+    if (_scanner_profile != nullptr) {
+        COUNTER_UPDATE(_rust_predicates_input, _conjuncts.size());
+        for (const auto& conjunct : _conjuncts) {
+            if (conjunct && conjunct->root() && conjunct->root()->is_rf_wrapper()) {
+                COUNTER_UPDATE(_rust_runtime_filters_input, 1);
+            }
+        }
+    }
+    VLOG_DEBUG << "paimon-rust predicate pushdown: " << _conjuncts.size() << " conjunct(s) input";
+    // The conjunct VSlotRefs carry table global indices (positions), so the v2
+    // converter mode resolves fields by the projected column names; partition
+    // keys are excluded because the rust reader does not read them.
+    std::vector<std::string> names;
+    std::vector<DataTypePtr> types;
+    names.reserve(_projected_columns.size());
+    types.reserve(_projected_columns.size());
+    for (const auto& col : _projected_columns) {
+        if (col.is_partition_key) {
+            continue;
+        }
+        names.push_back(col.name);
+        types.push_back(col.type);
+    }
+    PaimonRustPredicateConverter converter(names, types, _handles->table.get());
+    paimon_predicate* predicate = converter.build(_conjuncts);
+    if (_scanner_profile != nullptr) {
+        COUNTER_UPDATE(_rust_predicates_converted, converter.converted_conjuncts());
+    }
+    if (predicate == nullptr) {
+        VLOG_DEBUG << "paimon-rust predicate pushdown: nothing convertible, no filter applied";
+        return Status::OK();
+    }
+    // paimon_read_builder_with_filter consumes the predicate (ownership moves to
+    // the builder) on every path, so we must not free it here.
+    if (paimon_error* err =
+                paimon_read_builder_with_filter(_handles->read_builder.get(), predicate)) {
+        return Status::InternalError("paimon-rust apply filter failed: {}", consume_error(err));
+    }
+    // Count application only after the C API accepts the filter; reader timers
+    // alone cannot distinguish pushdown from Doris residual-only execution.
+    if (_scanner_profile != nullptr) {
+        COUNTER_UPDATE(_rust_predicates_applied, converter.converted_conjuncts());
+        COUNTER_UPDATE(_rust_runtime_filters_applied, converter.converted_runtime_filters());
+    }
+    VLOG_DEBUG << "paimon-rust predicate pushdown: applied";
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::_fill_block_from_record_batch(
+        const std::shared_ptr<arrow::RecordBatch>& batch, Block* block, size_t start, size_t rows) {
+    SCOPED_TIMER(_rust_arrow_to_block_time);
+    DORIS_CHECK(batch != nullptr);
+    DORIS_CHECK(block != nullptr);
+    DORIS_CHECK_LE(start + rows, static_cast<size_t>(batch->num_rows()));
+    const auto arrow_start = static_cast<int64_t>(start);
+    const auto arrow_end = static_cast<int64_t>(start + rows);
+    std::unordered_set<size_t> materialized_indices;
+    materialized_indices.reserve(_projected_columns.size());
+    {
+        auto columns_guard = block->mutate_columns_scoped();
+        auto& columns = columns_guard.mutable_columns();
+        for (int c = 0; c < batch->num_columns(); ++c) {
+            const auto& field = batch->schema()->field(c);
+            if (field->name() == VALUE_KIND_FIELD) {
+                continue;
+            }
+            // Projected column names are FE-normalized to lowercase.
+            // paimon-rust's case_sensitive=false setting also case-folds column
+            // names in the schema output, so exact match works — but tolerate
+            // mixed-case Rust output by folding here as well.
+            auto it = _output_name_to_idx.find(field->name());
+            if (it == _output_name_to_idx.end()) {
+                it = _output_name_to_idx.find(to_lower(field->name()));
+            }
+            if (it == _output_name_to_idx.end()) {
+                // Skip columns that are not in the block (e.g. columns dropped by
+                // slot pruning).
+                continue;
+            }
+            const auto output_idx = it->second;
+            if (!materialized_indices.emplace(output_idx).second) {
+                return Status::InternalError("paimon-rust returned duplicate column '{}'",
+                                             field->name());
+            }
+            try {
+                RETURN_IF_ERROR(columns_guard.get_datatype_by_position(output_idx)
+                                        ->get_serde()
+                                        ->read_column_from_arrow(*columns[output_idx],

Review Comment:
   Thanks. The original #66227 BE path also imports Rust Arrow batches directly into SerDe, so the sliced-BOOLEAN limitation predates this master port. Fixing the Arrow layout at the Rust/BE boundary deserves a focused change and a differential fixture; a blanket data-evolution fallback here would remove an original capability. Resolving as deferred, not fixed.



##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonRustReaderSelector.java:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.paimon;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.handle.ConnectorColumnHandle;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.rest.RESTTokenFileIO;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.types.ArrayType;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.DataTypeRoot;
+import org.apache.paimon.types.MapType;
+import org.apache.paimon.types.MultisetType;
+import org.apache.paimon.types.RowType;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Chooses the Rust reader only for table, split, storage and backend shapes it can read safely.
+ * Nested pruning and precision 7-9 timestamps still require JNI because the pinned Arrow ABI
+ * cannot preserve their full semantics. COUNT(*) metadata ranges bypass this selector entirely.
+ */
+final class PaimonRustReaderSelector {
+    static final String BACKEND_CAPABILITY = "paimon-rust-reader";
+    private static final String ENABLE_PAIMON_RUST_READER = "enable_paimon_rust_reader";
+    private static final String ENABLE_FILE_SCANNER_V2 = "enable_file_scanner_v2";
+    private static final String FORCE_JNI_SCANNER = "force_jni_scanner";
+    private static final String DELETION_VECTORS_MERGE_ON_READ = "deletion-vectors.merge-on-read";
+    private static final Set<String> VERIFIED_LOCATION_SCHEMES =
+            new HashSet<>(Arrays.asList("s3", "s3a", "oss", "hdfs", "file"));
+    private static final Set<String> VERIFIED_HDFS_OPTION_KEYS = new HashSet<>(Arrays.asList(
+            "fs.defaultFS",
+            "hadoop.security.authentication",
+            "hdfs.security.authentication",
+            "ipc.client.fallback-to-simple-auth-allowed"));
+
+    private final FileStoreTable table;
+    private final PaimonRustReaderCapabilities capabilities;
+    private final Map<String, String> backendStorageProperties;
+    private final boolean scanCompatible;
+    private final boolean partialUpdateOrAggregateWithDeletionVectors;
+    private final boolean deletionVectorsMergeOnRead;
+
+    PaimonRustReaderSelector(ConnectorSession session, boolean backendsSupportRust,
+            FileStoreTable table, List<ConnectorColumnHandle> columns,
+            Map<String, String> backendStorageProperties, boolean usesFallbackRead,
+            boolean incrementalRead, boolean projectedVariant) {
+        this.table = table;
+        this.backendStorageProperties = backendStorageProperties;
+        CoreOptions options = table.coreOptions();
+        CoreOptions.MergeEngine mergeEngine = options.mergeEngine();
+        partialUpdateOrAggregateWithDeletionVectors = options.deletionVectorsEnabled()
+                && (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+                        || mergeEngine == CoreOptions.MergeEngine.AGGREGATE);
+        TableSchema schema = table.schema();
+        String mergeOnRead = schema.options().get(DELETION_VECTORS_MERGE_ON_READ);
+        deletionVectorsMergeOnRead = "true".equalsIgnoreCase(mergeOnRead);
+
+        String location = table.location().toString();
+        boolean enabled = session != null && "true".equalsIgnoreCase(
+                session.getSessionProperties().get(ENABLE_PAIMON_RUST_READER));
+        boolean fileScannerV2 = session != null && "true".equalsIgnoreCase(
+                session.getSessionProperties().get(ENABLE_FILE_SCANNER_V2));
+        boolean forceJniScanner = session != null && "true".equalsIgnoreCase(
+                session.getSessionProperties().get(FORCE_JNI_SCANNER));
+        boolean restTokenTable = table.fileIO() instanceof RESTTokenFileIO;
+        boolean unsupportedMergeOption = (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+                || mergeEngine == CoreOptions.MergeEngine.AGGREGATE)
+                && hasRustUnsupportedMergeOption(schema.options(), mergeEngine);
+        boolean deduplicateIgnoreDelete = mergeEngine == CoreOptions.MergeEngine.DEDUPLICATE
+                && options.ignoreDelete();
+        String scanMode = PaimonScanParams.withoutTimeTravelSelectors(schema)
+                .options().get(CoreOptions.SCAN_MODE.key());
+        boolean scanModeSupported = scanMode == null || "default".equalsIgnoreCase(scanMode);
+
+        scanCompatible = enabled && fileScannerV2 && !forceJniScanner
+                && backendsSupportRust
+                && !usesFallbackRead && !incrementalRead && !projectedVariant
+                && !options.queryAuthEnabled() && !restTokenTable
+                && !(partialUpdateOrAggregateWithDeletionVectors && deletionVectorsMergeOnRead)
+                && !deduplicateIgnoreDelete
+                && !unsupportedMergeOption && scanModeSupported
+                && isRustVerifiedLocationScheme(location)
+                && (!isHdfsLocationScheme(location)
+                        || isRustVerifiedHdfsBackend(backendStorageProperties))
+                && isCredentialProviderTranslatable(location, backendStorageProperties);
+        capabilities = new PaimonRustReaderCapabilities(table, columns);
+    }
+
+    boolean canRead(DataSplit split) {
+        if (!scanCompatible || splitHasExternalFiles(split)) {
+            return false;
+        }
+        if (splitHasOrcFile(split) && table.schema().fields().stream()
+                .anyMatch(field -> containsTimestampLtz(field.type()))) {
+            return false;
+        }
+        if (partialUpdateOrAggregateWithDeletionVectors && !deletionVectorsMergeOnRead
+                && !isFullyMaterializedPkDvSplit(split)) {
+            return false;
+        }
+        return capabilities.canRead(split);
+    }
+
+    static boolean isRustVerifiedLocationScheme(String location) {
+        if (location == null) {
+            return false;
+        }
+        int separator = location.indexOf("://");

Review Comment:
   Thanks. I checked the original #66227 implementation: its location-scheme and HDFS eligibility checks use the same :// detection, so authority-less HDFS is an inherited edge case rather than a master-port regression. We will handle URI qualification/eligibility with a focused follow-up instead of changing this forward-port behavior. Resolving as deferred, not fixed.



##########
be/src/format_v2/table/paimon_rust_table_reader.cpp:
##########
@@ -0,0 +1,944 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace rust_handles {
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn)                \
+    struct type##_deleter {                       \
+        void operator()(paimon_##type* p) const { \
+            if (p) {                              \
+                freefn(p);                        \
+            }                                     \
+        }                                         \
+    };                                            \
+    using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+} // namespace rust_handles
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+    explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+    ~ArrowBatch() {
+        auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+        auto* array = static_cast<ArrowArray*>(batch_.array);
+        if (array && array->release) {
+            array->release(array);
+        }
+        if (schema && schema->release) {
+            schema->release(schema);
+        }
+        paimon_arrow_batch_free(batch_);
+    }
+
+    ArrowBatch(const ArrowBatch&) = delete;
+    ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+    ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+    ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+    paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+    rust_handles::error_ptr owned(err);
+    if (!owned) {
+        return "unknown error";
+    }
+    std::string msg;
+    if (owned->message.data != nullptr && owned->message.len > 0) {
+        msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+    }
+    return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+    std::string out;
+    for (const auto& kv : options) {
+        if (!out.empty()) {
+            out += ", ";
+        }
+        out += kv.first;
+    }
+    return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+    rust_handles::table_ptr table;
+    rust_handles::read_builder_ptr read_builder;
+    rust_handles::plan_ptr plan;
+    rust_handles::table_read_ptr table_read;
+    rust_handles::record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+    RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+    {
+        // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+        // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.init_timer);
+        // Materialize TIMESTAMP_LTZ in the session timezone — the same
+        // convention as the JNI reader (PaimonJniScanner reads time_zone from
+        // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+        // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+        // of _ctz, so NTZ wall-clock semantics are preserved.
+        DORIS_CHECK(_runtime_state != nullptr);
+        _ctz = _runtime_state->timezone_obj();
+        if (_scanner_profile != nullptr) {
+            file_scan_profile::ensure_hierarchy(_scanner_profile);
+            _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+                                               file_scan_profile::TABLE_READER);
+            _rust_open_split_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+            _rust_read_batch_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+            _rust_arrow_to_block_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+            _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+                                                       TUnit::UNIT, "PaimonRustReader");
+            _rust_predicates_converted = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+            _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+                                                         TUnit::UNIT, "PaimonRustReader");
+            _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+            _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+        }
+        // Projected column name -> fixed output position, registered with both the exact and
+        // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+        // semantics: exact match first, lower-case fallback on lookup).
+        _output_name_to_idx.reserve(_projected_columns.size() * 2);
+        for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+            _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+            _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+        }
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+    // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+    // are idempotent, and clear it only when a new split is explicitly prepared.
+    _close_split_reader();
+    _split_eof = false;
+    _current_range = options.current_range;
+    RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+    if (current_split_pruned()) {
+        return Status::OK();
+    }
+    if (_is_table_level_count_active()) {
+        // No rust pipeline is opened; get_block emits the synthetic count rows.
+        return Status::OK();
+    }
+    RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+    {
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.prepare_split_timer);
+        // Open may dominate the scan or fail before get_block; include it in the parent timer.
+        SCOPED_TIMER(_rust_total_time);
+        SCOPED_TIMER(_rust_open_split_time);
+        RETURN_IF_ERROR(_open_split_reader(options.current_range));
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+    SCOPED_TIMER(_profile.total_timer);
+    SCOPED_TIMER(_profile.exec_timer);
+    SCOPED_TIMER(_rust_total_time);
+    DORIS_CHECK(block != nullptr);
+    DORIS_CHECK(eos != nullptr);
+    DORIS_CHECK(block->columns() == _projected_columns.size());
+    block->clear_column_data(_projected_columns.size());
+    *eos = false;
+
+    if (_is_table_level_count_active()) {
+        return _read_table_level_count(block, eos);
+    }
+
+    // num_splits == 0 yields an empty (but valid) stream: report EOF.
+    if (_split_eof) {
+        *eos = true;
+        return Status::OK();
+    }
+    if (!_handles || !_handles->reader) {
+        return Status::InternalError("paimon-rust reader is not initialized");
+    }
+
+    while (true) {
+        // Mirror the base TableReader cancellation contract so a cancelled query does not
+        // drain the whole split.
+        if (_io_ctx != nullptr && _io_ctx->should_stop) {
+            _split_eof = true;
+            _close_split_reader();
+            *eos = true;
+            return Status::OK();
+        }
+
+        if (!_pending_record_batch) {
+            paimon_result_next_batch next;
+            {
+                SCOPED_TIMER(_rust_read_batch_time);
+                next = paimon_record_batch_reader_next(_handles->reader.get());

Review Comment:
   Thanks for the finding. Rust-owned read/decode buffers are not yet charged to the Doris query tracker. This limitation is also present in the original branch-4.1 implementation. We are keeping this forward-port aligned with #66227 and will address cross-FFI memory accounting separately; resolving this thread as a documented follow-up, not as fixed in this PR.



##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonRustReaderSelector.java:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.paimon;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.handle.ConnectorColumnHandle;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.rest.RESTTokenFileIO;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.types.ArrayType;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.DataTypeRoot;
+import org.apache.paimon.types.MapType;
+import org.apache.paimon.types.MultisetType;
+import org.apache.paimon.types.RowType;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Chooses the Rust reader only for table, split, storage and backend shapes it can read safely.
+ * Nested pruning and precision 7-9 timestamps still require JNI because the pinned Arrow ABI
+ * cannot preserve their full semantics. COUNT(*) metadata ranges bypass this selector entirely.
+ */
+final class PaimonRustReaderSelector {
+    static final String BACKEND_CAPABILITY = "paimon-rust-reader";
+    private static final String ENABLE_PAIMON_RUST_READER = "enable_paimon_rust_reader";
+    private static final String ENABLE_FILE_SCANNER_V2 = "enable_file_scanner_v2";
+    private static final String FORCE_JNI_SCANNER = "force_jni_scanner";
+    private static final String DELETION_VECTORS_MERGE_ON_READ = "deletion-vectors.merge-on-read";
+    private static final Set<String> VERIFIED_LOCATION_SCHEMES =
+            new HashSet<>(Arrays.asList("s3", "s3a", "oss", "hdfs", "file"));
+    private static final Set<String> VERIFIED_HDFS_OPTION_KEYS = new HashSet<>(Arrays.asList(
+            "fs.defaultFS",
+            "hadoop.security.authentication",
+            "hdfs.security.authentication",
+            "ipc.client.fallback-to-simple-auth-allowed"));
+
+    private final FileStoreTable table;
+    private final PaimonRustReaderCapabilities capabilities;
+    private final Map<String, String> backendStorageProperties;
+    private final boolean scanCompatible;
+    private final boolean partialUpdateOrAggregateWithDeletionVectors;
+    private final boolean deletionVectorsMergeOnRead;
+
+    PaimonRustReaderSelector(ConnectorSession session, boolean backendsSupportRust,
+            FileStoreTable table, List<ConnectorColumnHandle> columns,
+            Map<String, String> backendStorageProperties, boolean usesFallbackRead,
+            boolean incrementalRead, boolean projectedVariant) {
+        this.table = table;
+        this.backendStorageProperties = backendStorageProperties;
+        CoreOptions options = table.coreOptions();
+        CoreOptions.MergeEngine mergeEngine = options.mergeEngine();
+        partialUpdateOrAggregateWithDeletionVectors = options.deletionVectorsEnabled()
+                && (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+                        || mergeEngine == CoreOptions.MergeEngine.AGGREGATE);
+        TableSchema schema = table.schema();
+        String mergeOnRead = schema.options().get(DELETION_VECTORS_MERGE_ON_READ);
+        deletionVectorsMergeOnRead = "true".equalsIgnoreCase(mergeOnRead);
+
+        String location = table.location().toString();
+        boolean enabled = session != null && "true".equalsIgnoreCase(
+                session.getSessionProperties().get(ENABLE_PAIMON_RUST_READER));
+        boolean fileScannerV2 = session != null && "true".equalsIgnoreCase(
+                session.getSessionProperties().get(ENABLE_FILE_SCANNER_V2));
+        boolean forceJniScanner = session != null && "true".equalsIgnoreCase(
+                session.getSessionProperties().get(FORCE_JNI_SCANNER));
+        boolean restTokenTable = table.fileIO() instanceof RESTTokenFileIO;
+        boolean unsupportedMergeOption = (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+                || mergeEngine == CoreOptions.MergeEngine.AGGREGATE)
+                && hasRustUnsupportedMergeOption(schema.options(), mergeEngine);
+        boolean deduplicateIgnoreDelete = mergeEngine == CoreOptions.MergeEngine.DEDUPLICATE
+                && options.ignoreDelete();
+        String scanMode = PaimonScanParams.withoutTimeTravelSelectors(schema)
+                .options().get(CoreOptions.SCAN_MODE.key());
+        boolean scanModeSupported = scanMode == null || "default".equalsIgnoreCase(scanMode);
+
+        scanCompatible = enabled && fileScannerV2 && !forceJniScanner
+                && backendsSupportRust
+                && !usesFallbackRead && !incrementalRead && !projectedVariant
+                && !options.queryAuthEnabled() && !restTokenTable
+                && !(partialUpdateOrAggregateWithDeletionVectors && deletionVectorsMergeOnRead)
+                && !deduplicateIgnoreDelete
+                && !unsupportedMergeOption && scanModeSupported
+                && isRustVerifiedLocationScheme(location)
+                && (!isHdfsLocationScheme(location)
+                        || isRustVerifiedHdfsBackend(backendStorageProperties))
+                && isCredentialProviderTranslatable(location, backendStorageProperties);
+        capabilities = new PaimonRustReaderCapabilities(table, columns);
+    }
+
+    boolean canRead(DataSplit split) {
+        if (!scanCompatible || splitHasExternalFiles(split)) {
+            return false;
+        }
+        if (splitHasOrcFile(split) && table.schema().fields().stream()

Review Comment:
   Thanks. The original #66227 selector likewise gates ORC TIMESTAMP_LTZ but not this low-precision TIMESTAMP range. This is a valid inherited limitation. We will address ORC timestamp decoding or a narrowly scoped eligibility rule separately, with parity coverage, while preserving the behavior of this forward-port. Resolving as deferred, not fixed.



##########
be/src/format_v2/table/paimon_rust_table_reader.cpp:
##########
@@ -0,0 +1,944 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace rust_handles {
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn)                \
+    struct type##_deleter {                       \
+        void operator()(paimon_##type* p) const { \
+            if (p) {                              \
+                freefn(p);                        \
+            }                                     \
+        }                                         \
+    };                                            \
+    using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+} // namespace rust_handles
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+    explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+    ~ArrowBatch() {
+        auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+        auto* array = static_cast<ArrowArray*>(batch_.array);
+        if (array && array->release) {
+            array->release(array);
+        }
+        if (schema && schema->release) {
+            schema->release(schema);
+        }
+        paimon_arrow_batch_free(batch_);
+    }
+
+    ArrowBatch(const ArrowBatch&) = delete;
+    ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+    ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+    ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+    paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+    rust_handles::error_ptr owned(err);
+    if (!owned) {
+        return "unknown error";
+    }
+    std::string msg;
+    if (owned->message.data != nullptr && owned->message.len > 0) {
+        msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+    }
+    return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+    std::string out;
+    for (const auto& kv : options) {
+        if (!out.empty()) {
+            out += ", ";
+        }
+        out += kv.first;
+    }
+    return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+    rust_handles::table_ptr table;
+    rust_handles::read_builder_ptr read_builder;
+    rust_handles::plan_ptr plan;
+    rust_handles::table_read_ptr table_read;
+    rust_handles::record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+    RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+    {
+        // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+        // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.init_timer);
+        // Materialize TIMESTAMP_LTZ in the session timezone — the same
+        // convention as the JNI reader (PaimonJniScanner reads time_zone from
+        // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+        // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+        // of _ctz, so NTZ wall-clock semantics are preserved.
+        DORIS_CHECK(_runtime_state != nullptr);
+        _ctz = _runtime_state->timezone_obj();
+        if (_scanner_profile != nullptr) {
+            file_scan_profile::ensure_hierarchy(_scanner_profile);
+            _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+                                               file_scan_profile::TABLE_READER);
+            _rust_open_split_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+            _rust_read_batch_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+            _rust_arrow_to_block_time =
+                    ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+            _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+                                                       TUnit::UNIT, "PaimonRustReader");
+            _rust_predicates_converted = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+            _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+                                                         TUnit::UNIT, "PaimonRustReader");
+            _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+            _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+                    _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+        }
+        // Projected column name -> fixed output position, registered with both the exact and
+        // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+        // semantics: exact match first, lower-case fallback on lookup).
+        _output_name_to_idx.reserve(_projected_columns.size() * 2);
+        for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+            _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+            _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+        }
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+    // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+    // are idempotent, and clear it only when a new split is explicitly prepared.
+    _close_split_reader();
+    _split_eof = false;
+    _current_range = options.current_range;
+    RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+    if (current_split_pruned()) {
+        return Status::OK();
+    }
+    if (_is_table_level_count_active()) {
+        // No rust pipeline is opened; get_block emits the synthetic count rows.
+        return Status::OK();
+    }
+    RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+    {
+        SCOPED_TIMER(_profile.total_timer);
+        SCOPED_TIMER(_profile.prepare_split_timer);
+        // Open may dominate the scan or fail before get_block; include it in the parent timer.
+        SCOPED_TIMER(_rust_total_time);
+        SCOPED_TIMER(_rust_open_split_time);
+        RETURN_IF_ERROR(_open_split_reader(options.current_range));
+    }
+    return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+    SCOPED_TIMER(_profile.total_timer);
+    SCOPED_TIMER(_profile.exec_timer);
+    SCOPED_TIMER(_rust_total_time);
+    DORIS_CHECK(block != nullptr);
+    DORIS_CHECK(eos != nullptr);
+    DORIS_CHECK(block->columns() == _projected_columns.size());
+    block->clear_column_data(_projected_columns.size());
+    *eos = false;
+
+    if (_is_table_level_count_active()) {
+        return _read_table_level_count(block, eos);
+    }
+
+    // num_splits == 0 yields an empty (but valid) stream: report EOF.
+    if (_split_eof) {
+        *eos = true;
+        return Status::OK();
+    }
+    if (!_handles || !_handles->reader) {
+        return Status::InternalError("paimon-rust reader is not initialized");
+    }
+
+    while (true) {
+        // Mirror the base TableReader cancellation contract so a cancelled query does not
+        // drain the whole split.
+        if (_io_ctx != nullptr && _io_ctx->should_stop) {
+            _split_eof = true;
+            _close_split_reader();
+            *eos = true;
+            return Status::OK();
+        }
+
+        if (!_pending_record_batch) {
+            paimon_result_next_batch next;
+            {
+                SCOPED_TIMER(_rust_read_batch_time);
+                next = paimon_record_batch_reader_next(_handles->reader.get());
+            }
+            if (next.error != nullptr) {

Review Comment:
   Thanks. The original #66227 Rust reader also maps read errors to INTERNAL_ERROR, so missing-file classification is inherited. A correct fix needs a distinguishable missing-data-file error from the Rust binding; converting generic IO errors to NOT_FOUND would be unsafe. We will pursue that separately with differential coverage. Resolving as deferred, not fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6187718461

   # FE Regression Coverage Report
   Increment line coverage `  20.00% (19/95)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org