You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by yi...@apache.org on 2026/10/08 03:23:52 UTC
(doris) 21/35: branch-4.1:[feat](paimon) support paimon-rust reader on BE (#66227)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e98db0594f7a0ed47e5aaafc53a05dc3b222a349
Author: Fangyuan Deng <98...@qq.com>
AuthorDate: Wed Sep 30 21:08:37 2026 +0800
branch-4.1:[feat](paimon) support paimon-rust reader on BE (#66227)
### What problem does this PR solve?
Related PR: #65883 (for master ， current closed)
**this pr add paimon rust reader both in scanner v1 and v2.
the scanner v1 code is just for legacy, will be removed later.**
Problem Summary:
Adds a third BE-side reader for Paimon FORMAT_JNI splits, backed by the
`paimon-rust` C bindings (`libpaimon_c.a`, statically linked). Rust
Reader is gated by the session variable
`enable_paimon_rust_reader` (default `false`).
Compared with the existing readers:
- **vs paimon-jni**: no JVM round-trip on the hot path — the split is
deserialized in-process and rows arrive via the Arrow C Data Interface,
which shortens the scanner critical section and removes JNI copy cost.
**Architecture**
FE (`PaimonScanNode`): when `enable_paimon_rust_reader` is on and the
split
is a `DataSplit`, encode it with paimon's native binary serialization
(`PaimonUtil.encodeDataSplitToString`), stamp the range with
`TPaimonReaderType.PAIMON_RUST`, and attach `db_name` / `table_name` /
`paimon_table` so BE can open the table through a catalog.
BE (`FileScanner::_get_next_reader`): dispatches on `reader_type`. On
`PAIMON_RUST` it instantiates `PaimonRustReader`, which:
1. `paimon_catalog_create({warehouse, metastore=filesystem, ...})`
- warehouse is derived from the table root path by stripping
`/<db>.db/<table>` (paimon FS convention).
2. `paimon_identifier_new(db, tbl)` → `paimon_catalog_get_table`
3. `paimon_table_new_read_builder` → projection (case-insensitive) →
optional filter from `PaimonRustPredicateConverter` (translates Doris
push-down conjuncts into paimon-rust predicates, best-effort so any
non-representable conjunct is silently dropped and re-applied by the
engine).
4. `paimon_plan_from_split_bytes(split_bytes)` — one BE scanner reads
exactly its FE-planned split (no re-planning).
5. `paimon_table_read_to_arrow` → streamed arrow record batches → block.
Partition columns are back-filled in `on_after_read_block` from
FE-provided `columns_from_path*`, mirroring `PaimonCppReader`. OSS/S3
hadoop-style credential keys (`fs.oss.*` / `fs.s3a.*`) are remapped to
`AWS_ACCESS_KEY` / `AWS_ENDPOINT` / ... so paimon-rust's FileIO can
authenticate.
**Thirdparty**
- Introduces `paimon-rust-2fb5e49` (built with Rust 1.91.0, feature
`paimon/storage-hdfs`, `--locked`).
- Auto-installs `cbindgen` and regenerates the C header from the Rust
extern "C" surface via a locally-written `cbindgen.toml`
(`language=C, include_guard=PAIMON_C_H, pragma_once=true,
cpp_compat=true`),
so the shipped `paimon.h` always matches the built `.a`.
### Release note
Add `enable_paimon_rust_reader` session variable. When enabled, Paimon
non-native reads on BE use the paimon-rust reader instead of paimon-jni
/
paimon-cpp.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
1. `CREATE CATALOG paimon_fs PROPERTIES ("type"="paimon",
"paimon.catalog.type"="filesystem",
"warehouse"="file:///media/ssd1/test_paimon/warehouse/");` when using
"warehouse"="hdfs://.../warehouse/", add HADOOP_CONF_DIR=xx in be.conf
2. `SET enable_paimon_rust_reader=true; SET force_jni_scanner=1`
3. `SELECT` from a partitioned/DV paimon table; results must match
`enable_paimon_rust_reader=false and force_jni_scanner=0`.
4. `SELECT count(*)` — should hit table-level row count push-down.
5. `WHERE a=1 AND b>2` — BE INFO log should show `paimon-rust predicate
pushdown: applied`.
- [x] Regression test
- [ ] No need to test or manual test.
- Behavior changed:
- [x] No. Guarded by `enable_paimon_rust_reader` (default `false`); when
off, all paths are unchanged.
- Does this need documentation?
- [ ] No.
- [x] Yes. <!-- Add doris-website PR link here for the new session
variable -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---------
Co-authored-by: dengfangyuan <de...@kuaishou.com>
Co-authored-by: Claude Opus 5 (1M context) <no...@anthropic.com>
Co-authored-by: Gabriel <ga...@gmail.com>
---
be/CMakeLists.txt | 3 +-
be/cmake/thirdparty.cmake | 7 +
be/src/agent/heartbeat_server.cpp | 2 +
be/src/exec/scan/file_scanner.cpp | 19 +
be/src/exec/scan/file_scanner_v2.cpp | 15 +-
be/src/format_v2/table/paimon_reader.cpp | 62 +-
be/src/format_v2/table/paimon_reader.h | 18 +-
.../table/paimon_rust_predicate_converter.cpp | 840 +++++++++++
.../table/paimon_rust_predicate_converter.h | 161 ++
.../format_v2/table/paimon_rust_table_reader.cpp | 924 ++++++++++++
be/src/format_v2/table/paimon_rust_table_reader.h | 188 +++
be/test/exec/scan/file_scanner_v2_test.cpp | 42 +
.../test_data/paimon_rust_batch_size/README.md | 32 +
...-3b6dc957-0664-4ec4-946c-9535d228957a-0.parquet | Bin 0 -> 772 bytes
.../paimon_rust_batch_size/schema/schema-0 | 22 +
.../test_data/paimon_rust_batch_size/split.bin | Bin 0 -> 557 bytes
be/test/format_v2/table/paimon_reader_test.cpp | 167 ++-
.../table/paimon_rust_predicate_converter_test.cpp | 674 +++++++++
.../table/paimon_rust_table_reader_test.cpp | 712 +++++++++
build.sh | 17 +-
.../doris/common/profile/ExecutionProfile.java | 69 +-
.../org/apache/doris/common/profile/Profile.java | 68 +
.../doris/common/profile/SummaryProfile.java | 10 +
.../paimon/DorisToPaimonTypeVisitor.java | 17 +-
.../doris/datasource/paimon/PaimonScanParams.java | 59 +
.../apache/doris/datasource/paimon/PaimonUtil.java | 33 +
.../source/PaimonRustReaderCapabilities.java | 282 ++++
.../datasource/paimon/source/PaimonScanNode.java | 683 ++++++++-
.../java/org/apache/doris/qe/SessionVariable.java | 19 +
.../qe/runtime/PipelineExecutionTaskBuilder.java | 9 +-
.../main/java/org/apache/doris/system/Backend.java | 11 +
.../org/apache/doris/system/BackendHbResponse.java | 10 +
.../java/org/apache/doris/system/HeartbeatMgr.java | 5 +-
.../common/profile/ProfileCompletionStateTest.java | 217 +++
.../doris/common/profile/ProfileManagerTest.java | 29 +
.../paimon/DorisToPaimonTypeVisitorTest.java | 78 +
.../source/PaimonRustBackendCapabilityTest.java | 118 ++
.../source/PaimonRustReaderCompatibilityTest.java | 496 +++++++
.../source/PaimonRustReaderEvolutionTest.java | 184 +++
.../source/PaimonRustReaderLegacySchemaTest.java | 101 ++
.../paimon/source/PaimonScanNodeTest.java | 1545 +++++++++++++++++++-
.../runtime/PipelineExecutionTaskBuilderTest.java | 96 ++
.../system/BackendPaimonRustCapabilityTest.java | 70 +
gensrc/thrift/HeartbeatService.thrift | 1 +
gensrc/thrift/PaloInternalService.thrift | 3 +
gensrc/thrift/PlanNodes.thrift | 7 +
.../test_paimon_rust_reader_cast_predicates.groovy | 192 +++
.../test_paimon_rust_reader_compatibility.groovy | 308 ++++
.../test_paimon_rust_reader_eq_for_null.groovy | 555 +++++++
...test_paimon_rust_reader_schema_evolution.groovy | 134 ++
.../paimon/test_paimon_rust_reader_v2.groovy | 146 ++
thirdparty/build-thirdparty.sh | 182 ++-
thirdparty/test/paimon-build-test.py | 279 ++++
thirdparty/vars.sh | 7 +
54 files changed, 9840 insertions(+), 88 deletions(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index 0da7cdd2b0f..47e69d3907e 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -744,7 +744,8 @@ if (GLIBC_COMPATIBILITY)
glibc-compatibility-explicit
glibc-compatibility
-lm
- lance_c)
+ lance_c
+ paimon_c)
endif()
if (NOT OS_MACOSX)
diff --git a/be/cmake/thirdparty.cmake b/be/cmake/thirdparty.cmake
index 7f518a8cb5e..b67c8e9d0aa 100644
--- a/be/cmake/thirdparty.cmake
+++ b/be/cmake/thirdparty.cmake
@@ -117,6 +117,13 @@ add_thirdparty(lance_c LIB64 NOTADD)
if (NOT GLIBC_COMPATIBILITY)
list(APPEND COMMON_THIRDPARTY m lance_c)
endif()
+# libpaimon_c.a (built from paimon-rust) brings in Rust compiler_builtins that
+# would otherwise steal libm symbols. Place libm before it so the final linker
+# resolves C math symbols from the system library first.
+add_thirdparty(paimon_c LIB64 NOTADD)
+if (NOT GLIBC_COMPATIBILITY)
+ list(APPEND COMMON_THIRDPARTY m paimon_c)
+endif()
add_thirdparty(brpc LIB64)
add_thirdparty(rocksdb)
add_thirdparty(cyrus-sasl LIBNAME "lib/libsasl2.a")
diff --git a/be/src/agent/heartbeat_server.cpp b/be/src/agent/heartbeat_server.cpp
index 441218f469c..5cf8c7df01d 100644
--- a/be/src/agent/heartbeat_server.cpp
+++ b/be/src/agent/heartbeat_server.cpp
@@ -95,6 +95,8 @@ void HeartbeatServer::heartbeat(THeartbeatResult& heartbeat_result,
heartbeat_result.backend_info.__set_fragment_last_active_time(
get_fragment_last_active_time());
heartbeat_result.backend_info.__set_be_mem(MemInfo::physical_mem());
+ // Reader enum support cannot be inferred from a release version during rolling upgrades.
+ heartbeat_result.backend_info.__set_supports_paimon_rust_reader(true);
}
watch.stop();
if (watch.elapsed_time() > 1000L * 1000L * 1000L) {
diff --git a/be/src/exec/scan/file_scanner.cpp b/be/src/exec/scan/file_scanner.cpp
index 34cf0928553..ee5203f16f0 100644
--- a/be/src/exec/scan/file_scanner.cpp
+++ b/be/src/exec/scan/file_scanner.cpp
@@ -1117,6 +1117,25 @@ Status FileScanner::_get_next_reader() {
_cur_reader = std::move(mc_reader);
} else if (range.__isset.table_format_params &&
range.table_format_params.table_format_type == "paimon") {
+ const auto& paimon_params = range.table_format_params.paimon_params;
+ if (paimon_params.__isset.reader_type) {
+ switch (paimon_params.reader_type) {
+ case TPaimonReaderType::PAIMON_JNI:
+ break;
+ case TPaimonReaderType::PAIMON_RUST:
+ return Status::NotSupported(
+ "paimon rust reader requires enable_file_scanner_v2=true; the V1 "
+ "FileScanner no longer supports PAIMON_RUST splits");
+ case TPaimonReaderType::PAIMON_NATIVE:
+ return Status::InternalError(
+ "invalid PAIMON_NATIVE reader_type for paimon FORMAT_JNI split, "
+ "possibly caused by FE/BE protocol mismatch");
+ default:
+ return Status::InternalError(
+ "unknown paimon reader_type for paimon FORMAT_JNI split, possibly "
+ "caused by FE/BE protocol mismatch");
+ }
+ }
_cur_reader = PaimonJniReader::create_unique(_file_slot_descs, _state, _profile,
range, _params);
init_status = ((PaimonJniReader*)(_cur_reader.get()))->init_reader();
diff --git a/be/src/exec/scan/file_scanner_v2.cpp b/be/src/exec/scan/file_scanner_v2.cpp
index 3f824b57fc1..ab055f61dc3 100644
--- a/be/src/exec/scan/file_scanner_v2.cpp
+++ b/be/src/exec/scan/file_scanner_v2.cpp
@@ -150,8 +150,19 @@ bool is_supported_jni_table_format(const TFileRangeDesc& range) {
}
const auto& params = range.table_format_params.paimon_params;
if (params.__isset.reader_type) {
- return params.reader_type == TPaimonReaderType::PAIMON_JNI &&
- params.__isset.paimon_split;
+ if (params.reader_type == TPaimonReaderType::PAIMON_JNI) {
+ return params.__isset.paimon_split;
+ }
+ if (params.reader_type == TPaimonReaderType::PAIMON_RUST) {
+ // The rust leaf reader inside PaimonHybridReader consumes the logical DataSplit
+ // directly. It needs the schema-json pipeline fields; a missing one fails fast
+ // in PaimonRustTableReader::prepare_split with a precise error.
+ return params.__isset.paimon_split && params.__isset.paimon_table_schema_json &&
+ !params.paimon_table_schema_json.empty();
+ }
+ // V2 cannot pass a logical DataSplit through a raw native child without silently
+ // dropping its multi-file semantics, so PAIMON_CPP must remain on the V1 fallback.
+ return false;
}
if (params.__isset.paimon_split) {
// Before reader_type was added, an encoded split unambiguously selected the Java
diff --git a/be/src/format_v2/table/paimon_reader.cpp b/be/src/format_v2/table/paimon_reader.cpp
index 9a724716c35..2ec6bde53f2 100644
--- a/be/src/format_v2/table/paimon_reader.cpp
+++ b/be/src/format_v2/table/paimon_reader.cpp
@@ -35,6 +35,7 @@
#include "format/table/paimon_reader.h"
#include "format_v2/column_mapper.h"
#include "format_v2/jni/paimon_jni_reader.h"
+#include "format_v2/table/paimon_rust_table_reader.h"
#include "format_v2/table/schema_history_util.h"
#include "gen_cpp/PlanNodes_types.h"
@@ -326,15 +327,18 @@ Status PaimonHybridReader::prepare_split(const format::SplitReadOptions& options
// timer around the first native or JNI child and double-count that initialization.
RETURN_IF_ERROR(_ensure_current_split_reader(options));
DORIS_CHECK(_current_split_reader != nullptr);
- if (!_is_jni_split(options.current_range)) {
- auto native_options = options;
- // Legacy FE plans wrap native files in FORMAT_JNI; normalize the child contract so the
- // physical reader does not overwrite its recovered Parquet/ORC format with that wrapper.
- RETURN_IF_ERROR(
- _to_file_format(options.current_range, &native_options.current_split_format));
- return _current_split_reader->prepare_split(native_options);
- }
- return _current_split_reader->prepare_split(options);
+ // Rust splits (like JNI splits) stay on the serialized DataSplit path. They must not go
+ // through the native format normalization below: FE sets paimon_params.file_format on
+ // serialized splits too, so FORMAT_JNI would be rewritten to PARQUET and dispatched to the
+ // native child.
+ if (_is_jni_split(options.current_range) || _is_rust_split(options.current_range)) {
+ return _current_split_reader->prepare_split(options);
+ }
+ auto native_options = options;
+ // Legacy FE plans wrap native files in FORMAT_JNI; normalize the child contract so the
+ // physical reader does not overwrite its recovered Parquet/ORC format with that wrapper.
+ RETURN_IF_ERROR(_to_file_format(options.current_range, &native_options.current_split_format));
+ return _current_split_reader->prepare_split(native_options);
}
Status PaimonHybridReader::build_physical_splits(const FileScanSplit& source_split,
@@ -391,6 +395,12 @@ Status PaimonHybridReader::close() {
close_status = std::move(status);
}
}
+ if (_rust_reader != nullptr) {
+ auto status = _rust_reader->close();
+ if (!status.ok() && close_status.ok()) {
+ close_status = std::move(status);
+ }
+ }
_current_split_reader = nullptr;
return close_status;
}
@@ -403,17 +413,36 @@ void PaimonHybridReader::set_batch_size(size_t batch_size) {
if (_jni_reader != nullptr) {
_jni_reader->set_batch_size(_batch_size);
}
+ if (_rust_reader != nullptr) {
+ _rust_reader->set_batch_size(_batch_size);
+ }
}
int64_t PaimonHybridReader::condition_cache_hit_count() const {
- // Both children survive split switches, so the wrapper must publish their cumulative totals;
+ // Child readers survive split switches, so the wrapper must publish their cumulative totals;
// returning only the active child would make FileScannerV2's monotonic delta go backwards.
return (_native_reader == nullptr ? 0 : _native_reader->condition_cache_hit_count()) +
- (_jni_reader == nullptr ? 0 : _jni_reader->condition_cache_hit_count());
+ (_jni_reader == nullptr ? 0 : _jni_reader->condition_cache_hit_count()) +
+ (_rust_reader == nullptr ? 0 : _rust_reader->condition_cache_hit_count());
}
Status PaimonHybridReader::_ensure_current_split_reader(const format::SplitReadOptions& options) {
- if (_is_jni_split(options.current_range)) {
+ if (_is_rust_split(options.current_range)) {
+ DCHECK(options.current_split_format == format::FileFormat::JNI);
+ if (_rust_reader == nullptr) {
+#ifdef BE_TEST
+ if (_test_rust_reader_factory) {
+ _rust_reader = _test_rust_reader_factory();
+ } else {
+ _rust_reader = std::make_unique<format::paimon::PaimonRustTableReader>();
+ }
+#else
+ _rust_reader = std::make_unique<format::paimon::PaimonRustTableReader>();
+#endif
+ RETURN_IF_ERROR(_init_child_reader(_rust_reader.get(), format::FileFormat::JNI));
+ }
+ _current_split_reader = _rust_reader.get();
+ } else if (_is_jni_split(options.current_range)) {
DCHECK(options.current_split_format == format::FileFormat::JNI);
if (_jni_reader == nullptr) {
#ifdef BE_TEST
@@ -500,6 +529,15 @@ bool PaimonHybridReader::_is_jni_split(const TFileRangeDesc& range) {
(!params.__isset.reader_type || params.reader_type == TPaimonReaderType::PAIMON_JNI);
}
+bool PaimonHybridReader::_is_rust_split(const TFileRangeDesc& range) {
+ if (!range.__isset.table_format_params || !range.table_format_params.__isset.paimon_params) {
+ return false;
+ }
+ const auto& params = range.table_format_params.paimon_params;
+ return params.__isset.paimon_split && params.__isset.reader_type &&
+ params.reader_type == TPaimonReaderType::PAIMON_RUST;
+}
+
Status PaimonHybridReader::_to_file_format(const TFileRangeDesc& range,
format::FileFormat* file_format) {
DORIS_CHECK(file_format != nullptr);
diff --git a/be/src/format_v2/table/paimon_reader.h b/be/src/format_v2/table/paimon_reader.h
index 25c6eb75b61..f0ac448828b 100644
--- a/be/src/format_v2/table/paimon_reader.h
+++ b/be/src/format_v2/table/paimon_reader.h
@@ -89,6 +89,7 @@ public:
#ifdef BE_TEST
static bool TEST_is_jni_split(const TFileRangeDesc& range) { return _is_jni_split(range); }
+ static bool TEST_is_rust_split(const TFileRangeDesc& range) { return _is_rust_split(range); }
static Status TEST_to_file_format(const TFileRangeDesc& range,
format::FileFormat* file_format) {
return _to_file_format(range, file_format);
@@ -96,19 +97,25 @@ public:
void TEST_install_batch_size_children() {
_native_reader = std::make_unique<format::TableReader>();
_jni_reader = std::make_unique<format::TableReader>();
+ _rust_reader = std::make_unique<format::TableReader>();
}
- std::pair<size_t, size_t> TEST_child_batch_sizes() const {
- return {_native_reader->TEST_batch_size(), _jni_reader->TEST_batch_size()};
+ std::tuple<size_t, size_t, size_t> TEST_child_batch_sizes() const {
+ return {_native_reader->TEST_batch_size(), _jni_reader->TEST_batch_size(),
+ _rust_reader->TEST_batch_size()};
}
- void TEST_set_child_condition_cache_hits(int64_t native_hits, int64_t jni_hits) {
+ void TEST_set_child_condition_cache_hits(int64_t native_hits, int64_t jni_hits,
+ int64_t rust_hits) {
_native_reader->TEST_set_condition_cache_hit_count(native_hits);
_jni_reader->TEST_set_condition_cache_hit_count(jni_hits);
+ _rust_reader->TEST_set_condition_cache_hit_count(rust_hits);
}
void TEST_set_child_reader_factories(
std::function<std::unique_ptr<format::TableReader>()> native_factory,
- std::function<std::unique_ptr<format::TableReader>()> jni_factory) {
+ std::function<std::unique_ptr<format::TableReader>()> jni_factory,
+ std::function<std::unique_ptr<format::TableReader>()> rust_factory) {
_test_native_reader_factory = std::move(native_factory);
_test_jni_reader_factory = std::move(jni_factory);
+ _test_rust_reader_factory = std::move(rust_factory);
}
#endif
@@ -117,14 +124,17 @@ private:
Status _init_child_reader(format::TableReader* reader, format::FileFormat file_format);
Status _clone_conjuncts(VExprContextSPtrs* conjuncts) const;
static bool _is_jni_split(const TFileRangeDesc& range);
+ static bool _is_rust_split(const TFileRangeDesc& range);
static Status _to_file_format(const TFileRangeDesc& range, format::FileFormat* file_format);
std::unique_ptr<format::TableReader> _native_reader; // handle parquet/orc native splits
std::unique_ptr<format::TableReader> _jni_reader; // handle serialized JNI splits
+ std::unique_ptr<format::TableReader> _rust_reader; // handle serialized rust splits
format::TableReader* _current_split_reader = nullptr;
#ifdef BE_TEST
std::function<std::unique_ptr<format::TableReader>()> _test_native_reader_factory;
std::function<std::unique_ptr<format::TableReader>()> _test_jni_reader_factory;
+ std::function<std::unique_ptr<format::TableReader>()> _test_rust_reader_factory;
#endif
};
diff --git a/be/src/format_v2/table/paimon_rust_predicate_converter.cpp b/be/src/format_v2/table/paimon_rust_predicate_converter.cpp
new file mode 100644
index 00000000000..8b1e280399e
--- /dev/null
+++ b/be/src/format_v2/table/paimon_rust_predicate_converter.cpp
@@ -0,0 +1,840 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+
+#include <algorithm>
+#include <cctype>
+#include <limits>
+#include <memory>
+#include <utility>
+
+#include "common/logging.h"
+#include "core/column/column_const.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "core/types.h"
+#include "core/value/decimalv2_value.h"
+#include "core/value/timestamptz_value.h"
+#include "core/value/vdatetime_value.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vcompound_pred.h"
+#include "exprs/vdirect_in_predicate.h"
+#include "exprs/vectorized_fn_call.h"
+#include "exprs/vexpr.h"
+#include "exprs/vin_predicate.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+
+namespace doris {
+
+namespace {
+// paimon_datum tags (see paimon.h / bindings/c/src/table.rs::datum_from_c).
+constexpr int32_t kTagBool = 0;
+constexpr int32_t kTagTinyInt = 1;
+constexpr int32_t kTagSmallInt = 2;
+constexpr int32_t kTagInt = 3;
+constexpr int32_t kTagLong = 4;
+constexpr int32_t kTagDouble = 6;
+constexpr int32_t kTagString = 7;
+constexpr int32_t kTagDate = 8;
+constexpr int32_t kTagTimestamp = 10;
+constexpr int32_t kTagDecimal = 12;
+constexpr int32_t kTagBytes = 13;
+
+// paimon decimal precision ceiling (paimon::Decimal::MAX_PRECISION).
+constexpr int32_t kPaimonDecimalMaxPrecision = 38;
+
+// RAII for an owned paimon_predicate*. and/or/not consume their inputs, so we
+// release() before handing pointers to them.
+struct predicate_deleter {
+ void operator()(paimon_predicate* p) const {
+ if (p) {
+ paimon_predicate_free(p);
+ }
+ }
+};
+using predicate_ptr = std::unique_ptr<paimon_predicate, predicate_deleter>;
+
+// RAII for an owned paimon_error*.
+struct error_deleter {
+ void operator()(paimon_error* p) const {
+ if (p) {
+ paimon_error_free(p);
+ }
+ }
+};
+using error_ptr = std::unique_ptr<paimon_error, error_deleter>;
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_predicate_error(paimon_error* err) {
+ error_ptr owned(err);
+ if (!owned) {
+ return "unknown error";
+ }
+ std::string msg;
+ if (owned->message.data != nullptr && owned->message.len > 0) {
+ msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+ }
+ return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+} // namespace
+
+PaimonRustPredicateConverter::PaimonRustPredicateConverter(
+ const std::vector<std::string>& column_names, const std::vector<DataTypePtr>& column_types,
+ const paimon_table* table)
+ : _table(table) {
+ DORIS_CHECK(column_names.size() == column_types.size());
+ _columns_by_name.reserve(column_names.size());
+ for (size_t i = 0; i < column_names.size(); ++i) {
+ _columns_by_name.emplace(_normalize_name(column_names[i]),
+ std::make_pair(column_names[i], column_types[i]));
+ }
+ // Paimon TIMESTAMP (wall clock) is stored as epoch-millis-of-the-wall-time
+ // and the DateTimeV2 serde decodes timezone-naive arrow values in UTC, so
+ // timestamp literals convert wall->epoch in UTC. utc_time_zone() needs no
+ // tzdata lookup, so the conversion cannot silently fall back to a
+ // machine-local zone.
+ _utc_tz = cctz::utc_time_zone();
+}
+
+paimon_predicate* PaimonRustPredicateConverter::build(const VExprContextSPtrs& conjuncts) {
+ _converted_conjuncts = 0;
+ _converted_runtime_filters = 0;
+ if (_table == nullptr) {
+ return nullptr;
+ }
+ predicate_ptr result;
+ for (const auto& conjunct : conjuncts) {
+ if (!conjunct || !conjunct->root()) {
+ continue;
+ }
+ auto root = conjunct->root();
+ const bool is_runtime_filter = root->is_rf_wrapper();
+ if (root->is_rf_wrapper()) {
+ if (auto impl = root->get_impl()) {
+ // A null-aware runtime filter (an EQ_FOR_NULL join) must stay
+ // residual: its wrapper execution restores NULL probe rows to
+ // true (RuntimeFilterExpr::change_null_to_true), while the
+ // unwrapped impl — rebuilt as an ordinary IN predicate through
+ // VDirectInPredicate::get_slot_in_expr — treats NULL as
+ // not-in-set and would prune the NULL probes permanently
+ // before the join sees them. Keep the wrapper itself: it fails
+ // every dispatch below, so the conjunct stays in the residual,
+ // and it is safe to execute on selected rows, so later
+ // conjuncts keep pushing. The lance pushdown declines
+ // is_null_aware() filters for the same reason.
+ // is_null_aware() is concrete on RuntimeFilterExpr (the only
+ // class whose is_rf_wrapper() is true), so the dynamic_cast
+ // never fails in practice; the null guard keeps the unwrap for
+ // any future wrapper shape.
+ auto* rf_wrapper = dynamic_cast<RuntimeFilterExpr*>(root.get());
+ if (rf_wrapper == nullptr || !rf_wrapper->is_null_aware()) {
+ root = impl;
+ }
+ }
+ }
+ // Preserve a safe prefix of the conjunct order: a later pushed
+ // predicate (e.g. an arrived IN runtime filter) could otherwise prune
+ // rows on which an earlier error-preserving conjunct —
+ // assert_true(...), a failing cast, ... — must still raise. The v1
+ // partition-pruning path (FileScanner::_init_runtime_filter_partition_
+ // prune_ctxs) stops at is_safe_to_execute_on_selected_rows() for the
+ // same reason, so a convertible predicate after an unsafe conjunct
+ // must not be pushed. Safe conjuncts that cannot be converted keep
+ // the old skip: they cannot raise, so pruning rows before they are
+ // evaluated as the residual never loses an error.
+ if (!_is_safe_to_push(root)) {
+ break;
+ }
+ predicate_ptr pred(_convert_expr(root));
+ if (!pred) {
+ continue;
+ }
+ ++_converted_conjuncts;
+ _converted_runtime_filters += is_runtime_filter;
+ if (!result) {
+ result = std::move(pred);
+ } else {
+ // and consumes both inputs regardless of success.
+ result.reset(paimon_predicate_and(result.release(), pred.release()));
+ if (!result) {
+ _converted_conjuncts = 0;
+ _converted_runtime_filters = 0;
+ return nullptr;
+ }
+ }
+ }
+ return result.release();
+}
+
+bool PaimonRustPredicateConverter::_is_safe_to_push(const VExprSPtr& expr) {
+ if (expr->is_safe_to_execute_on_selected_rows()) {
+ return true;
+ }
+ // VectorizedFnCall::is_safe_to_execute_on_selected_rows() admits a fixed
+ // whitelist that does not include `like`, so a like conjunct would always
+ // stop the safe prefix here and never reach _convert_like. A like call
+ // whose children are themselves safe is just as total as the whitelisted
+ // comparisons: it only compares strings, and its regex is built by escaping
+ // the pattern operand (FunctionLike::convert_like_pattern), so no scanned
+ // value can make it raise — a malformed pattern or escape operand fails
+ // at open() on every execution, regardless of which rows survive. Admit
+ // it here rather than widening the generic whitelist, which gates shared
+ // BE pushdown paths outside this reader's scope.
+ auto* fn = dynamic_cast<VectorizedFnCall*>(expr.get());
+ if (fn == nullptr || _normalize_name(fn->function_name()) != "like") {
+ return false;
+ }
+ for (uint16_t i = 0; i < fn->get_num_children(); ++i) {
+ if (!fn->get_child(i)->is_safe_to_execute_on_selected_rows()) {
+ return false;
+ }
+ }
+ return true;
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_convert_expr(const VExprSPtr& expr) {
+ if (!expr) {
+ return nullptr;
+ }
+
+ // Casts are not unwrapped anywhere (predicate root included): a cast node
+ // fails every dispatch below and the conjunct stays in the Doris residual,
+ // mirroring the FE converter, which keeps casted expressions unconverted.
+ if (auto* direct_in = dynamic_cast<VDirectInPredicate*>(expr.get())) {
+ VExprSPtr in_expr;
+ if (direct_in->get_slot_in_expr(in_expr)) {
+ return _convert_in(in_expr);
+ }
+ return nullptr;
+ }
+
+ if (dynamic_cast<VInPredicate*>(expr.get()) != nullptr) {
+ return _convert_in(expr);
+ }
+
+ switch (expr->op()) {
+ case TExprOpcode::COMPOUND_AND:
+ case TExprOpcode::COMPOUND_OR:
+ return _convert_compound(expr);
+ case TExprOpcode::COMPOUND_NOT:
+ return nullptr;
+ case TExprOpcode::EQ:
+ case TExprOpcode::EQ_FOR_NULL:
+ case TExprOpcode::NE:
+ case TExprOpcode::GE:
+ case TExprOpcode::GT:
+ case TExprOpcode::LE:
+ case TExprOpcode::LT:
+ return _convert_binary(expr);
+ default:
+ break;
+ }
+
+ if (auto* fn = dynamic_cast<VectorizedFnCall*>(expr.get())) {
+ auto fn_name = _normalize_name(fn->function_name());
+ if (fn_name == "is_null_pred" || fn_name == "is_not_null_pred") {
+ return _convert_is_null(expr, fn_name);
+ }
+ if (fn_name == "like") {
+ return _convert_like(expr);
+ }
+ }
+
+ return nullptr;
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_convert_compound(const VExprSPtr& expr) {
+ if (!expr || expr->get_num_children() != 2) {
+ return nullptr;
+ }
+ predicate_ptr left(_convert_expr(expr->get_child(0)));
+ if (!left) {
+ return nullptr;
+ }
+ predicate_ptr right(_convert_expr(expr->get_child(1)));
+ if (!right) {
+ return nullptr;
+ }
+
+ if (expr->op() == TExprOpcode::COMPOUND_AND) {
+ return paimon_predicate_and(left.release(), right.release());
+ }
+ if (expr->op() == TExprOpcode::COMPOUND_OR) {
+ return paimon_predicate_or(left.release(), right.release());
+ }
+ return nullptr;
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_convert_in(const VExprSPtr& expr) {
+ auto* in_pred = dynamic_cast<VInPredicate*>(expr.get());
+ // Runtime-generated IN sets can exceed VExpr's 16-bit child count. A wrapped
+ // count would push only a prefix and permanently discard valid probe rows.
+ if (!in_pred || expr->children().size() < 2 ||
+ expr->children().size() > std::numeric_limits<uint16_t>::max()) {
+ return nullptr;
+ }
+ auto field_meta = _resolve_field(expr->get_child(0));
+ if (!field_meta) {
+ return nullptr;
+ }
+
+ const auto num_values = expr->get_num_children() - 1;
+ // Reserve up front so the backing strings never reallocate: each datum's
+ // str_data points into storages[i], which must stay stable.
+ std::vector<std::string> storages;
+ std::vector<paimon_datum> datums;
+ storages.reserve(num_values);
+ datums.reserve(num_values);
+ for (uint16_t i = 1; i < expr->get_num_children(); ++i) {
+ // Casted list values are rejected by _convert_literal (the same rule
+ // as the binary RHS): with debug_skip_fold_constant the cast reaches
+ // the BE un-folded, and unwrapping it would filter on the pre-cast
+ // value — in `amount IN (CAST(1.24 AS DECIMAL(10,1)))` Doris keeps
+ // the 1.2 rows while the unwrapped 1.24 push removes them. Rejecting
+ // the value rejects the whole predicate; the residual applies the
+ // cast correctly.
+ auto holder = _convert_literal(expr->get_child(i), field_meta->type);
+ if (!holder) {
+ return nullptr;
+ }
+ storages.emplace_back(std::move(holder->storage));
+ paimon_datum datum = holder->datum;
+ _bind_datum_storage(&datum, storages.back());
+ datums.emplace_back(datum);
+ }
+
+ if (datums.empty()) {
+ return nullptr;
+ }
+ if (in_pred->is_not_in()) {
+ return _take(paimon_predicate_is_not_in(_table, field_meta->column.c_str(), datums.data(),
+ datums.size()));
+ }
+ return _take(paimon_predicate_is_in(_table, field_meta->column.c_str(), datums.data(),
+ datums.size()));
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_convert_binary(const VExprSPtr& expr) {
+ if (!expr || expr->get_num_children() != 2) {
+ return nullptr;
+ }
+ auto field_meta = _resolve_field(expr->get_child(0));
+ if (!field_meta) {
+ return nullptr;
+ }
+ const char* column = field_meta->column.c_str();
+
+ // Convert the RHS first so EQ_FOR_NULL (<=>) only converts when the RHS is
+ // a convertible literal, mirroring the FE converter, which rejects a
+ // non-literal RHS. A column-to-column `a <=> b` must therefore stay in the
+ // Doris residual: it has no single-column rust predicate, and pushing
+ // `a IS NULL` would wrongly discard rows like (1, 1) — rows dropped by the
+ // pushed filter cannot be recovered by the residual conjunct.
+ auto holder = _convert_literal(expr->get_child(1), field_meta->type);
+ if (!holder) {
+ return nullptr;
+ }
+
+ if (expr->op() == TExprOpcode::EQ_FOR_NULL) {
+ return _take(paimon_predicate_is_null(_table, column));
+ }
+
+ // `holder` is a local, so its storage stays put for the duration of the call.
+ _bind_datum_storage(&holder->datum, holder->storage);
+ const paimon_datum& datum = holder->datum;
+
+ switch (expr->op()) {
+ case TExprOpcode::EQ:
+ return _take(paimon_predicate_equal(_table, column, datum));
+ case TExprOpcode::NE:
+ return _take(paimon_predicate_not_equal(_table, column, datum));
+ case TExprOpcode::GE:
+ return _take(paimon_predicate_greater_or_equal(_table, column, datum));
+ case TExprOpcode::GT:
+ return _take(paimon_predicate_greater_than(_table, column, datum));
+ case TExprOpcode::LE:
+ return _take(paimon_predicate_less_or_equal(_table, column, datum));
+ case TExprOpcode::LT:
+ return _take(paimon_predicate_less_than(_table, column, datum));
+ default:
+ break;
+ }
+ return nullptr;
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_convert_is_null(const VExprSPtr& expr,
+ const std::string& fn_name) {
+ if (!expr || expr->get_num_children() != 1) {
+ return nullptr;
+ }
+ auto field_meta = _resolve_field(expr->get_child(0));
+ if (!field_meta) {
+ return nullptr;
+ }
+ if (fn_name == "is_not_null_pred") {
+ return _take(paimon_predicate_is_not_null(_table, field_meta->column.c_str()));
+ }
+ return _take(paimon_predicate_is_null(_table, field_meta->column.c_str()));
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_convert_like(const VExprSPtr& expr) {
+ if (!expr || expr->get_num_children() < 2) {
+ return nullptr;
+ }
+ auto field_meta = _resolve_field(expr->get_child(0));
+ if (!field_meta || !_is_string_type(field_meta->type->get_primitive_type())) {
+ return nullptr;
+ }
+
+ auto pattern_opt = _extract_string_literal(expr->get_child(1));
+ if (!pattern_opt) {
+ return nullptr;
+ }
+ const std::string& pattern = *pattern_opt;
+
+ // Doris's 3-arg like(col, pattern, escape) form: the pinned rust
+ // predicate only supports the backslash escape (its builder rejects any
+ // other escape character), so only the default-escape shape converts.
+ // The 2-arg form always carries the Doris/SQL default '\'.
+ if (expr->get_num_children() >= 3) {
+ auto escape_opt = _extract_string_literal(expr->get_child(2));
+ if (!escape_opt || escape_opt->size() != 1 || (*escape_opt)[0] != '\\') {
+ return nullptr;
+ }
+ }
+
+ // The pinned rust like implements SQL LIKE (% = any run, _ = one char,
+ // \X = literal X for any X — arrow's like kernel), and its builder
+ // optimizes `prefix%` / `%suffix` / `%mid%` shapes internally, so every
+ // pattern without a backslash matches Doris semantics exactly. The one
+ // divergence is the escape handling for a backslash before an ordinary
+ // character: Doris keeps both characters (only \%, \_ and \\ are
+ // escapes; `a\qb` matches a backslash followed by q b) while rust
+ // consumes the backslash (`a\qb` matches aqb) — pushing such a pattern
+ // would wrongly prune the Doris-matching rows before the residual can
+ // see them. Reject exactly the divergent shapes: every backslash must
+ // precede %, _ or \ (identical literal semantics on both sides) or end
+ // the pattern (literal backslash on both sides); anything else stays in
+ // the residual.
+ for (size_t i = 0; i < pattern.size(); ++i) {
+ if (pattern[i] == '\\') {
+ if (i + 1 >= pattern.size()) {
+ continue; // trailing backslash: literal on both sides
+ }
+ char next = pattern[i + 1];
+ if (next != '%' && next != '_' && next != '\\') {
+ return nullptr;
+ }
+ ++i;
+ }
+ }
+
+ paimon_datum datum {};
+ datum.tag = kTagString;
+ // `pattern` outlives the build call (it is a const ref into the caller's
+ // optional), so the datum may point straight at it.
+ _bind_datum_storage(&datum, pattern);
+ return _take(paimon_predicate_like(_table, field_meta->column.c_str(), datum, '\\'));
+}
+
+std::optional<PaimonRustPredicateConverter::FieldMeta> PaimonRustPredicateConverter::_resolve_field(
+ const VExprSPtr& expr) const {
+ if (!expr) {
+ return std::nullopt;
+ }
+ // Mirror the FE converter's convertDorisExprToSlotRef: a casted column is
+ // rejected, never unwrapped. Stripping a lossy cast changes which rows match
+ // — for a DECIMAL(10,2) column, CAST(amount AS DECIMAL(10,1)) = 1.2 keeps
+ // the row 1.24 while the unwrapped `amount = 1.2` prunes it — and rows
+ // pruned by the pushed filter cannot be recovered by the Doris residual.
+ // The conjunct stays in the residual instead.
+ auto* slot_ref = dynamic_cast<VSlotRef*>(expr.get());
+ if (!slot_ref) {
+ return std::nullopt;
+ }
+ // FileScannerV2 rewrites conjunct VSlotRefs to table global indices, so slot_id
+ // is a position, not a slot id; resolve by the carried column name against the
+ // projected-column registry instead of the desc table.
+ auto it = _columns_by_name.find(_normalize_name(slot_ref->column_name()));
+ if (it == _columns_by_name.end()) {
+ return std::nullopt;
+ }
+ const auto& [column, type] = it->second;
+ if (!_is_supported_slot_type(type->get_primitive_type())) {
+ return std::nullopt;
+ }
+ return FieldMeta {column, type};
+}
+
+std::optional<PaimonRustPredicateConverter::DatumHolder>
+PaimonRustPredicateConverter::_convert_literal(const VExprSPtr& expr,
+ const DataTypePtr& column_type) const {
+ // A casted literal is rejected, never unwrapped: the cast is not executed
+ // here (with constant folding disabled — debug_skip_fold_constant — it
+ // reaches the BE un-folded), so unwrapping would push the pre-cast value
+ // while the Doris residual compares against the cast result. A
+ // scale-reducing cast makes the two disagree — `amount =
+ // CAST(1.24 AS DECIMAL(10,1))` keeps a stored 1.20 row in Doris (1.24
+ // rounds down to 1.2) but the unwrapped `amount = 1.24` push prunes it, and
+ // rows pruned by the pushed filter cannot be recovered by the residual.
+ // The conjunct stays in the residual, where the cast is evaluated exactly.
+ if (expr->node_type() == TExprNodeType::CAST_EXPR) {
+ return std::nullopt;
+ }
+ auto* literal = dynamic_cast<VLiteral*>(expr.get());
+ if (!literal) {
+ return std::nullopt;
+ }
+
+ auto literal_type = remove_nullable(literal->get_data_type());
+ PrimitiveType literal_primitive = literal_type->get_primitive_type();
+ PrimitiveType slot_primitive = column_type->get_primitive_type();
+
+ ColumnPtr col = literal->get_column_ptr()->convert_to_full_column_if_const();
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(*col)) {
+ if (nullable->is_null_at(0)) {
+ return std::nullopt;
+ }
+ col = nullable->get_nested_column_ptr();
+ }
+
+ Field field;
+ col->get(0, field);
+
+ DatumHolder holder;
+ paimon_datum& datum = holder.datum;
+
+ switch (slot_primitive) {
+ case TYPE_BOOLEAN: {
+ if (literal_primitive != TYPE_BOOLEAN) {
+ return std::nullopt;
+ }
+ datum.tag = kTagBool;
+ datum.int_val = static_cast<bool>(field.get<TYPE_BOOLEAN>()) ? 1 : 0;
+ return holder;
+ }
+ case TYPE_TINYINT:
+ case TYPE_SMALLINT:
+ case TYPE_INT:
+ case TYPE_BIGINT: {
+ if (!_is_integer_type(literal_primitive)) {
+ return std::nullopt;
+ }
+ int64_t value = 0;
+ switch (literal_primitive) {
+ case TYPE_TINYINT:
+ value = field.get<TYPE_TINYINT>();
+ break;
+ case TYPE_SMALLINT:
+ value = field.get<TYPE_SMALLINT>();
+ break;
+ case TYPE_INT:
+ value = field.get<TYPE_INT>();
+ break;
+ case TYPE_BIGINT:
+ value = field.get<TYPE_BIGINT>();
+ break;
+ default:
+ return std::nullopt;
+ }
+ datum.int_val = value;
+ switch (slot_primitive) {
+ case TYPE_TINYINT:
+ datum.tag = kTagTinyInt;
+ break;
+ case TYPE_SMALLINT:
+ datum.tag = kTagSmallInt;
+ break;
+ case TYPE_INT:
+ datum.tag = kTagInt;
+ break;
+ default:
+ datum.tag = kTagLong;
+ break;
+ }
+ return holder;
+ }
+ case TYPE_DOUBLE: {
+ if (literal_primitive != TYPE_DOUBLE && literal_primitive != TYPE_FLOAT) {
+ return std::nullopt;
+ }
+ datum.tag = kTagDouble;
+ datum.double_val = literal_primitive == TYPE_FLOAT
+ ? static_cast<double>(field.get<TYPE_FLOAT>())
+ : field.get<TYPE_DOUBLE>();
+ return holder;
+ }
+ case TYPE_DATE:
+ case TYPE_DATEV2: {
+ if (!_is_date_type(literal_primitive)) {
+ return std::nullopt;
+ }
+ int64_t seconds = 0;
+ if (literal_primitive == TYPE_DATE) {
+ const auto& dt = field.get<TYPE_DATE>();
+ if (!dt.is_valid_date()) {
+ return std::nullopt;
+ }
+ dt.unix_timestamp(&seconds, _utc_tz);
+ } else {
+ const auto& dt = field.get<TYPE_DATEV2>();
+ if (!dt.is_valid_date()) {
+ return std::nullopt;
+ }
+ dt.unix_timestamp(&seconds, _utc_tz);
+ }
+ datum.tag = kTagDate;
+ datum.int_val = _seconds_to_days(seconds);
+ return holder;
+ }
+ case TYPE_DATETIME:
+ case TYPE_DATETIMEV2: {
+ if (!_is_datetime_type(literal_primitive)) {
+ return std::nullopt;
+ }
+ datum.tag = kTagTimestamp;
+ if (literal_primitive == TYPE_DATETIME) {
+ const auto& dt = field.get<TYPE_DATETIME>();
+ if (!dt.is_valid_date()) {
+ return std::nullopt;
+ }
+ int64_t seconds = 0;
+ dt.unix_timestamp(&seconds, _utc_tz);
+ // No sub-second part in datetime (v1): millis only, nanos stays 0.
+ datum.int_val = seconds * 1000;
+ } else {
+ const auto& dt = field.get<TYPE_DATETIMEV2>();
+ if (!dt.is_valid_date()) {
+ return std::nullopt;
+ }
+ // ts is (seconds since epoch, the microsecond-of-second part);
+ // split it into the rust timestamp's (millis, nanos): truncating
+ // the sub-millisecond remainder would make an equality/IN predicate
+ // more selective than the original conjunct (e.g. .123456 pushed as
+ // .123000), wrongly discarding matching rows. DATETIMEV2 carries at
+ // most microseconds, and micros % 1000 * 1000 <= 999000 fits the
+ // nanos field, so the value is always representable exactly.
+ std::pair<int64_t, int64_t> ts;
+ dt.unix_timestamp(&ts, _utc_tz);
+ datum.int_val = ts.first * 1000 + ts.second / 1000;
+ datum.int_val2 = (ts.second % 1000) * 1000;
+ }
+ return holder;
+ }
+ case TYPE_VARCHAR:
+ case TYPE_STRING: {
+ if (!_is_string_type(literal_primitive)) {
+ return std::nullopt;
+ }
+ const auto& value = field.get<TYPE_STRING>();
+ datum.tag = kTagString;
+ holder.storage.assign(value.data(), value.size());
+ return holder;
+ }
+ case TYPE_DECIMALV2:
+ case TYPE_DECIMAL32:
+ case TYPE_DECIMAL64:
+ case TYPE_DECIMAL128I:
+ case TYPE_DECIMAL256: {
+ if (!_is_decimal_type(literal_primitive)) {
+ return std::nullopt;
+ }
+ int32_t precision = static_cast<int32_t>(literal_type->get_precision());
+ int32_t scale = static_cast<int32_t>(literal_type->get_scale());
+ if (precision <= 0 || precision > kPaimonDecimalMaxPrecision) {
+ return std::nullopt;
+ }
+
+ __int128 value = 0;
+ switch (literal_primitive) {
+ case TYPE_DECIMALV2:
+ value = field.get<TYPE_DECIMALV2>().value();
+ break;
+ case TYPE_DECIMAL32:
+ value = field.get<TYPE_DECIMAL32>().value;
+ break;
+ case TYPE_DECIMAL64:
+ value = field.get<TYPE_DECIMAL64>().value;
+ break;
+ case TYPE_DECIMAL128I:
+ value = field.get<TYPE_DECIMAL128I>().value;
+ break;
+ default:
+ return std::nullopt;
+ }
+ datum.tag = kTagDecimal;
+ // rust reassembles as ((int_val2 as i128) << 64) | (int_val as u64 as i128):
+ // int_val holds the low 64 bits, int_val2 the (sign-extended) high 64 bits.
+ datum.int_val = static_cast<int64_t>(static_cast<uint64_t>(value));
+ datum.int_val2 = static_cast<int64_t>(value >> 64);
+ datum.uint_val = static_cast<uint32_t>(precision);
+ datum.uint_val2 = static_cast<uint32_t>(scale);
+ return holder;
+ }
+ default:
+ break;
+ }
+ return std::nullopt;
+}
+
+std::optional<std::string> PaimonRustPredicateConverter::_extract_string_literal(
+ const VExprSPtr& expr) const {
+ // Same rule as _convert_literal: a bare string literal converts, a casted
+ // one is rejected — the cast is not executed here, so the pushed pattern
+ // must come from the literal's own value. A truncating cast (e.g. to
+ // CHAR(k)) would otherwise push a different prefix than the residual
+ // matches. Cast trees stay in the residual.
+ if (expr->node_type() == TExprNodeType::CAST_EXPR) {
+ return std::nullopt;
+ }
+ auto* literal = dynamic_cast<VLiteral*>(expr.get());
+ if (!literal) {
+ return std::nullopt;
+ }
+ auto literal_type = remove_nullable(literal->get_data_type());
+ if (!_is_string_type(literal_type->get_primitive_type())) {
+ return std::nullopt;
+ }
+
+ ColumnPtr col = literal->get_column_ptr()->convert_to_full_column_if_const();
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(*col)) {
+ if (nullable->is_null_at(0)) {
+ return std::nullopt;
+ }
+ col = nullable->get_nested_column_ptr();
+ }
+ Field field;
+ col->get(0, field);
+ const auto& value = field.get<TYPE_STRING>();
+ return std::string(value.data(), value.size());
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_take(paimon_result_predicate result) {
+ if (result.error != nullptr) {
+ // A single leaf failing to build is not fatal: that conjunct is simply
+ // dropped from the pushed-down filter and the engine still re-applies it.
+ // Log at WARNING so it is visible without verbose logging enabled.
+ LOG(WARNING) << "paimon-rust build predicate failed: "
+ << consume_predicate_error(result.error);
+ return nullptr;
+ }
+ return result.predicate;
+}
+
+void PaimonRustPredicateConverter::_bind_datum_storage(paimon_datum* datum,
+ const std::string& storage) {
+ if (datum->tag == kTagString || datum->tag == kTagBytes) {
+ datum->str_data = reinterpret_cast<const uint8_t*>(storage.data());
+ datum->str_len = storage.size();
+ }
+}
+
+std::string PaimonRustPredicateConverter::_normalize_name(std::string_view name) {
+ std::string out(name);
+ std::transform(out.begin(), out.end(), out.begin(),
+ [](unsigned char c) { return static_cast<char>(std::tolower(c)); });
+ return out;
+}
+
+int32_t PaimonRustPredicateConverter::_seconds_to_days(int64_t seconds) {
+ static constexpr int64_t kSecondsPerDay = 24 * 60 * 60;
+ int64_t days = seconds / kSecondsPerDay;
+ if (seconds < 0 && seconds % kSecondsPerDay != 0) {
+ --days;
+ }
+ return static_cast<int32_t>(days);
+}
+
+bool PaimonRustPredicateConverter::_is_integer_type(PrimitiveType type) {
+ switch (type) {
+ case TYPE_TINYINT:
+ case TYPE_SMALLINT:
+ case TYPE_INT:
+ case TYPE_BIGINT:
+ return true;
+ default:
+ return false;
+ }
+}
+
+bool PaimonRustPredicateConverter::_is_string_type(PrimitiveType type) {
+ return type == TYPE_CHAR || type == TYPE_VARCHAR || type == TYPE_STRING;
+}
+
+bool PaimonRustPredicateConverter::_is_decimal_type(PrimitiveType type) {
+ switch (type) {
+ case TYPE_DECIMALV2:
+ case TYPE_DECIMAL32:
+ case TYPE_DECIMAL64:
+ case TYPE_DECIMAL128I:
+ case TYPE_DECIMAL256:
+ return true;
+ default:
+ return false;
+ }
+}
+
+bool PaimonRustPredicateConverter::_is_date_type(PrimitiveType type) {
+ return type == TYPE_DATE || type == TYPE_DATEV2;
+}
+
+bool PaimonRustPredicateConverter::_is_datetime_type(PrimitiveType type) {
+ return type == TYPE_DATETIME || type == TYPE_DATETIMEV2;
+}
+
+bool PaimonRustPredicateConverter::_is_supported_slot_type(PrimitiveType type) {
+ switch (type) {
+ case TYPE_BOOLEAN:
+ case TYPE_TINYINT:
+ case TYPE_SMALLINT:
+ case TYPE_INT:
+ case TYPE_BIGINT:
+ case TYPE_STRING:
+ case TYPE_DATE:
+ case TYPE_DATEV2:
+ return true;
+ case TYPE_VARCHAR:
+ case TYPE_DATETIME:
+ case TYPE_DATETIMEV2:
+ case TYPE_DECIMALV2:
+ case TYPE_DECIMAL32:
+ case TYPE_DECIMAL64:
+ case TYPE_DECIMAL128I:
+ case TYPE_DECIMAL256:
+ // Rust filters files/statistics before reconciling them to the current
+ // schema. The slot cannot prove that historical VARCHAR bounds/decimal
+ // scales or source timestamp precision match Doris. A current-domain IN
+ // member could reject a row that truncation/rescaling would make equal.
+ // Keep these predicates residual until per-file equivalence is known.
+ return false;
+ case TYPE_DOUBLE:
+ case TYPE_FLOAT:
+ // Rust uses IEEE partial ordering; Doris orders NaN above finite values
+ // and equal to itself. Pruned NaN rows cannot be restored by a residual.
+ return false;
+ case TYPE_CHAR:
+ default:
+ return false;
+ }
+}
+
+} // namespace doris
diff --git a/be/src/format_v2/table/paimon_rust_predicate_converter.h b/be/src/format_v2/table/paimon_rust_predicate_converter.h
new file mode 100644
index 00000000000..76c76037b51
--- /dev/null
+++ b/be/src/format_v2/table/paimon_rust_predicate_converter.h
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <cstdint>
+#include <optional>
+#include <string>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "cctz/time_zone.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/define_primitive_type.h"
+#include "exprs/vexpr_fwd.h"
+
+// paimon-rust C bindings. This header (and the paimon_* C API) is only pulled
+// into the rust-specific translation units (the converter, the rust reader and
+// its test); it is intentionally kept out of file_scanner and other generic BE
+// code, mirroring how paimon-cpp's headers are kept isolated.
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris {} // namespace doris
+
+namespace doris {
+
+// Converts Doris push-down conjuncts into a paimon-rust filter predicate.
+//
+// This mirrors PaimonPredicateConverter (the paimon-cpp variant) but targets the
+// paimon-rust C bindings. The key API differences drive the design:
+// * paimon-cpp builds predicates standalone via PredicateBuilder using a field
+// index + paimon::FieldType + paimon::Literal.
+// * paimon-rust builds predicates from a live paimon_table handle and a column
+// name (`paimon_predicate_equal(table, column, datum)`), resolving the field
+// index and type from the table schema itself, and takes typed values as a
+// flat tagged `paimon_datum`.
+//
+// Because the predicate functions need the opened table, the converter is
+// constructed with the table handle and must run after the table is open. The
+// returned `paimon_predicate*` is owned by the caller, who must either transfer
+// it to `paimon_read_builder_with_filter` (which consumes it) or release it via
+// `paimon_predicate_free`.
+//
+// Conversion preserves a safe prefix: conjuncts that are not safe to execute
+// on selected rows (assert_true(...), failing casts — anything that must keep
+// raising on rows a later pushed predicate could prune) stop the pushdown, so
+// no conjunct after them is pushed either; safe conjuncts that cannot be
+// represented are skipped (the engine still re-applies the full conjunct list
+// to the scanned block), so a partial filter only ever prunes a superset and
+// never changes results.
+class PaimonRustPredicateConverter {
+public:
+ // Resolve fields by column name from the supplied registry. FileScannerV2
+ // rewrites conjunct VSlotRefs to table global indices (positions, not slot
+ // ids), so a desc-table lookup must not run; the rewritten refs carry the
+ // original column name and data type instead. `column_names` should list only
+ // the columns the reader actually reads (e.g. exclude partition keys); the
+ // caller guarantees names and types are parallel.
+ PaimonRustPredicateConverter(const std::vector<std::string>& column_names,
+ const std::vector<DataTypePtr>& column_types,
+ const paimon_table* table);
+
+ // Builds an owned predicate AND-combining every convertible conjunct, or
+ // nullptr when nothing can be pushed down.
+ paimon_predicate* build(const VExprContextSPtrs& conjuncts);
+
+ // Counts top-level conjuncts in the returned filter, not its internal nodes.
+ size_t converted_conjuncts() const { return _converted_conjuncts; }
+ size_t converted_runtime_filters() const { return _converted_runtime_filters; }
+
+private:
+ struct FieldMeta {
+ // Original file column name passed verbatim to the paimon_predicate_*
+ // functions (rust resolves the field index/type from the schema by name).
+ std::string column;
+ // Column type used for literal conversion.
+ DataTypePtr type;
+ };
+
+ // A datum plus its backing byte storage. String/Bytes datums hold a borrowed
+ // pointer into `storage`; the storage must outlive the predicate build call,
+ // so callers keep the holder alive and (re)bind str_data after final placement.
+ struct DatumHolder {
+ // The `{}` value-initializes every paimon_datum field to zero. Conversion
+ // code relies on this: each branch only sets the fields its tag uses (e.g.
+ // a Timestamp sets int_val but leaves int_val2/nanos at 0), so the rest
+ // must already be zeroed. Do not drop the `{}`.
+ paimon_datum datum {};
+ std::string storage;
+ };
+
+ // Each returns an owned paimon_predicate* (transfer or free) or nullptr.
+ paimon_predicate* _convert_expr(const VExprSPtr& expr);
+ paimon_predicate* _convert_compound(const VExprSPtr& expr);
+ paimon_predicate* _convert_in(const VExprSPtr& expr);
+ paimon_predicate* _convert_binary(const VExprSPtr& expr);
+ paimon_predicate* _convert_is_null(const VExprSPtr& expr, const std::string& fn_name);
+ paimon_predicate* _convert_like(const VExprSPtr& expr);
+
+ std::optional<FieldMeta> _resolve_field(const VExprSPtr& expr) const;
+ // Whether the conjunct may join the pushed prefix: total over its input
+ // domain, on top of the generic is_safe_to_execute_on_selected_rows gate.
+ static bool _is_safe_to_push(const VExprSPtr& expr);
+ std::optional<DatumHolder> _convert_literal(const VExprSPtr& expr,
+ const DataTypePtr& column_type) const;
+ std::optional<std::string> _extract_string_literal(const VExprSPtr& expr) const;
+
+ // Consumes a paimon_result_predicate: frees the error (logging it) and
+ // returns the predicate, or nullptr on error.
+ static paimon_predicate* _take(paimon_result_predicate result);
+ // Point a String/Bytes datum's str_data/str_len at the given stable storage.
+ static void _bind_datum_storage(paimon_datum* datum, const std::string& storage);
+
+ static std::string _normalize_name(std::string_view name);
+ static int32_t _seconds_to_days(int64_t seconds);
+ static bool _is_integer_type(PrimitiveType type);
+ static bool _is_string_type(PrimitiveType type);
+ static bool _is_decimal_type(PrimitiveType type);
+ static bool _is_date_type(PrimitiveType type);
+ static bool _is_datetime_type(PrimitiveType type);
+ // Only push types whose file and Doris comparison domains are equivalent
+ // without per-file schema bounds, scale, or timestamp precision metadata.
+ static bool _is_supported_slot_type(PrimitiveType type);
+
+ std::unordered_map<std::string, std::pair<std::string, DataTypePtr>> _columns_by_name;
+ const paimon_table* _table = nullptr;
+ cctz::time_zone _utc_tz;
+ size_t _converted_conjuncts = 0;
+ size_t _converted_runtime_filters = 0;
+
+#ifdef BE_TEST
+ // Exposed for unit tests so they can inspect the converted paimon_datum
+ // (tag / int_val / int_val2) without a live predicate build. Declared
+ // here, after DatumHolder, because the return type is not in the
+ // complete-class context and cannot reference a member declared later.
+ std::optional<DatumHolder> TEST_convert_literal(const VExprSPtr& expr,
+ const DataTypePtr& column_type) const {
+ return _convert_literal(expr, column_type);
+ }
+#endif
+};
+
+} // namespace doris
diff --git a/be/src/format_v2/table/paimon_rust_table_reader.cpp b/be/src/format_v2/table/paimon_rust_table_reader.cpp
new file mode 100644
index 00000000000..731508957bb
--- /dev/null
+++ b/be/src/format_v2/table/paimon_rust_table_reader.cpp
@@ -0,0 +1,924 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn) \
+ struct type##_deleter { \
+ void operator()(paimon_##type* p) const { \
+ if (p) { \
+ freefn(p); \
+ } \
+ } \
+ }; \
+ using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+ explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+ ~ArrowBatch() {
+ auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+ auto* array = static_cast<ArrowArray*>(batch_.array);
+ if (array && array->release) {
+ array->release(array);
+ }
+ if (schema && schema->release) {
+ schema->release(schema);
+ }
+ paimon_arrow_batch_free(batch_);
+ }
+
+ ArrowBatch(const ArrowBatch&) = delete;
+ ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+ ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+ ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+ paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+ error_ptr owned(err);
+ if (!owned) {
+ return "unknown error";
+ }
+ std::string msg;
+ if (owned->message.data != nullptr && owned->message.len > 0) {
+ msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+ }
+ return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+ std::string out;
+ for (const auto& kv : options) {
+ if (!out.empty()) {
+ out += ", ";
+ }
+ out += kv.first;
+ }
+ return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+ table_ptr table;
+ read_builder_ptr read_builder;
+ plan_ptr plan;
+ table_read_ptr table_read;
+ record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+ RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+ {
+ // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+ // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.init_timer);
+ // Materialize TIMESTAMP_LTZ in the session timezone — the same
+ // convention as the JNI reader (PaimonJniScanner reads time_zone from
+ // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+ // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+ // of _ctz, so NTZ wall-clock semantics are preserved.
+ DORIS_CHECK(_runtime_state != nullptr);
+ _ctz = _runtime_state->timezone_obj();
+ if (_scanner_profile != nullptr) {
+ file_scan_profile::ensure_hierarchy(_scanner_profile);
+ _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+ file_scan_profile::TABLE_READER);
+ _rust_open_split_time =
+ ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+ _rust_read_batch_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+ _rust_arrow_to_block_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+ _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_converted = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+ }
+ // Projected column name -> fixed output position, registered with both the exact and
+ // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+ // semantics: exact match first, lower-case fallback on lookup).
+ _output_name_to_idx.reserve(_projected_columns.size() * 2);
+ for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+ _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+ _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+ }
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+ // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+ // are idempotent, and clear it only when a new split is explicitly prepared.
+ _close_split_reader();
+ _split_eof = false;
+ _current_range = options.current_range;
+ RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+ if (current_split_pruned()) {
+ return Status::OK();
+ }
+ if (_is_table_level_count_active()) {
+ // No rust pipeline is opened; get_block emits the synthetic count rows.
+ return Status::OK();
+ }
+ RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.prepare_split_timer);
+ // Open may dominate the scan or fail before get_block; include it in the parent timer.
+ SCOPED_TIMER(_rust_total_time);
+ SCOPED_TIMER(_rust_open_split_time);
+ RETURN_IF_ERROR(_open_split_reader(options.current_range));
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.exec_timer);
+ SCOPED_TIMER(_rust_total_time);
+ DORIS_CHECK(block != nullptr);
+ DORIS_CHECK(eos != nullptr);
+ DORIS_CHECK(block->columns() == _projected_columns.size());
+ block->clear_column_data(_projected_columns.size());
+ *eos = false;
+
+ if (_is_table_level_count_active()) {
+ return _read_table_level_count(block, eos);
+ }
+
+ // num_splits == 0 yields an empty (but valid) stream: report EOF.
+ if (_split_eof) {
+ *eos = true;
+ return Status::OK();
+ }
+ if (!_handles || !_handles->reader) {
+ return Status::InternalError("paimon-rust reader is not initialized");
+ }
+
+ while (true) {
+ // Mirror the base TableReader cancellation contract so a cancelled query does not
+ // drain the whole split.
+ if (_io_ctx != nullptr && _io_ctx->should_stop) {
+ _split_eof = true;
+ _close_split_reader();
+ *eos = true;
+ return Status::OK();
+ }
+
+ paimon_result_next_batch next;
+ {
+ SCOPED_TIMER(_rust_read_batch_time);
+ next = paimon_record_batch_reader_next(_handles->reader.get());
+ }
+ if (next.error != nullptr) {
+ return Status::InternalError("paimon-rust read batch failed: {}",
+ consume_error(next.error));
+ }
+ // End of stream: both pointers are null.
+ if (next.batch.array == nullptr && next.batch.schema == nullptr) {
+ _split_eof = true;
+ _close_split_reader();
+ *eos = true;
+ return Status::OK();
+ }
+
+ // RAII: the batch's Arrow release callbacks + container free run when
+ // `batch` leaves this scope, including on any early return.
+ ArrowBatch batch(next.batch);
+
+ auto* c_array = batch.array();
+ auto* c_schema = batch.schema();
+ arrow::Result<std::shared_ptr<arrow::RecordBatch>> import_result =
+ arrow::ImportRecordBatch(c_array, c_schema);
+ if (!import_result.ok()) {
+ return Status::InternalError("failed to import paimon-rust arrow batch: {}",
+ import_result.status().message());
+ }
+
+ auto record_batch = std::move(import_result).ValueUnsafe();
+ const auto rows = static_cast<size_t>(record_batch->num_rows());
+ if (rows == 0) {
+ // Skip empty batches and keep draining the stream.
+ continue;
+ }
+ RETURN_IF_ERROR(_fill_block_from_record_batch(record_batch, block, rows));
+ _record_scan_rows(rows);
+ *eos = false;
+ return Status::OK();
+ }
+}
+
+Status PaimonRustTableReader::abort_split() {
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.close_timer);
+ _close_split_reader();
+ _split_eof = false;
+ }
+ return format::TableReader::abort_split();
+}
+
+#ifdef BE_TEST
+std::string PaimonRustTableReader::TEST_format_options(
+ const std::map<std::string, std::string>& options) {
+ return format_options(options);
+}
+
+std::map<std::string, std::string> PaimonRustTableReader::TEST_build_options(
+ TFileScanRangeParams* scan_params, const TFileRangeDesc& range) {
+ TFileScanRangeParams* previous_params = _scan_params;
+ TFileRangeDesc previous_range = _current_range;
+ _scan_params = scan_params;
+ _current_range = range;
+ std::map<std::string, std::string> options = _build_options();
+ _scan_params = previous_params;
+ _current_range = std::move(previous_range);
+ return options;
+}
+#endif
+
+Status PaimonRustTableReader::close() {
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.close_timer);
+ _close_split_reader();
+ _close_table();
+ }
+ return format::TableReader::close();
+}
+
+Status PaimonRustTableReader::_validate_rust_split(const TFileRangeDesc& range) const {
+ if (!range.__isset.table_format_params || !range.table_format_params.__isset.paimon_params) {
+ return Status::InternalError(
+ "missing paimon_params for paimon rust reader, possibly caused by FE/BE protocol "
+ "mismatch");
+ }
+ const auto& params = range.table_format_params.paimon_params;
+ if (!params.__isset.paimon_split || params.paimon_split.empty()) {
+ return Status::InternalError(
+ "missing paimon_split for paimon rust reader, possibly caused by FE/BE protocol "
+ "mismatch");
+ }
+ if (params.__isset.reader_type && params.reader_type != TPaimonReaderType::PAIMON_RUST) {
+ return Status::InternalError(
+ "invalid reader_type for paimon rust reader, possibly caused by FE/BE protocol "
+ "mismatch");
+ }
+ if (!_resolve_table_path(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing paimon_table; cannot resolve paimon table location");
+ }
+ if (!_resolve_db_name(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing db_name; cannot open paimon table via schema json");
+ }
+ if (!_resolve_table_name(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing table_name; cannot open paimon table via schema json");
+ }
+ if (!_resolve_table_schema_json(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing paimon_table_schema_json; cannot open paimon table via "
+ "schema json");
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::_open_split_reader(const TFileRangeDesc& range) {
+ // 1. Decode the FE-planned split first so we fail fast (and without any
+ // filesystem IO) when it is missing or malformed.
+ std::string split_bytes;
+ RETURN_IF_ERROR(_decode_split_bytes(&split_bytes));
+
+ // 2. Resolve identifier + table_path + FE-supplied TableSchema JSON.
+ auto table_path = _resolve_table_path(range).value();
+ auto db_name = _resolve_db_name(range).value();
+ auto table_name = _resolve_table_name(range).value();
+ auto schema_json = _resolve_table_schema_json(range).value();
+ auto branch_opt = _resolve_branch(range);
+
+ // 3. Assemble storage options: FE-supplied paimon options + hadoop_conf +
+ // OSS/S3 → AWS_* translations. These feed FileIO only (per
+ // paimon_table_from_schema_json contract); they are NOT merged into the
+ // supplied table schema.
+ auto options = _build_options();
+
+ auto opened_table_key =
+ std::make_tuple(table_path, schema_json, db_name, table_name, branch_opt, options);
+ if (!_handles || !_handles->table || _opened_table_key != opened_table_key) {
+ // A paimon scan reads one table, so the handle is opened at most once per
+ // distinct identity (e.g. re-created after a close); splits of the same
+ // table reuse it and only rebuild the read pipeline below.
+ _close_table();
+ _handles = std::make_unique<PaimonHandles>();
+
+ std::vector<paimon_option> c_options;
+ c_options.reserve(options.size());
+ for (const auto& kv : options) {
+ c_options.push_back(paimon_option {kv.first.c_str(), kv.second.c_str()});
+ }
+
+ LOG(INFO) << "paimon-rust opening table via schema json: db=" << db_name
+ << " table=" << table_name << " path=" << table_path
+ << " branch=" << (branch_opt.has_value() ? branch_opt.value() : "main")
+ << " storage_options=[" << format_options(options) << "]";
+
+ // Build the table directly from the FE-supplied schema JSON. The Rust
+ // side rejects null / empty branch, so we default to paimon's canonical
+ // "main" sentinel when FE did not set paimon_branch (i.e. the table is
+ // on the main branch — matches upstream Identifier.DEFAULT_MAIN_BRANCH).
+ const std::string& branch_str = branch_opt.has_value() ? branch_opt.value() : "main";
+ paimon_result_get_table tbl_res = paimon_table_from_schema_json(
+ table_path.c_str(), schema_json.c_str(), db_name.c_str(), table_name.c_str(),
+ branch_str.c_str(), c_options.empty() ? nullptr : c_options.data(),
+ c_options.size());
+ if (tbl_res.error != nullptr) {
+ return Status::InternalError(
+ "paimon-rust table_from_schema_json failed: db={} table={} err={}", db_name,
+ table_name, consume_error(tbl_res.error));
+ }
+ _handles->table.reset(tbl_res.table);
+ _opened_table_key = std::move(opened_table_key);
+ }
+
+ // 4. Build the read pipeline: read_builder -> case-insensitive -> projection.
+ // Bound the Rust Arrow allocation itself: wide rows must respect the scanner's
+ // adaptive probe size before they are materialized into a Doris block.
+ const auto batch_size = std::to_string(
+ _batch_size > 0 ? _batch_size : std::max(1, _runtime_state->batch_size()));
+ const paimon_option batch_option {"read.batch-size", batch_size.c_str()};
+ paimon_result_read_builder rb_res =
+ paimon_table_new_read_builder_with_options(_handles->table.get(), &batch_option, 1);
+ if (rb_res.error != nullptr) {
+ return Status::InternalError("paimon-rust new read builder failed: {}",
+ consume_error(rb_res.error));
+ }
+ _handles->read_builder.reset(rb_res.read_builder);
+
+ // Fold column casing on the Rust side so FE-normalized lowercase names
+ // resolve against tables with mixed-case column definitions.
+ if (paimon_error* case_err =
+ paimon_read_builder_with_case_sensitive(_handles->read_builder.get(), false)) {
+ return Status::InternalError("paimon-rust set case_sensitive failed: {}",
+ consume_error(case_err));
+ }
+
+ // Partition keys are excluded: they are materialized from split metadata
+ // (see _fill_non_arrow_columns), and paimon-rust does not emit them.
+ auto read_columns = _build_read_columns();
+ std::vector<const char*> projection;
+ projection.reserve(read_columns.size() + 1);
+ for (const auto& col : read_columns) {
+ projection.push_back(col.c_str());
+ }
+ projection.push_back(nullptr);
+ if (paimon_error* proj_err = paimon_read_builder_with_projection(_handles->read_builder.get(),
+ projection.data())) {
+ return Status::InternalError("paimon-rust set projection failed: {}",
+ consume_error(proj_err));
+ }
+
+ // Convert the scanner conjuncts into a paimon-rust filter and apply it.
+ RETURN_IF_ERROR(_apply_predicate());
+
+ // 5. Deserialize the FE-planned split into a one-split plan, so this
+ // scanner reads exactly the split it was assigned rather than replanning
+ // the whole table. The wire form is identical to what paimon-cpp consumes
+ // (`paimon::table::DataSplit::serialize`).
+ paimon_result_plan plan_res = paimon_plan_from_split_bytes(
+ reinterpret_cast<const uint8_t*>(split_bytes.data()), split_bytes.size());
+ if (plan_res.error != nullptr) {
+ return Status::InternalError("paimon-rust build plan failed: {}",
+ consume_error(plan_res.error));
+ }
+ _handles->plan.reset(plan_res.plan);
+
+ size_t num_splits = paimon_plan_num_splits(_handles->plan.get());
+ if (num_splits == 0) {
+ _split_eof = true;
+ return Status::OK();
+ }
+
+ // 6. Open the arrow stream over the plan.
+ paimon_result_new_read read_res = paimon_read_builder_new_read(_handles->read_builder.get());
+ if (read_res.error != nullptr) {
+ return Status::InternalError("paimon-rust new read failed: {}",
+ consume_error(read_res.error));
+ }
+ _handles->table_read.reset(read_res.read);
+
+ paimon_result_record_batch_reader rdr_res = paimon_table_read_to_arrow(
+ _handles->table_read.get(), _handles->plan.get(), /*offset=*/0, /*length=*/num_splits);
+ if (rdr_res.error != nullptr) {
+ return Status::InternalError("paimon-rust open arrow reader failed: {}",
+ consume_error(rdr_res.error));
+ }
+ _handles->reader.reset(rdr_res.reader);
+ return Status::OK();
+}
+
+void PaimonRustTableReader::_close_split_reader() {
+ if (!_handles) {
+ return;
+ }
+ // Reverse of the declaration order in PaimonHandles.
+ _handles->reader.reset();
+ _handles->table_read.reset();
+ _handles->plan.reset();
+ _handles->read_builder.reset();
+}
+
+void PaimonRustTableReader::_close_table() {
+ if (!_handles) {
+ return;
+ }
+ _close_split_reader();
+ _handles->table.reset();
+ _opened_table_key.reset();
+}
+
+Status PaimonRustTableReader::_apply_predicate() {
+ if (_conjuncts.empty() || !_handles || !_handles->table || !_handles->read_builder) {
+ return Status::OK();
+ }
+ if (_scanner_profile != nullptr) {
+ COUNTER_UPDATE(_rust_predicates_input, _conjuncts.size());
+ for (const auto& conjunct : _conjuncts) {
+ if (conjunct && conjunct->root() && conjunct->root()->is_rf_wrapper()) {
+ COUNTER_UPDATE(_rust_runtime_filters_input, 1);
+ }
+ }
+ }
+ LOG(INFO) << "paimon-rust predicate pushdown: " << _conjuncts.size() << " conjunct(s) input";
+ // The conjunct VSlotRefs carry table global indices (positions), so the v2
+ // converter mode resolves fields by the projected column names; partition
+ // keys are excluded because the rust reader does not read them.
+ std::vector<std::string> names;
+ std::vector<DataTypePtr> types;
+ names.reserve(_projected_columns.size());
+ types.reserve(_projected_columns.size());
+ for (const auto& col : _projected_columns) {
+ if (col.is_partition_key) {
+ continue;
+ }
+ names.push_back(col.name);
+ types.push_back(col.type);
+ }
+ PaimonRustPredicateConverter converter(names, types, _handles->table.get());
+ paimon_predicate* predicate = converter.build(_conjuncts);
+ if (_scanner_profile != nullptr) {
+ COUNTER_UPDATE(_rust_predicates_converted, converter.converted_conjuncts());
+ }
+ if (predicate == nullptr) {
+ LOG(INFO) << "paimon-rust predicate pushdown: nothing convertible, no filter applied";
+ return Status::OK();
+ }
+ // paimon_read_builder_with_filter consumes the predicate (ownership moves to
+ // the builder) on every path, so we must not free it here.
+ if (paimon_error* err =
+ paimon_read_builder_with_filter(_handles->read_builder.get(), predicate)) {
+ return Status::InternalError("paimon-rust apply filter failed: {}", consume_error(err));
+ }
+ // Count application only after the C API accepts the filter; reader timers
+ // alone cannot distinguish pushdown from Doris residual-only execution.
+ if (_scanner_profile != nullptr) {
+ COUNTER_UPDATE(_rust_predicates_applied, converter.converted_conjuncts());
+ COUNTER_UPDATE(_rust_runtime_filters_applied, converter.converted_runtime_filters());
+ }
+ LOG(INFO) << "paimon-rust predicate pushdown: applied";
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::_fill_block_from_record_batch(
+ const std::shared_ptr<arrow::RecordBatch>& batch, Block* block, size_t rows) {
+ SCOPED_TIMER(_rust_arrow_to_block_time);
+ DORIS_CHECK(batch != nullptr);
+ DORIS_CHECK(block != nullptr);
+ std::unordered_set<size_t> materialized_indices;
+ materialized_indices.reserve(_projected_columns.size());
+ {
+ auto columns_guard = block->mutate_columns_scoped();
+ auto& columns = columns_guard.mutable_columns();
+ for (int c = 0; c < batch->num_columns(); ++c) {
+ const auto& field = batch->schema()->field(c);
+ if (field->name() == VALUE_KIND_FIELD) {
+ continue;
+ }
+ // Projected column names are FE-normalized to lowercase.
+ // paimon-rust's case_sensitive=false setting also case-folds column
+ // names in the schema output, so exact match works — but tolerate
+ // mixed-case Rust output by folding here as well.
+ auto it = _output_name_to_idx.find(field->name());
+ if (it == _output_name_to_idx.end()) {
+ it = _output_name_to_idx.find(to_lower(field->name()));
+ }
+ if (it == _output_name_to_idx.end()) {
+ // Skip columns that are not in the block (e.g. columns dropped by
+ // slot pruning).
+ continue;
+ }
+ const auto output_idx = it->second;
+ if (!materialized_indices.emplace(output_idx).second) {
+ return Status::InternalError("paimon-rust returned duplicate column '{}'",
+ field->name());
+ }
+ try {
+ RETURN_IF_ERROR(columns_guard.get_datatype_by_position(output_idx)
+ ->get_serde()
+ ->read_column_from_arrow(*columns[output_idx],
+ batch->column(c).get(), 0, rows,
+ _ctz));
+ } catch (Exception& e) {
+ return Status::InternalError("Failed to convert from arrow to block: {}", e.what());
+ }
+ }
+ }
+ // Partition columns and other projected columns absent from the arrow batch
+ // are back-filled from split metadata / defaults.
+ RETURN_IF_ERROR(_fill_non_arrow_columns(block, rows, materialized_indices));
+ // This direct Arrow path bypasses TableReader::finalize_chunk, whose last
+ // step enforces truncate_char_or_varchar_columns — without it, a column
+ // narrowed by schema evolution returns untruncated historical values.
+ RETURN_IF_ERROR(_truncate_char_or_varchar_columns(block));
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::_truncate_char_or_varchar_columns(Block* block) {
+ if (_runtime_state == nullptr ||
+ !_runtime_state->query_options().truncate_char_or_varchar_columns) {
+ return Status::OK();
+ }
+ for (size_t idx = 0; idx < block->columns(); ++idx) {
+ const auto& column_type = block->get_by_position(idx).type;
+ if (column_type == nullptr) {
+ continue;
+ }
+ const auto type = remove_nullable(column_type);
+ const auto primitive = type->get_primitive_type();
+ if (primitive != TYPE_VARCHAR && primitive != TYPE_CHAR) {
+ continue;
+ }
+ const auto target_len = assert_cast<const DataTypeString*>(type.get())->len();
+ if (target_len <= 0) {
+ continue;
+ }
+ // Reuses TableReader's vectorized truncation (substring(column, 1,
+ // len)); the base variant maps through column_mapper metadata, which
+ // the direct rust path does not populate, so iterate the block's own
+ // slot-derived types here — the arrow side is always lengthless Utf8,
+ // so any bounded CHAR/VARCHAR target truncates to its declared length.
+ // Partition/default expressions can produce ColumnConst(ColumnNullable). The shared
+ // truncation helper expects a materialized nullable column, as in finalize_chunk.
+ auto& column = block->get_by_position(idx).column;
+ column = column->convert_to_full_column_if_const();
+ _truncate_char_or_varchar_column(block, idx, target_len);
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::_fill_non_arrow_columns(
+ Block* block, size_t rows, const std::unordered_set<size_t>& materialized_indices) {
+ for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+ if (materialized_indices.count(idx) != 0) {
+ continue;
+ }
+ const auto& column = _projected_columns[idx];
+ VExprContextSPtr constant_expr;
+ if (const Field* value = find_partition_value(column, _partition_values);
+ column.is_partition_key && value != nullptr) {
+ // Partition values are split constants (same materialization the
+ // TableColumnMapper builds for native readers).
+ constant_expr =
+ VExprContext::create_shared(VLiteral::create_shared(column.type, *value));
+ } else if (column.default_expr != nullptr) {
+ constant_expr = column.default_expr;
+ } else {
+ // The column is genuinely absent from the arrow batch. Schema
+ // evolution is handled by paimon-rust itself, so reaching here means
+ // an unexpected schema drift: fill defaults so the scan remains
+ // well-defined instead of failing the query.
+ LOG(WARNING) << "paimon-rust did not return projected column '" << column.name
+ << "'; filling with defaults";
+ auto data = column.type->create_column();
+ data->insert_many_defaults(rows);
+ block->replace_by_position(idx, std::move(data));
+ continue;
+ }
+ ColumnPtr constant_column;
+ RETURN_IF_ERROR(_materialize_constant_column(constant_expr, column.type, column.name, rows,
+ &constant_column));
+ block->replace_by_position(idx, std::move(constant_column));
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::_materialize_constant_column(const VExprContextSPtr& expr,
+ const DataTypePtr& type,
+ const std::string& name, size_t rows,
+ ColumnPtr* column) {
+ DORIS_CHECK(expr != nullptr);
+ DORIS_CHECK(column != nullptr);
+ RowDescriptor row_desc;
+ RETURN_IF_ERROR(expr->prepare(_runtime_state, row_desc));
+ RETURN_IF_ERROR(expr->open(_runtime_state));
+ // Constants evaluate per input row, so a rows-sized synthetic block yields a
+ // rows-sized result for both plain literals and default expressions.
+ Block eval_block;
+ eval_block.insert({type->create_column_const_with_default_value(rows), type, name});
+ int result_column_id = -1;
+ RETURN_IF_ERROR(expr->execute(&eval_block, &result_column_id));
+ DORIS_CHECK(result_column_id >= 0);
+ ColumnPtr result_column = eval_block.get_by_position(result_column_id).column;
+ if (result_column->size() == 1 && rows > 1) {
+ result_column = ColumnConst::create(std::move(result_column), rows);
+ }
+ *column = std::move(result_column);
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::_decode_split_bytes(std::string* out) const {
+ if (!_current_range.__isset.table_format_params ||
+ !_current_range.table_format_params.__isset.paimon_params ||
+ !_current_range.table_format_params.paimon_params.__isset.paimon_split) {
+ return Status::InternalError("paimon-rust missing paimon_split in scan range");
+ }
+ const auto& encoded_split = _current_range.table_format_params.paimon_params.paimon_split;
+ if (!base64_decode(encoded_split, out)) {
+ return Status::InternalError("paimon-rust base64 decode paimon_split failed");
+ }
+ if (out->empty()) {
+ return Status::InternalError("paimon-rust decoded paimon_split is empty");
+ }
+ return Status::OK();
+}
+
+std::optional<std::string> PaimonRustTableReader::_resolve_table_path(
+ const TFileRangeDesc& range) const {
+ if (range.__isset.table_format_params && range.table_format_params.__isset.paimon_params &&
+ range.table_format_params.paimon_params.__isset.paimon_table &&
+ !range.table_format_params.paimon_params.paimon_table.empty()) {
+ return range.table_format_params.paimon_params.paimon_table;
+ }
+ return std::nullopt;
+}
+
+std::optional<std::string> PaimonRustTableReader::_resolve_db_name(
+ const TFileRangeDesc& range) const {
+ if (range.__isset.table_format_params && range.table_format_params.__isset.paimon_params &&
+ range.table_format_params.paimon_params.__isset.db_name &&
+ !range.table_format_params.paimon_params.db_name.empty()) {
+ return range.table_format_params.paimon_params.db_name;
+ }
+ return std::nullopt;
+}
+
+std::optional<std::string> PaimonRustTableReader::_resolve_table_name(
+ const TFileRangeDesc& range) const {
+ if (range.__isset.table_format_params && range.table_format_params.__isset.paimon_params &&
+ range.table_format_params.paimon_params.__isset.table_name &&
+ !range.table_format_params.paimon_params.table_name.empty()) {
+ return range.table_format_params.paimon_params.table_name;
+ }
+ return std::nullopt;
+}
+
+std::optional<std::string> PaimonRustTableReader::_resolve_table_schema_json(
+ const TFileRangeDesc& range) const {
+ if (range.__isset.table_format_params && range.table_format_params.__isset.paimon_params &&
+ range.table_format_params.paimon_params.__isset.paimon_table_schema_json &&
+ !range.table_format_params.paimon_params.paimon_table_schema_json.empty()) {
+ return range.table_format_params.paimon_params.paimon_table_schema_json;
+ }
+ return std::nullopt;
+}
+
+std::optional<std::string> PaimonRustTableReader::_resolve_branch(
+ const TFileRangeDesc& range) const {
+ // FE only sets paimon_branch when the branch is not `main` (matches
+ // upstream paimon commit 742da63: null-if-DEFAULT_MAIN_BRANCH). Unset here
+ // means main-branch semantics.
+ if (range.__isset.table_format_params && range.table_format_params.__isset.paimon_params &&
+ range.table_format_params.paimon_params.__isset.paimon_branch &&
+ !range.table_format_params.paimon_params.paimon_branch.empty()) {
+ return range.table_format_params.paimon_params.paimon_branch;
+ }
+ return std::nullopt;
+}
+
+std::vector<std::string> PaimonRustTableReader::_build_read_columns() const {
+ std::vector<std::string> columns;
+ columns.reserve(_projected_columns.size());
+ for (const auto& column : _projected_columns) {
+ if (column.is_partition_key) {
+ continue;
+ }
+ columns.emplace_back(column.name);
+ }
+ return columns;
+}
+
+std::map<std::string, std::string> PaimonRustTableReader::_build_options() const {
+ std::map<std::string, std::string> options;
+ if (_scan_params && _scan_params->__isset.paimon_options &&
+ !_scan_params->paimon_options.empty()) {
+ options.insert(_scan_params->paimon_options.begin(), _scan_params->paimon_options.end());
+ } else if (_current_range.__isset.table_format_params &&
+ _current_range.table_format_params.__isset.paimon_params &&
+ _current_range.table_format_params.paimon_params.__isset.paimon_options) {
+ options.insert(_current_range.table_format_params.paimon_params.paimon_options.begin(),
+ _current_range.table_format_params.paimon_params.paimon_options.end());
+ }
+
+ if (_scan_params && _scan_params->__isset.properties && !_scan_params->properties.empty()) {
+ for (const auto& kv : _scan_params->properties) {
+ options[kv.first] = kv.second;
+ }
+ } else if (_current_range.__isset.table_format_params &&
+ _current_range.table_format_params.__isset.paimon_params &&
+ _current_range.table_format_params.paimon_params.__isset.hadoop_conf) {
+ for (const auto& kv : _current_range.table_format_params.paimon_params.hadoop_conf) {
+ options[kv.first] = kv.second;
+ }
+ }
+
+ auto copy_if_missing = [&](const char* from_key, const char* to_key) {
+ if (options.find(to_key) != options.end()) {
+ return;
+ }
+ auto it = options.find(from_key);
+ if (it != options.end() && !it->second.empty()) {
+ options[to_key] = it->second;
+ }
+ };
+
+ // The pinned paimon-rust storage dispatcher (io/storage.rs) selects the
+ // FileIO parser from the table path's URI scheme, and libpaimon_c.a
+ // compiles in separate COS, OBS, GCS and Azdls parsers besides the OSS
+ // and S3 ones. Doris's FE normalizes every object store's credentials
+ // into the AWS_* / use_path_style aliases, which this bridge can only
+ // translate into the two key families those two parsers read:
+ // oss:// -> fs.oss.endpoint / fs.oss.accessKeyId /
+ // fs.oss.accessKeySecret (+ fs.oss.securityToken STS)
+ // s3:// / s3a:// -> paimon-java's s3.* family (access-key, secret-key,
+ // session.token, endpoint, region, path-style-access,
+ // normalized from the fs.s3a. / s3a. / s3. prefixes)
+ // The FE therefore gates the rust reader to exactly these schemes
+ // (PaimonScanNode: cosn:// / obs:// / gs:// / abfs:// warehouses fall
+ // back to JNI before the split is encoded — their parsers read the
+ // fs.cosn.userinfo.* / fs.obs.* / gcs.* / azure.* families the AWS_*
+ // aliases cannot express). The mapping here is scoped the same way so a
+ // version-skewed FE cannot smuggle AWS_* aliases into another parser's
+ // property map: a cosn:// table reaching this bridge with synthesized
+ // s3.* keys would hit the COS parser without
+ // fs.cosn.userinfo.secretId / secretKey and fail the open with a
+ // misleading auth error instead of the JNI fallback.
+ const std::string table_path = _resolve_table_path(_current_range).value_or("");
+ std::string scheme;
+ if (const auto sep = table_path.find("://"); sep != std::string::npos) {
+ scheme = to_lower(table_path.substr(0, sep));
+ }
+ if (scheme == "oss") {
+ // The OSS parser reads only the four fs.oss.* keys (and retry
+ // settings); native fs.oss.* options pass through untouched. It has
+ // no region / anonymous / assume-role handling, so nothing else is
+ // mapped for this scheme.
+ copy_if_missing("AWS_ENDPOINT", "fs.oss.endpoint");
+ copy_if_missing("AWS_ACCESS_KEY", "fs.oss.accessKeyId");
+ copy_if_missing("AWS_SECRET_KEY", "fs.oss.accessKeySecret");
+ copy_if_missing("AWS_TOKEN", "fs.oss.securityToken");
+ return options;
+ }
+ if (scheme == "s3" || scheme == "s3a") {
+ copy_if_missing("AWS_ACCESS_KEY", "s3.access-key");
+ copy_if_missing("AWS_SECRET_KEY", "s3.secret-key");
+ copy_if_missing("AWS_TOKEN", "s3.session.token");
+ copy_if_missing("AWS_ENDPOINT", "s3.endpoint");
+ copy_if_missing("AWS_REGION", "s3.region");
+ copy_if_missing("use_path_style", "s3.path-style-access");
+ // Authentication modes: the FE storage-properties channel marks anonymous
+ // access with AWS_CREDENTIALS_PROVIDER_TYPE=ANONYMOUS (emitted when no
+ // static credentials are configured) and assume-role with
+ // AWS_ROLE_ARN / AWS_EXTERNAL_ID (from the s3.role_arn / s3.external_id
+ // catalog properties). The crate reads s3.anonymous (skip_signature) and
+ // the s3.assumed.role.* family, so map both; without these, anonymous
+ // catalogs would consult the ambient credential chain and role-only
+ // catalogs would never assume the requested role. The remaining provider
+ // modes are ambient JVM credential chains (ENV, SYSTEM_PROPERTIES,
+ // WEB_IDENTITY, CONTAINER, INSTANCE_PROFILE) with no paimon-rust
+ // equivalent — the FE gates those away from the rust reader before the
+ // split is encoded.
+ if (options.contains("AWS_CREDENTIALS_PROVIDER_TYPE") &&
+ options.at("AWS_CREDENTIALS_PROVIDER_TYPE") == "ANONYMOUS") {
+ options["s3.anonymous"] = "true";
+ }
+ copy_if_missing("AWS_ROLE_ARN", "s3.assumed.role.arn");
+ copy_if_missing("AWS_EXTERNAL_ID", "s3.assumed.role.externalId");
+ // OSS-shaped options on an S3 warehouse (cross-protocol alias): map them
+ // to the s3.* family as well.
+ copy_if_missing("fs.oss.accessKeyId", "s3.access-key");
+ copy_if_missing("fs.oss.accessKeySecret", "s3.secret-key");
+ copy_if_missing("fs.oss.sessionToken", "s3.session.token");
+ copy_if_missing("fs.oss.endpoint", "s3.endpoint");
+ copy_if_missing("fs.oss.region", "s3.region");
+ }
+ // Every other scheme passes through untouched: hdfs:// tables read their
+ // hadoop conf as delivered, scheme-less / file:// paths need no
+ // credentials, and the COS / OBS / GCS / Azdls families are gated to JNI
+ // on the FE — synthesizing s3.* keys for any of them would be dead
+ // weight at best and a silent misconfiguration at worst.
+ return options;
+}
+
+} // namespace doris::format::paimon
diff --git a/be/src/format_v2/table/paimon_rust_table_reader.h b/be/src/format_v2/table/paimon_rust_table_reader.h
new file mode 100644
index 00000000000..0c04f2f1f04
--- /dev/null
+++ b/be/src/format_v2/table/paimon_rust_table_reader.h
@@ -0,0 +1,188 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <map>
+#include <memory>
+#include <optional>
+#include <string>
+#include <tuple>
+#include <unordered_map>
+#include <unordered_set>
+#include <vector>
+
+#include "cctz/time_zone.h"
+#include "common/status.h"
+#include "format_v2/table_reader.h"
+#include "runtime/runtime_profile.h"
+
+namespace arrow {
+class RecordBatch;
+}
+
+namespace doris::format::paimon {
+
+// Reads one FE-planned Paimon DataSplit per split through the paimon-rust C
+// bindings (libpaimon_c), using the schema-json pipeline:
+//
+// paimon_table_from_schema_json(table_path, schema_json, db, table, branch, options)
+// -> read_builder (case-insensitive, projection, filter)
+// -> paimon_plan_from_split_bytes(base64-decoded FE split)
+// -> paimon_read_builder_new_read -> paimon_table_read_to_arrow
+// -> per-batch paimon_record_batch_reader_next (Arrow C Data Interface).
+//
+// Leaf reader inside PaimonHybridReader: a DataSplit is a logical multi-file
+// split, not a file range, so this reader does not go through
+// FileReader/TableColumnMapper and fills the table-schema output block directly
+// (same shape as LanceTableReader). Partition columns that paimon-rust does not
+// emit are materialized from the partition values captured by the TableReader
+// base class; columns with default expressions are materialized from those
+// expressions.
+class PaimonRustTableReader final : public format::TableReader {
+public:
+ // Out-of-line (in the .cpp) so TUs that construct or destroy the reader do not
+ // need the complete PaimonHandles pimpl type.
+ PaimonRustTableReader();
+ ~PaimonRustTableReader() override;
+
+ Status init(format::TableReadOptions&& options) override;
+ // Each split's Rust read builder applies the batch size stored by the base class
+ // before opening the Arrow reader.
+ Status prepare_split(const format::SplitReadOptions& options) override;
+ Status get_block(Block* block, bool* eos) override;
+ Status abort_split() override;
+ Status close() override;
+
+#ifdef BE_TEST
+ Status TEST_validate_rust_split(const TFileRangeDesc& range) const {
+ return _validate_rust_split(range);
+ }
+ Status TEST_fill_non_arrow_columns(
+ Block* block, size_t rows,
+ const std::unordered_set<size_t>& materialized_indices = {}) {
+ return _fill_non_arrow_columns(block, rows, materialized_indices);
+ }
+ // The direct-path bounded-string truncation, exposed for unit tests.
+ Status TEST_truncate_char_or_varchar_columns(Block* block) {
+ return _truncate_char_or_varchar_columns(block);
+ }
+ void TEST_set_projected_columns(std::vector<format::ColumnDefinition> columns) {
+ _projected_columns = std::move(columns);
+ }
+ void TEST_set_partition_values(std::map<std::string, Field> values) {
+ _partition_values = std::move(values);
+ }
+ // The session timezone the reader materializes TIMESTAMP_LTZ with.
+ const cctz::time_zone& TEST_ctz() const { return _ctz; }
+ // Exposed for unit tests so the keys-only option-logging contract can be
+ // checked without a live table open.
+ static std::string TEST_format_options(const std::map<std::string, std::string>& options);
+ // Storage options the reader would build for the given FE-planned scan
+ // params and split (S3 key remapping, provider modes, scheme dispatch);
+ // both must outlive the call. Exposed for unit tests.
+ std::map<std::string, std::string> TEST_build_options(TFileScanRangeParams* scan_params,
+ const TFileRangeDesc& range);
+#endif
+
+private:
+ // Opaque holder for the paimon-rust C handles (defined in the .cpp so that
+ // paimon_rust/paimon.h does not leak into other translation units).
+ struct PaimonHandles;
+
+ // Fail fast (without filesystem IO) when the schema-json pipeline fields are
+ // missing from the split; the error text points at an FE/BE protocol mismatch.
+ Status _validate_rust_split(const TFileRangeDesc& range) const;
+ // Open (or reuse the cached) paimon_table and build the per-split read
+ // pipeline: read_builder -> projection -> filter -> plan -> arrow reader.
+ Status _open_split_reader(const TFileRangeDesc& range);
+ // Drop the per-split pipeline (read_builder .. record batch reader), keeping
+ // the cached table handle for the next split of the same table.
+ void _close_split_reader();
+ // Drop the cached paimon_table handle.
+ void _close_table();
+ // Convert _conjuncts (global-index VSlotRefs) into a paimon-rust filter and
+ // apply it to the read builder. Best effort: non-convertible conjuncts are
+ // dropped silently and re-applied by the scanner row-level filter.
+ Status _apply_predicate();
+ // Fill the output block from one arrow record batch: arrow columns are
+ // matched by name (exact then lower-case, v1 semantics) onto fixed projected
+ // positions, then _fill_non_arrow_columns back-fills the rest.
+ Status _fill_block_from_record_batch(const std::shared_ptr<arrow::RecordBatch>& batch,
+ Block* block, size_t rows);
+ // Materialize projected columns the arrow batch did not contain: partition
+ // constants, default expressions, or a default/NULL fill as a last resort.
+ Status _fill_non_arrow_columns(Block* block, size_t rows,
+ const std::unordered_set<size_t>& materialized_indices);
+ // Enforce truncate_char_or_varchar_columns on the direct Arrow path: the
+ // pinned rust reader maps paimon CHAR(n)/VARCHAR(n) to lengthless Arrow
+ // Utf8, so a column narrowed by schema evolution would return untruncated
+ // historical values here while the normal (finalize_chunk) path truncates.
+ // The block's own column types carry the table-side bounds (the reader
+ // builds them from the FE slot types); the arrow side is always boundless,
+ // so every bounded CHAR/VARCHAR column truncates to its declared length.
+ Status _truncate_char_or_varchar_columns(Block* block);
+ // Evaluate a constant expression (VLiteral or default_expr) on a one-row
+ // block and broadcast the result to `rows` as a ColumnConst.
+ Status _materialize_constant_column(const VExprContextSPtr& expr, const DataTypePtr& type,
+ const std::string& name, size_t rows, ColumnPtr* column);
+
+ std::optional<std::string> _resolve_table_path(const TFileRangeDesc& range) const;
+ std::optional<std::string> _resolve_db_name(const TFileRangeDesc& range) const;
+ std::optional<std::string> _resolve_table_name(const TFileRangeDesc& range) const;
+ std::optional<std::string> _resolve_table_schema_json(const TFileRangeDesc& range) const;
+ // Non-default branch name (unset means main branch, matching upstream
+ // paimon commit 742da63: null-if-DEFAULT_MAIN_BRANCH).
+ std::optional<std::string> _resolve_branch(const TFileRangeDesc& range) const;
+ // Base64-decode the FE-planned split into the raw serialized DataSplit bytes.
+ Status _decode_split_bytes(std::string* out) const;
+ // Projected column names the rust reader actually reads (partition keys are
+ // excluded; they are materialized from split metadata, not from arrow).
+ std::vector<std::string> _build_read_columns() const;
+ // Storage options: FE paimon_options + hadoop conf/properties, with the
+ // OSS/S3 -> AWS_* key remap (v1 semantics).
+ std::map<std::string, std::string> _build_options() const;
+
+ std::unique_ptr<PaimonHandles> _handles; // cached table + per-split pipeline
+ // Identity of the opened table so consecutive splits reuse the handle.
+ std::optional<
+ std::tuple<std::string /*table_path*/, std::string /*schema_json*/, std::string /*db*/,
+ std::string /*table*/, std::optional<std::string> /*branch*/,
+ std::map<std::string, std::string> /*options*/>>
+ _opened_table_key;
+ // Projected column name (exact and lower-case) -> output block position.
+ std::unordered_map<std::string, size_t> _output_name_to_idx;
+ TFileRangeDesc _current_range;
+ cctz::time_zone _ctz;
+ // EOF belongs to the current split; reset when a new split is prepared.
+ bool _split_eof = false;
+
+ // Profile counters under the "PaimonRustReader" profile group
+ // (file_scan_profile::TABLE_READER parent, mirroring JniTableReader).
+ RuntimeProfile::Counter* _rust_predicates_input = nullptr;
+ RuntimeProfile::Counter* _rust_predicates_converted = nullptr;
+ RuntimeProfile::Counter* _rust_predicates_applied = nullptr;
+ RuntimeProfile::Counter* _rust_runtime_filters_input = nullptr;
+ RuntimeProfile::Counter* _rust_runtime_filters_applied = nullptr;
+ RuntimeProfile::Counter* _rust_total_time = nullptr;
+ RuntimeProfile::Counter* _rust_open_split_time = nullptr;
+ RuntimeProfile::Counter* _rust_read_batch_time = nullptr;
+ RuntimeProfile::Counter* _rust_arrow_to_block_time = nullptr;
+};
+
+} // namespace doris::format::paimon
diff --git a/be/test/exec/scan/file_scanner_v2_test.cpp b/be/test/exec/scan/file_scanner_v2_test.cpp
index 57a17a79e5a..ef7c6ab19e2 100644
--- a/be/test/exec/scan/file_scanner_v2_test.cpp
+++ b/be/test/exec/scan/file_scanner_v2_test.cpp
@@ -911,6 +911,22 @@ TEST(FileScannerV2Test, GeneratedChildrenKeepOneGlobalRowIdSourceMapping) {
EXPECT_EQ(first_id, second_id);
}
+TFileRangeDesc paimon_rust_jni_range(bool with_schema_json = true) {
+ auto range = range_with_format("paimon", TFileFormatType::FORMAT_JNI);
+ TPaimonFileDesc paimon_params;
+ paimon_params.__set_reader_type(TPaimonReaderType::PAIMON_RUST);
+ paimon_params.__set_file_format("parquet");
+ paimon_params.__set_paimon_split("serialized-rust-split");
+ paimon_params.__set_paimon_table("/paimon/warehouse/db.db/t");
+ paimon_params.__set_db_name("db");
+ paimon_params.__set_table_name("t");
+ if (with_schema_json) {
+ paimon_params.__set_paimon_table_schema_json("{}");
+ }
+ range.table_format_params.__set_paimon_params(std::move(paimon_params));
+ return range;
+}
+
TEST(FileScannerV2Test, JniCompatibilityShapesUseV2Scanner) {
TQueryOptions query_options;
query_options.__set_enable_file_scanner_v2(true);
@@ -921,6 +937,32 @@ TEST(FileScannerV2Test, JniCompatibilityShapesUseV2Scanner) {
EXPECT_TRUE(FileScannerV2::is_supported(params, legacy_paimon_jni_range_without_reader_type()));
}
+TEST(FileScannerV2Test, PaimonRustSplitUsesV2Scanner) {
+ TFileScanRangeParams params;
+ params.__set_format_type(TFileFormatType::FORMAT_JNI);
+
+ // A complete rust split (schema json + serialized split) is supported by V2.
+ EXPECT_TRUE(FileScannerV2::is_supported(params, paimon_rust_jni_range()));
+ EXPECT_TRUE(FileScannerV2::TEST_validate_scan_range(params, paimon_rust_jni_range()).ok());
+
+ // The schema-json pipeline field is required: a rust split without it cannot open the
+ // table and is rejected at validation instead of failing deep inside the reader.
+ const auto rust_range_without_schema = paimon_rust_jni_range(/*with_schema_json=*/false);
+ EXPECT_FALSE(FileScannerV2::is_supported(params, rust_range_without_schema));
+ const auto status = FileScannerV2::TEST_validate_scan_range(params, rust_range_without_schema);
+ EXPECT_TRUE(status.is<ErrorCode::NOT_IMPLEMENTED_ERROR>());
+
+ // Regression guard: PAIMON_CPP splits must remain on the V1 fallback.
+ auto cpp_range = range_with_format("paimon", TFileFormatType::FORMAT_JNI);
+ {
+ TPaimonFileDesc cpp_paimon_params;
+ cpp_paimon_params.__set_reader_type(TPaimonReaderType::PAIMON_CPP);
+ cpp_paimon_params.__set_file_format("parquet");
+ cpp_range.table_format_params.__set_paimon_params(std::move(cpp_paimon_params));
+ }
+ EXPECT_FALSE(FileScannerV2::is_supported(params, cpp_range));
+}
+
TEST(FileScannerV2Test, IcebergDeleteSplitKeepsInitialScannerCountCap) {
TQueryOptions query_options;
query_options.__set_enable_file_scanner_v2(true);
diff --git a/be/test/exec/test_data/paimon_rust_batch_size/README.md b/be/test/exec/test_data/paimon_rust_batch_size/README.md
new file mode 100644
index 00000000000..b9e939b66ec
--- /dev/null
+++ b/be/test/exec/test_data/paimon_rust_batch_size/README.md
@@ -0,0 +1,32 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Paimon batch-size fixture
+
+Generated with Apache Paimon 1.3.1 as an append-only Parquet table with
+`bucket=-1` and `write-only=true`. It contains 65 rows: `k` ranges from 0 to 64,
+and every `v` is 128 KiB of the ASCII character `x`.
+
+`schema/schema-0` is the table schema. `split.bin` is a Java `DataSplit`
+serialization whose bucket path is the neutral `__BUCKET__` placeholder.
+The reader test replaces that Java `writeUTF` field with the fixture's bucket
+path at runtime. No catalog, object store, or Java process is needed to read it.
+
+The fixture exercises the actual Rust Arrow reader with small batch sizes and
+checks all values across batch boundaries and repeated split preparation.
diff --git a/be/test/exec/test_data/paimon_rust_batch_size/bucket-0/data-3b6dc957-0664-4ec4-946c-9535d228957a-0.parquet b/be/test/exec/test_data/paimon_rust_batch_size/bucket-0/data-3b6dc957-0664-4ec4-946c-9535d228957a-0.parquet
new file mode 100644
index 00000000000..160939df2fb
Binary files /dev/null and b/be/test/exec/test_data/paimon_rust_batch_size/bucket-0/data-3b6dc957-0664-4ec4-946c-9535d228957a-0.parquet differ
diff --git a/be/test/exec/test_data/paimon_rust_batch_size/schema/schema-0 b/be/test/exec/test_data/paimon_rust_batch_size/schema/schema-0
new file mode 100644
index 00000000000..2c057fbb6d2
--- /dev/null
+++ b/be/test/exec/test_data/paimon_rust_batch_size/schema/schema-0
@@ -0,0 +1,22 @@
+{
+ "version" : 3,
+ "id" : 0,
+ "fields" : [ {
+ "id" : 0,
+ "name" : "k",
+ "type" : "INT"
+ }, {
+ "id" : 1,
+ "name" : "v",
+ "type" : "STRING"
+ } ],
+ "highestFieldId" : 1,
+ "partitionKeys" : [ ],
+ "primaryKeys" : [ ],
+ "options" : {
+ "bucket" : "-1",
+ "write-only" : "true",
+ "file.format" : "parquet"
+ },
+ "timeMillis" : 1790690378190
+}
\ No newline at end of file
diff --git a/be/test/exec/test_data/paimon_rust_batch_size/split.bin b/be/test/exec/test_data/paimon_rust_batch_size/split.bin
new file mode 100644
index 00000000000..10135ae00fb
Binary files /dev/null and b/be/test/exec/test_data/paimon_rust_batch_size/split.bin differ
diff --git a/be/test/format_v2/table/paimon_reader_test.cpp b/be/test/format_v2/table/paimon_reader_test.cpp
index 91900dffd99..1eaa841b507 100644
--- a/be/test/format_v2/table/paimon_reader_test.cpp
+++ b/be/test/format_v2/table/paimon_reader_test.cpp
@@ -356,6 +356,25 @@ TFileRangeDesc make_paimon_jni_range() {
return range;
}
+TFileRangeDesc make_paimon_rust_range() {
+ TFileRangeDesc range;
+ range.__set_path("/data-placeholder.parquet");
+ range.__set_format_type(TFileFormatType::FORMAT_JNI);
+ TTableFormatFileDesc table_format_params;
+ table_format_params.__set_table_format_type("paimon");
+ TPaimonFileDesc paimon_params;
+ paimon_params.__set_file_format("parquet");
+ paimon_params.__set_reader_type(TPaimonReaderType::PAIMON_RUST);
+ paimon_params.__set_paimon_split("serialized-paimon-split");
+ paimon_params.__set_paimon_table("/paimon/warehouse/db.db/t");
+ paimon_params.__set_db_name("db");
+ paimon_params.__set_table_name("t");
+ paimon_params.__set_paimon_table_schema_json("{}");
+ table_format_params.__set_paimon_params(paimon_params);
+ range.__set_table_format_params(table_format_params);
+ return range;
+}
+
TFileRangeDesc make_legacy_paimon_native_range(TFileFormatType::type physical_format_type) {
TFileRangeDesc range = make_paimon_native_range(physical_format_type);
range.__set_format_type(TFileFormatType::FORMAT_JNI);
@@ -988,6 +1007,24 @@ TEST(PaimonHybridReaderTest, ClassifiesJniSplitByReaderType) {
EXPECT_TRUE(paimon::PaimonHybridReader::TEST_is_jni_split(make_paimon_jni_range()));
}
+TEST(PaimonHybridReaderTest, ClassifiesRustSplitByReaderType) {
+ // A rust split is neither a JNI split nor a native split; the dispatch order in
+ // _ensure_current_split_reader relies on the rust check running before the JNI one.
+ EXPECT_TRUE(paimon::PaimonHybridReader::TEST_is_rust_split(make_paimon_rust_range()));
+ EXPECT_FALSE(paimon::PaimonHybridReader::TEST_is_rust_split(make_paimon_jni_range()));
+ EXPECT_FALSE(paimon::PaimonHybridReader::TEST_is_rust_split(
+ make_paimon_native_range(TFileFormatType::FORMAT_PARQUET)));
+ EXPECT_FALSE(paimon::PaimonHybridReader::TEST_is_rust_split(
+ make_legacy_paimon_native_range(TFileFormatType::FORMAT_PARQUET)));
+ EXPECT_FALSE(paimon::PaimonHybridReader::TEST_is_jni_split(make_paimon_rust_range()));
+
+ // A serialized split without a reader_type stays on the legacy JNI classification.
+ TFileRangeDesc legacy_split = make_paimon_rust_range();
+ legacy_split.table_format_params.paimon_params.__isset.reader_type = false;
+ EXPECT_FALSE(paimon::PaimonHybridReader::TEST_is_rust_split(legacy_split));
+ EXPECT_TRUE(paimon::PaimonHybridReader::TEST_is_jni_split(legacy_split));
+}
+
TEST(PaimonHybridReaderTest, ConvertsNativeSplitFileFormat) {
FileFormat file_format;
ASSERT_TRUE(paimon::PaimonHybridReader::TEST_to_file_format(
@@ -1028,6 +1065,7 @@ TEST(PaimonHybridReaderTest, NormalizesLegacyNativeSplitFormatBeforeChildPrepare
tracking_reader = child.get();
return child;
},
+ [] { return std::make_unique<TableReader>(); },
[] { return std::make_unique<TableReader>(); });
ASSERT_TRUE(reader.init({
.projected_columns = {},
@@ -1048,20 +1086,21 @@ TEST(PaimonHybridReaderTest, NormalizesLegacyNativeSplitFormatBeforeChildPrepare
EXPECT_EQ(tracking_reader->prepared_format, FileFormat::PARQUET);
}
-TEST(PaimonHybridReaderTest, AdaptiveBatchSizeReachesBothChildReaders) {
+TEST(PaimonHybridReaderTest, AdaptiveBatchSizeReachesAllChildReaders) {
paimon::PaimonHybridReader reader;
reader.TEST_install_batch_size_children();
reader.set_batch_size(321);
const auto child_batch_sizes = reader.TEST_child_batch_sizes();
- EXPECT_EQ(child_batch_sizes.first, 321);
- EXPECT_EQ(child_batch_sizes.second, 321);
+ EXPECT_EQ(std::get<0>(child_batch_sizes), 321);
+ EXPECT_EQ(std::get<1>(child_batch_sizes), 321);
+ EXPECT_EQ(std::get<2>(child_batch_sizes), 321);
}
-TEST(PaimonHybridReaderTest, AggregatesConditionCacheHitsFromBothChildren) {
+TEST(PaimonHybridReaderTest, AggregatesConditionCacheHitsFromAllChildren) {
paimon::PaimonHybridReader reader;
reader.TEST_install_batch_size_children();
- reader.TEST_set_child_condition_cache_hits(3, 5);
- EXPECT_EQ(reader.condition_cache_hit_count(), 8);
+ reader.TEST_set_child_condition_cache_hits(3, 5, 7);
+ EXPECT_EQ(reader.condition_cache_hit_count(), 15);
}
TEST(PaimonHybridReaderTest, ForwardsLatePredicatesToActiveChild) {
@@ -1075,6 +1114,7 @@ TEST(PaimonHybridReaderTest, ForwardsLatePredicatesToActiveChild) {
child = tracking.get();
return tracking;
},
+ [] { return std::make_unique<TableReader>(); },
[] { return std::make_unique<TableReader>(); });
ASSERT_TRUE(reader.init({
.projected_columns = {},
@@ -1184,6 +1224,120 @@ TEST(PaimonHybridReaderTest, DispatchesNativeThenJniSplitToMatchingReader) {
ASSERT_TRUE(reader.close().ok());
}
+TEST(PaimonHybridReaderTest, DispatchesRustSplitToRustChild) {
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ auto scan_params = make_local_parquet_scan_params();
+ paimon::PaimonHybridReader reader;
+ SplitFormatTrackingTableReader* tracking_reader = nullptr;
+ reader.TEST_set_child_reader_factories(
+ [] { return std::make_unique<TableReader>(); },
+ [] { return std::make_unique<TableReader>(); },
+ [&] {
+ auto child = std::make_unique<SplitFormatTrackingTableReader>();
+ tracking_reader = child.get();
+ return child;
+ });
+ ASSERT_TRUE(reader.init({
+ .projected_columns = {},
+ .conjuncts = {},
+ .format = FileFormat::JNI,
+ .scan_params = &scan_params,
+ .io_ctx = nullptr,
+ .runtime_state = &state,
+ .scanner_profile = nullptr,
+ })
+ .ok());
+
+ SplitReadOptions rust_split;
+ rust_split.current_range = make_paimon_rust_range();
+ rust_split.current_split_format = FileFormat::JNI;
+ ASSERT_TRUE(reader.prepare_split(rust_split).ok());
+ ASSERT_NE(tracking_reader, nullptr);
+ // A serialized rust split stays on the JNI wrapper format. FE also sets
+ // paimon_params.file_format="parquet" on serialized splits, so normalizing it
+ // to PARQUET would mis-dispatch the split to the native child.
+ EXPECT_EQ(tracking_reader->prepared_format, FileFormat::JNI);
+
+ ASSERT_TRUE(reader.close().ok());
+}
+
+TEST(PaimonHybridReaderTest, DispatchesNativeThenRustSplitToMatchingReader) {
+ RuntimeProfile profile("test_profile");
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ auto scan_params = make_local_parquet_scan_params();
+ io::FileReaderStats file_reader_stats;
+ io::FileCacheStatistics file_cache_stats;
+ auto io_ctx = make_io_context(&file_reader_stats, &file_cache_stats);
+
+ paimon::PaimonHybridReader reader;
+ // The native child is the real PaimonReader over a placeholder path; the rust
+ // child is a tracking stub, proving dispatch reaches it without native
+ // normalization.
+ SplitFormatTrackingTableReader* rust_tracking_reader = nullptr;
+ reader.TEST_set_child_reader_factories(nullptr, nullptr, [&] {
+ auto child = std::make_unique<SplitFormatTrackingTableReader>();
+ rust_tracking_reader = child.get();
+ return child;
+ });
+ ASSERT_TRUE(reader.init({
+ .projected_columns = {},
+ .conjuncts = {},
+ .format = FileFormat::PARQUET,
+ .scan_params = &scan_params,
+ .io_ctx = io_ctx,
+ .runtime_state = &state,
+ .scanner_profile = &profile,
+ })
+ .ok());
+
+ SplitReadOptions native_split;
+ native_split.current_range = make_paimon_native_range(TFileFormatType::FORMAT_PARQUET);
+ native_split.current_split_format = FileFormat::PARQUET;
+ ASSERT_TRUE(reader.prepare_split(native_split).ok());
+
+ SplitReadOptions rust_split;
+ rust_split.current_range = make_paimon_rust_range();
+ rust_split.current_split_format = FileFormat::JNI;
+ ASSERT_TRUE(reader.prepare_split(rust_split).ok());
+ ASSERT_NE(rust_tracking_reader, nullptr);
+ EXPECT_EQ(rust_tracking_reader->prepared_format, FileFormat::JNI);
+
+ ASSERT_TRUE(reader.close().ok());
+}
+
+TEST(PaimonHybridReaderTest, ForwardsLatePredicatesToRustChild) {
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ auto scan_params = make_local_parquet_scan_params();
+ paimon::PaimonHybridReader reader;
+ RefreshTrackingTableReader* rust_child = nullptr;
+ reader.TEST_set_child_reader_factories(
+ [] { return std::make_unique<TableReader>(); },
+ [] { return std::make_unique<TableReader>(); },
+ [&] {
+ auto tracking = std::make_unique<RefreshTrackingTableReader>();
+ rust_child = tracking.get();
+ return tracking;
+ });
+ ASSERT_TRUE(reader.init({
+ .projected_columns = {},
+ .conjuncts = {},
+ .format = FileFormat::JNI,
+ .scan_params = &scan_params,
+ .io_ctx = nullptr,
+ .runtime_state = &state,
+ .scanner_profile = nullptr,
+ })
+ .ok());
+
+ SplitReadOptions rust_split;
+ rust_split.current_range = make_paimon_rust_range();
+ rust_split.current_split_format = FileFormat::JNI;
+ ASSERT_TRUE(reader.prepare_split(rust_split).ok());
+ ASSERT_NE(rust_child, nullptr);
+ ASSERT_TRUE(reader.refresh_conjuncts({}).ok());
+ EXPECT_EQ(rust_child->refresh_count, 1);
+}
+
TEST(PaimonHybridReaderTest, FirstNativeAndJniChildInitAreCountedOnce) {
RuntimeProfile profile("test_profile");
RuntimeState state {TQueryOptions(), TQueryGlobals()};
@@ -1203,6 +1357,7 @@ TEST(PaimonHybridReaderTest, FirstNativeAndJniChildInitAreCountedOnce) {
})
.ok());
reader.TEST_set_child_reader_factories([] { return std::make_unique<SlowInitTableReader>(); },
+ [] { return std::make_unique<SlowInitTableReader>(); },
[] { return std::make_unique<SlowInitTableReader>(); });
auto* total = profile.get_counter("TableReader");
diff --git a/be/test/format_v2/table/paimon_rust_predicate_converter_test.cpp b/be/test/format_v2/table/paimon_rust_predicate_converter_test.cpp
new file mode 100644
index 00000000000..6aee77faa8d
--- /dev/null
+++ b/be/test/format_v2/table/paimon_rust_predicate_converter_test.cpp
@@ -0,0 +1,674 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+
+#include <gtest/gtest.h>
+
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/status.h"
+#include "core/block/block.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type_date_or_datetime_v2.h"
+#include "core/data_type/data_type_decimal.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_string.h"
+#include "core/data_type/primitive_type.h"
+#include "core/field.h"
+#include "core/types.h"
+#include "core/value/vdatetime_value.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vectorized_fn_call.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vin_predicate.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+
+namespace doris {
+namespace {
+
+struct table_deleter {
+ void operator()(paimon_table* table) const {
+ if (table) {
+ paimon_table_free(table);
+ }
+ }
+};
+using table_ptr = std::unique_ptr<paimon_table, table_deleter>;
+
+struct predicate_deleter {
+ void operator()(paimon_predicate* predicate) const {
+ if (predicate) {
+ paimon_predicate_free(predicate);
+ }
+ }
+};
+using predicate_ptr = std::unique_ptr<paimon_predicate, predicate_deleter>;
+
+// A binary predicate node with a hand-set opcode. The converter only inspects
+// op() and the children, never executes the expr, so execute_column_impl is a
+// stub that is never reached.
+class TestBinaryPredicate final : public VExpr {
+public:
+ TestBinaryPredicate(TExprOpcode::type opcode, VExprSPtr left, VExprSPtr right)
+ : VExpr(std::make_shared<DataTypeUInt8>(), false) {
+ _node_type = opcode == TExprOpcode::EQ_FOR_NULL ? TExprNodeType::NULL_AWARE_BINARY_PRED
+ : TExprNodeType::BINARY_PRED;
+ _opcode = opcode;
+ add_child(std::move(left));
+ add_child(std::move(right));
+ }
+
+ const std::string& expr_name() const override { return _name; }
+
+ Status execute_column_impl(VExprContext* /*context*/, const Block* /*block*/,
+ const Selector* /*selector*/, size_t /*count*/,
+ ColumnPtr& result_column) const override {
+ result_column = ColumnUInt8::create();
+ return Status::OK();
+ }
+
+private:
+ const std::string _name = "test_binary_predicate";
+};
+
+// A cast node wrapping an operand, like the BE tree of
+// `CAST(amount AS DECIMAL(10,1)) = 1.2`. The converter only inspects
+// node_type() and the children, never executes the expr.
+class TestCastExpr final : public VExpr {
+public:
+ TestCastExpr(const DataTypePtr& target_type, VExprSPtr child) : VExpr(target_type, false) {
+ _node_type = TExprNodeType::CAST_EXPR;
+ add_child(std::move(child));
+ }
+
+ const std::string& expr_name() const override { return _name; }
+
+ Status execute_column_impl(VExprContext* /*context*/, const Block* /*block*/,
+ const Selector* /*selector*/, size_t /*count*/,
+ ColumnPtr& result_column) const override {
+ result_column = ColumnUInt8::create();
+ return Status::OK();
+ }
+
+private:
+ const std::string _name = "test_cast";
+};
+
+// An error-preserving conjunct like assert_true(...): executing it only on
+// rows selected by other predicates could discard the error it must raise, so
+// VExpr::is_safe_to_execute_on_selected_rows reports it unsafe (a failing
+// VCastExpr behaves the same way).
+class UnsafeExpr final : public VExpr {
+public:
+ explicit UnsafeExpr(VExprSPtr child)
+ : VExpr(make_nullable(std::make_shared<DataTypeInt32>()), false) {
+ _node_type = TExprNodeType::BINARY_PRED;
+ add_child(std::move(child));
+ }
+
+ bool is_safe_to_execute_on_selected_rows() const override { return false; }
+
+ const std::string& expr_name() const override { return _name; }
+
+ Status execute_column_impl(VExprContext* /*context*/, const Block* /*block*/,
+ const Selector* /*selector*/, size_t /*count*/,
+ ColumnPtr& result_column) const override {
+ result_column = ColumnUInt8::create();
+ return Status::OK();
+ }
+
+private:
+ const std::string _name = "unsafe_expr";
+};
+
+// A slot ref resolved by column name, like FileScannerV2's rewritten conjuncts.
+VExprSPtr slot_ref(const std::string& column_name, const DataTypePtr& type) {
+ return std::make_shared<VSlotRef>(0, 0, -1, type, column_name);
+}
+
+VExprSPtr slot_ref(const std::string& column_name) {
+ return slot_ref(column_name, make_nullable(std::make_shared<DataTypeInt32>()));
+}
+
+VExprSPtr int_literal(int32_t value) {
+ return VLiteral::create_shared(std::make_shared<DataTypeInt32>(),
+ Field::create_field<TYPE_INT>(value));
+}
+
+// A STRING column type.
+const DataTypePtr& string_type() {
+ static const auto type = make_nullable(std::make_shared<DataTypeString>());
+ return type;
+}
+
+VExprSPtr string_literal(std::string value) {
+ return VLiteral::create_shared(std::make_shared<DataTypeString>(),
+ Field::create_field<TYPE_STRING>(std::move(value)));
+}
+
+// A like(col, pattern[, escape]) function call. The converter only inspects
+// function_name() and the children, never executes the expr.
+VExprSPtr like_call(std::vector<VExprSPtr> children) {
+ TExprNode node;
+ node.__set_type(create_type_desc(PrimitiveType::TYPE_BOOLEAN));
+ node.__set_node_type(TExprNodeType::FUNCTION_CALL);
+ node.__set_fn(TFunction());
+ node.fn.__set_name(TFunctionName());
+ node.fn.name.function_name = "like";
+ node.__set_is_nullable(false);
+ auto call = VectorizedFnCall::create_shared(node);
+ for (auto& child : children) {
+ call->add_child(std::move(child));
+ }
+ return call;
+}
+
+// An IN / NOT IN predicate over the given children (built through the
+// TExprNode path like VDirectInPredicate::get_slot_in_expr does, so
+// is_not_in() is initialized). The converter only inspects node_type(),
+// op() and the children.
+VExprSPtr in_predicate(bool is_not, std::vector<VExprSPtr> children) {
+ TExprNode node;
+ node.__set_type(create_type_desc(PrimitiveType::TYPE_BOOLEAN));
+ node.__set_node_type(TExprNodeType::IN_PRED);
+ node.in_predicate.__set_is_not_in(is_not);
+ node.__set_opcode(is_not ? TExprOpcode::FILTER_NOT_IN : TExprOpcode::FILTER_IN);
+ node.__set_is_nullable(false);
+ auto predicate = VInPredicate::create_shared(node);
+ for (auto& child : children) {
+ predicate->add_child(std::move(child));
+ }
+ return predicate;
+}
+
+// 2024-01-01 00:00:00.<microsecond> as a DATETIMEV2(6) literal.
+const DataTypePtr& datetimev2_type() {
+ static const auto type = make_nullable(std::make_shared<DataTypeDateTimeV2>(6));
+ return type;
+}
+
+VExprSPtr datetimev2_literal(uint32_t microsecond) {
+ DateV2Value<DateTimeV2ValueType> value;
+ value.unchecked_set_time(2024, 1, 1, 0, 0, 0, microsecond);
+ return VLiteral::create_shared(std::make_shared<DataTypeDateTimeV2>(6),
+ Field::create_field<TYPE_DATETIMEV2>(value));
+}
+
+// A DECIMAL(10,2) column type.
+const DataTypePtr& decimal_type() {
+ static const auto type = make_nullable(std::make_shared<DataTypeDecimal64>(10, 2));
+ return type;
+}
+
+// A DECIMAL(10,2) literal, e.g. decimal_literal(1, 24) is 1.24.
+VExprSPtr decimal_literal(int64_t integer, int64_t fraction) {
+ return VLiteral::create_shared(
+ std::make_shared<DataTypeDecimal64>(10, 2),
+ Field::create_field<TYPE_DECIMAL64>(Decimal64::from_int_frac(integer, fraction, 2)));
+}
+
+VExprSPtr runtime_in_filter(VExprSPtr predicate) {
+ TExprNode node;
+ node.__set_type(create_type_desc(PrimitiveType::TYPE_BOOLEAN));
+ node.__set_node_type(TExprNodeType::IN_PRED);
+ node.__set_is_nullable(true);
+ return RuntimeFilterExpr::create_shared(node, std::move(predicate), 0.0, false, 1);
+}
+
+// A DOUBLE column type.
+const DataTypePtr& double_type() {
+ static const auto type = make_nullable(std::make_shared<DataTypeFloat64>());
+ return type;
+}
+
+} // namespace
+
+// Pins the EQ_FOR_NULL (<=>) decision matrix of the converter. Semantics are
+// aligned with the FE PaimonPredicateConverter, which only converts a binary
+// conjunct when the RHS is a convertible literal: a column-to-column
+// `a <=> b` must NOT be pushed down — the old code pushed `a IS NULL` and
+// wrongly discarded rows like (1, 1), which the residual conjunct cannot
+// recover.
+class PaimonRustPredicateConverterTest : public testing::Test {
+protected:
+ void SetUp() override {
+ // The table handle is built in memory from the schema JSON (the same
+ // format as an on-disk schema/schema-N file): building predicates only
+ // resolves fields against the schema, so the path never needs to exist.
+ // NOTE: the custom `json` delimiter matters — the schema contains
+ // `TIMESTAMP(6)"`, whose `)"` would terminate a plain R"(...)" early.
+ static constexpr const char* kSchemaJson =
+ R"json({"version":3,"id":0,"fields":[{"id":0,"name":"a","type":"INT"},)json"
+ R"json({"id":1,"name":"b","type":"INT"},{"id":2,"name":"ts",)json"
+ R"json("type":"TIMESTAMP(6)"},{"id":3,"name":"amount",)json"
+ R"json("type":"DECIMAL(10, 2)"},{"id":4,"name":"d",)json"
+ R"json("type":"DOUBLE"},{"id":5,"name":"s","type":"STRING"}],)json"
+ R"json("highestFieldId":5,"partitionKeys":[],"primaryKeys":[],)json"
+ R"json("options":{},"timeMillis":0})json";
+ auto result = paimon_table_from_schema_json("/tmp/paimon_rust_predicate_converter_test",
+ kSchemaJson, "db", "t", "main", nullptr, 0);
+ if (result.error != nullptr) {
+ std::string message;
+ if (result.error->message.data != nullptr && result.error->message.len > 0) {
+ message.assign(reinterpret_cast<const char*>(result.error->message.data),
+ result.error->message.len);
+ }
+ const int32_t code = result.error->code;
+ paimon_error_free(result.error);
+ FAIL() << "paimon_table_from_schema_json failed: code=" << code << ", msg=" << message;
+ }
+ _table.reset(result.table);
+
+ _column_names = {"a", "b", "ts", "amount", "d", "s"};
+ _column_types = {make_nullable(std::make_shared<DataTypeInt32>()),
+ make_nullable(std::make_shared<DataTypeInt32>()),
+ datetimev2_type(),
+ decimal_type(),
+ double_type(),
+ string_type()};
+ }
+
+ // Runs one conjunct through a fresh converter; a null return means the
+ // conjunct was rejected (left to the Doris residual).
+ predicate_ptr push(TExprOpcode::type opcode, VExprSPtr left, VExprSPtr right) {
+ PaimonRustPredicateConverter converter(_column_names, _column_types, _table.get());
+ VExprSPtr root =
+ std::make_shared<TestBinaryPredicate>(opcode, std::move(left), std::move(right));
+ VExprContextSPtrs conjuncts {VExprContext::create_shared(std::move(root))};
+ return predicate_ptr(converter.build(conjuncts));
+ }
+
+ // Runs one pre-built expr (e.g. an IN predicate) through a fresh
+ // converter; a null return means the conjunct was rejected.
+ predicate_ptr push_expr(VExprSPtr expr) {
+ PaimonRustPredicateConverter converter(_column_names, _column_types, _table.get());
+ VExprContextSPtrs conjuncts {VExprContext::create_shared(std::move(expr))};
+ return predicate_ptr(converter.build(conjuncts));
+ }
+
+ // Converts a literal expr against a column type so tests can inspect the
+ // resulting paimon_datum (tag / int_val / int_val2).
+ auto convert_literal(const VExprSPtr& expr, const DataTypePtr& column_type) {
+ PaimonRustPredicateConverter converter(_column_names, _column_types, _table.get());
+ return converter.TEST_convert_literal(expr, column_type);
+ }
+
+ // Base epoch millis of 2024-01-01 00:00:00 through the same conversion;
+ // the absolute value depends on the converter's timezone resolution.
+ int64_t converted_base_millis() {
+ auto holder = convert_literal(datetimev2_literal(0), datetimev2_type());
+ EXPECT_TRUE(holder.has_value());
+ return holder->datum.int_val;
+ }
+
+ table_ptr _table;
+ std::vector<std::string> _column_names;
+ std::vector<DataTypePtr> _column_types;
+};
+
+TEST_F(PaimonRustPredicateConverterTest, EqForNullColumnToColumnIsNotPushed) {
+ // The P1 case: `a <=> b` must stay in the residual. Pushing `a IS NULL`
+ // would keep only the (NULL, NULL) row and drop (1, 1) forever.
+ // (The literal forms `a <=> NULL` / `a <=> 1` are not covered: FE's
+ // NullSafeEqualToEqual rewrite turns them into IS NULL / plain equality
+ // before they ever reach a BE converter, so EQ_FOR_NULL arriving here is
+ // always column-to-column.)
+ EXPECT_EQ(push(TExprOpcode::EQ_FOR_NULL, slot_ref("a"), slot_ref("b")).get(), nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, EqLiteralIsPushed) {
+ // Sanity: plain equality still converts.
+ auto predicate = push(TExprOpcode::EQ, slot_ref("a"), int_literal(1));
+ EXPECT_NE(predicate.get(), nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, UnknownColumnIsNotPushed) {
+ // A slot outside the reader's projected columns cannot be resolved.
+ EXPECT_EQ(push(TExprOpcode::EQ, slot_ref("nope"), int_literal(1)).get(), nullptr);
+}
+
+// ---- pushdown stops at unsafe conjuncts (safe-prefix rule) ----
+
+TEST_F(PaimonRustPredicateConverterTest, UnsafeConjunctStopsLaterPredicatesFromBeingPushed) {
+ // The review's P1 case: an error-preserving conjunct (assert_true(...),
+ // a failing cast) that precedes a convertible predicate — in practice an
+ // arrived IN runtime filter, modeled here by a plain equality — must stop
+ // the pushdown. Pushing the later predicate would let rust prune the row
+ // on which the unsafe conjunct must still raise.
+ PaimonRustPredicateConverter converter(_column_names, _column_types, _table.get());
+ VExprContextSPtrs conjuncts;
+ conjuncts.push_back(VExprContext::create_shared(std::make_shared<UnsafeExpr>(slot_ref("a"))));
+ conjuncts.push_back(VExprContext::create_shared(
+ std::make_shared<TestBinaryPredicate>(TExprOpcode::EQ, slot_ref("b"), int_literal(1))));
+
+ predicate_ptr predicate(converter.build(conjuncts));
+ EXPECT_EQ(predicate.get(), nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, SafePrefixBeforeUnsafeConjunctIsStillPushed) {
+ // The prefix itself stays pushable: the convertible conjunct before the
+ // unsafe one is pushed, and the unsafe one only blocks everything after
+ // it (the later equality must not be pushed).
+ PaimonRustPredicateConverter converter(_column_names, _column_types, _table.get());
+ VExprContextSPtrs conjuncts;
+ conjuncts.push_back(VExprContext::create_shared(
+ std::make_shared<TestBinaryPredicate>(TExprOpcode::EQ, slot_ref("a"), int_literal(1))));
+ conjuncts.push_back(VExprContext::create_shared(std::make_shared<UnsafeExpr>(slot_ref("b"))));
+ conjuncts.push_back(VExprContext::create_shared(
+ std::make_shared<TestBinaryPredicate>(TExprOpcode::EQ, slot_ref("b"), int_literal(2))));
+
+ predicate_ptr predicate(converter.build(conjuncts));
+ EXPECT_NE(predicate.get(), nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, DoubleColumnPredicatesAreNotPushed) {
+ // Doris defines NaN as equal to itself and greater than every finite
+ // value, but the pinned rust evaluator compares doubles with
+ // f64::partial_cmp (IEEE partial ordering: NaN unordered, NaN != NaN), so
+ // a pushed `d > 1.0` would drop a stored NaN row Doris retains and a
+ // pushed IN (NaN) would reject it — rows pruned by the rust filter cannot
+ // be recovered by the residual. DOUBLE pushdown is skipped entirely
+ // (paimon-java's JNI path is unaffected: its CompareUtils goes through
+ // Double.compareTo, which is Doris's total ordering).
+ EXPECT_EQ(push(TExprOpcode::EQ, slot_ref("d", double_type()), int_literal(1)).get(), nullptr);
+ EXPECT_EQ(push(TExprOpcode::GT, slot_ref("d", double_type()), int_literal(1)).get(), nullptr);
+}
+
+// ---- IN list values must be bare literals (FE doInPredicate parity) ----
+
+TEST_F(PaimonRustPredicateConverterTest, InListIsPushed) {
+ // Sanity: an IN over bare literals still converts, so the rejections
+ // below come from the casted children, not from IN handling.
+ auto predicate =
+ push_expr(in_predicate(false, {slot_ref("a"), int_literal(1), int_literal(2)}));
+ EXPECT_NE(predicate.get(), nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, OversizedRuntimeInFilterStaysResidual) {
+ // Arrived runtime filters can exceed the 16-bit expression child count even
+ // though SQL IN lists normally cannot. Never push only a wrapped prefix.
+ for (size_t values : {65534, 65535, 65536, 65537}) {
+ SCOPED_TRACE(values);
+ std::vector<VExprSPtr> children {slot_ref("a")};
+ for (size_t i = 0; i < values; ++i) {
+ children.emplace_back(int_literal(static_cast<int32_t>(i)));
+ }
+ auto predicate = push_expr(runtime_in_filter(in_predicate(false, std::move(children))));
+ EXPECT_EQ(predicate != nullptr, values < 65535);
+ }
+}
+
+TEST_F(PaimonRustPredicateConverterTest, NullAwareRuntimeFilterStaysResidual) {
+ // A null-aware runtime filter (arrived from an EQ_FOR_NULL join) restores
+ // NULL probe rows to true in its residual execution
+ // (RuntimeFilterExpr::change_null_to_true). Unwrapping it to the ordinary
+ // IN set — rebuilt through VDirectInPredicate::get_slot_in_expr — treats
+ // NULL as not-in-set and would prune the NULL probes permanently before
+ // the join sees them. The wrapper is kept instead: it fails every
+ // dispatch, so the conjunct stays in the residual, while a non-null-aware
+ // wrapper still unwraps and pushes its impl (the arrived-IN case).
+ // VExpr(node) derives the wrapper's data type from the node's type
+ // descriptor, so it needs a valid one (a bare node aborts in the thrift
+ // type-vector walk).
+ TExprNode node;
+ node.__set_type(create_type_desc(PrimitiveType::TYPE_BOOLEAN));
+ node.__set_node_type(TExprNodeType::IN_PRED);
+ node.__set_is_nullable(true);
+ VExprSPtr null_aware_rf = RuntimeFilterExpr::create_shared(
+ node, in_predicate(false, {slot_ref("a"), int_literal(1), int_literal(2)}), 0.0,
+ /*null_aware=*/true, 1);
+ EXPECT_EQ(push_expr(std::move(null_aware_rf)).get(), nullptr);
+
+ VExprSPtr plain_rf = RuntimeFilterExpr::create_shared(
+ node, in_predicate(false, {slot_ref("a"), int_literal(1), int_literal(2)}), 0.0,
+ /*null_aware=*/false, 2);
+ EXPECT_NE(push_expr(std::move(plain_rf)).get(), nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, CastedInListValueIsNotPushed) {
+ // The review's P1 case (IN form): with debug_skip_fold_constant=true, a
+ // cast in the IN list reaches the BE un-folded. Unwrapping it (the old
+ // one-level cast rule from _convert_literal) would push the pre-cast
+ // value, but Doris compares against the cast result — `amount IN
+ // (CAST(1.24 AS DECIMAL(10,1)))` must compare against 1.2, and the
+ // unwrapped 1.24 push would permanently remove the 1.2 rows. _convert_literal
+ // rejects the casted value, which rejects the whole predicate; the
+ // residual applies the cast correctly.
+ auto casted = std::make_shared<TestCastExpr>(make_nullable(std::make_shared<DataTypeInt64>()),
+ int_literal(1));
+ EXPECT_EQ(push_expr(in_predicate(false, {slot_ref("a"), int_literal(1), std::move(casted)}))
+ .get(),
+ nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, CastedNotInListValueIsNotPushed) {
+ // Same rule for NOT IN: rejecting the push keeps both readers on the
+ // Doris residual, where the cast is evaluated correctly.
+ auto casted = std::make_shared<TestCastExpr>(make_nullable(std::make_shared<DataTypeInt64>()),
+ int_literal(1));
+ EXPECT_EQ(push_expr(in_predicate(true, {slot_ref("a"), std::move(casted)})).get(), nullptr);
+}
+
+// The sub-millisecond remainder must land in int_val2 (nanos) instead of being
+// truncated — the rust datum holds (millis = int_val, nanos = int_val2) and
+// timestamp comparison includes both. A truncated conversion would turn
+// `ts = '2024-01-01 00:00:00.123456'` into `.123000` and wrongly drop the
+// matching rows — the runtime-filter IN scenario from the review. (The base
+// epoch millis depend on the converter's timezone resolution, so expectations
+// are expressed relative to a whole-second literal converted the same way.)
+TEST_F(PaimonRustPredicateConverterTest, TimestampV2FractionalMicrosArePreserved) {
+ const int64_t base = converted_base_millis();
+ auto holder = convert_literal(datetimev2_literal(123456), datetimev2_type());
+ ASSERT_TRUE(holder.has_value());
+ // paimon_datum tag 10 = Timestamp (see paimon.h).
+ EXPECT_EQ(holder->datum.tag, 10);
+ EXPECT_EQ(holder->datum.int_val, base + 123);
+ EXPECT_EQ(holder->datum.int_val2, 456000);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, TimestampV2WholeMillisLeaveNanosZero) {
+ const int64_t base = converted_base_millis();
+ auto holder = convert_literal(datetimev2_literal(123000), datetimev2_type());
+ ASSERT_TRUE(holder.has_value());
+ EXPECT_EQ(holder->datum.int_val, base + 123);
+ EXPECT_EQ(holder->datum.int_val2, 0);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, TimestampV2MaxFractionIsPreserved) {
+ const int64_t base = converted_base_millis();
+ auto holder = convert_literal(datetimev2_literal(999999), datetimev2_type());
+ ASSERT_TRUE(holder.has_value());
+ EXPECT_EQ(holder->datum.int_val, base + 999);
+ EXPECT_EQ(holder->datum.int_val2, 999000);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, TimestampPredicatesWithoutSourcePrecisionStayResidual) {
+ // A DATETIMEV2 slot does not reveal whether its source was TIMESTAMP(9).
+ auto predicate =
+ push(TExprOpcode::EQ, slot_ref("ts", datetimev2_type()), datetimev2_literal(123456));
+ EXPECT_EQ(predicate.get(), nullptr);
+ EXPECT_EQ(push_expr(runtime_in_filter(in_predicate(false, {slot_ref("ts", datetimev2_type()),
+ datetimev2_literal(123456)})))
+ .get(),
+ nullptr);
+}
+
+// ---- casted operands are rejected, mirroring the FE converter ----
+
+TEST_F(PaimonRustPredicateConverterTest, CastedColumnEqIsNotPushed) {
+ // `CAST(a AS BIGINT) = 1` must stay in the residual: a cast on the column
+ // is not equivalence-preserving in general, so like the FE converter's
+ // convertDorisExprToSlotRef the cast is not unwrapped. (The old code
+ // stripped it via expr_without_cast and pushed against the raw column.)
+ auto casted = std::make_shared<TestCastExpr>(make_nullable(std::make_shared<DataTypeInt64>()),
+ slot_ref("a"));
+ EXPECT_EQ(push(TExprOpcode::EQ, std::move(casted), int_literal(1)).get(), nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, DecimalScaleCastColumnIsNotPushed) {
+ // The review's P1 case: for a DECIMAL(10,2) column holding 1.24,
+ // `CAST(amount AS DECIMAL(10,1)) = 1.2` must retain the row, but pushing
+ // the unwrapped `amount = 1.2` against the original column prunes it. The
+ // rust filter does not save us either: it accepts decimal literals with
+ // different scales and compares their mathematical values (1.24 != 1.2),
+ // so the unsafe predicate is not rejected as a type mismatch. The conjunct
+ // must stay in the Doris residual.
+ auto casted = std::make_shared<TestCastExpr>(
+ make_nullable(std::make_shared<DataTypeDecimal64>(10, 1)),
+ slot_ref("amount", decimal_type()));
+ EXPECT_EQ(push(TExprOpcode::EQ, std::move(casted), decimal_literal(1, 2)).get(), nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, DecimalPredicatesWithoutFileScaleStayResidual) {
+ // Historical file statistics may still have a larger scale than this slot.
+ auto predicate =
+ push(TExprOpcode::EQ, slot_ref("amount", decimal_type()), decimal_literal(1, 24));
+ EXPECT_EQ(predicate.get(), nullptr);
+ EXPECT_EQ(push_expr(runtime_in_filter(in_predicate(false, {slot_ref("amount", decimal_type()),
+ decimal_literal(1, 23)})))
+ .get(),
+ nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, SingleCastLiteralRhsIsNotPushed) {
+ // The review's P1 case (binary form): a single cast wrapping a literal on
+ // the RHS must stay in the residual. Unwrapping it (the old one-level cast
+ // rule from _convert_literal) pushed the pre-cast value, but Doris compares
+ // against the cast result, so with debug_skip_fold_constant=true the rust
+ // filter and the residual disagree on lossy casts. The cast is not
+ // evaluated here, so the predicate cannot be built from the raw literal.
+ auto casted = std::make_shared<TestCastExpr>(make_nullable(std::make_shared<DataTypeInt64>()),
+ int_literal(1));
+ EXPECT_EQ(push(TExprOpcode::EQ, slot_ref("a"), std::move(casted)).get(), nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, DecimalScaleCastLiteralRhsIsNotPushed) {
+ // The differential case: `amount = CAST(1.24 AS DECIMAL(10,1))` on a
+ // DECIMAL(10,2) column. The cast evaluates to 1.2, so Doris's residual
+ // keeps a stored 1.20 row; the unwrapped literal 1.24 pushed against the
+ // raw column would prune it before the residual runs, and pruned rows
+ // cannot be recovered. The rust filter does not reject this as a type
+ // mismatch either — it accepts decimals of different scales and compares
+ // their mathematical values (1.24 != 1.20) — so the only safe conversion
+ // is none: the conjunct stays in the residual.
+ auto casted = std::make_shared<TestCastExpr>(
+ make_nullable(std::make_shared<DataTypeDecimal64>(10, 1)), decimal_literal(1, 24));
+ EXPECT_EQ(push(TExprOpcode::EQ, slot_ref("amount", decimal_type()), std::move(casted)).get(),
+ nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, NestedCastLiteralIsNotPushed) {
+ // `CAST(CAST(1 AS BIGINT) AS DOUBLE)`: cast trees are rejected at the
+ // outer level now — the old code stripped all cast levels, and the
+ // interim code unwrapped exactly one.
+ auto inner = std::make_shared<TestCastExpr>(make_nullable(std::make_shared<DataTypeInt64>()),
+ int_literal(1));
+ auto outer = std::make_shared<TestCastExpr>(make_nullable(std::make_shared<DataTypeFloat64>()),
+ std::move(inner));
+ EXPECT_EQ(push(TExprOpcode::EQ, slot_ref("a"), std::move(outer)).get(), nullptr);
+}
+
+// ---- LIKE patterns must be bare literals ----
+
+TEST_F(PaimonRustPredicateConverterTest, LikeIsPushed) {
+ // Sanity: a plain `s LIKE 'abc%'` converts, so the rejections below come
+ // from the pattern / escape handling, not from LIKE support itself.
+ auto predicate = push_expr(like_call({slot_ref("s", string_type()), string_literal("abc%")}));
+ EXPECT_NE(predicate.get(), nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, CastedLikePatternIsNotPushed) {
+ // A cast wrapping the pattern must stay in the residual: _extract_string_
+ // literal does not execute the cast, so the pushed pattern would be the
+ // pre-cast value while the Doris residual matches the cast result (a
+ // truncating cast like CAST('abc%' AS CHAR(2)) changes the prefix). Same
+ // rule as _convert_literal for the binary RHS and IN-list values.
+ auto casted = std::make_shared<TestCastExpr>(make_nullable(std::make_shared<DataTypeString>()),
+ string_literal("abc%"));
+ EXPECT_EQ(push_expr(like_call({slot_ref("s", string_type()), std::move(casted)})).get(),
+ nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, LikeWithDivergentEscapeIsNotPushed) {
+ // A backslash before an ordinary character diverges between Doris and the
+ // pinned rust like (Doris keeps both characters, rust consumes the
+ // backslash), so such patterns stay in the residual. Escapes before %, _
+ // and \\ (and a trailing backslash) have identical semantics on both
+ // sides and still convert.
+ EXPECT_EQ(push_expr(like_call({slot_ref("s", string_type()), string_literal("a\\qb%")})).get(),
+ nullptr);
+ EXPECT_NE(push_expr(like_call({slot_ref("s", string_type()), string_literal("a\\%b")})).get(),
+ nullptr);
+ EXPECT_NE(push_expr(like_call({slot_ref("s", string_type()), string_literal("a\\\\b_")})).get(),
+ nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, LikeWithNonDefaultEscapeIsNotPushed) {
+ // The 3-arg like(col, pattern, escape) form only converts for the Doris
+ // default escape '\'; the rust predicate builder rejects anything else.
+ EXPECT_NE(push_expr(like_call({slot_ref("s", string_type()), string_literal("abc%"),
+ string_literal("\\")}))
+ .get(),
+ nullptr);
+ EXPECT_EQ(push_expr(like_call({slot_ref("s", string_type()), string_literal("a!b"),
+ string_literal("!")}))
+ .get(),
+ nullptr);
+ // A casted escape char is rejected like a casted pattern.
+ auto casted_escape = std::make_shared<TestCastExpr>(
+ make_nullable(std::make_shared<DataTypeString>()), string_literal("\\"));
+ EXPECT_EQ(push_expr(like_call({slot_ref("s", string_type()), string_literal("abc%"),
+ std::move(casted_escape)}))
+ .get(),
+ nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, BoundedStringPredicatesStayResidual) {
+ // An old physical value can be longer than the current VARCHAR bound.
+ auto bounded = make_nullable(std::make_shared<DataTypeString>(3, TYPE_VARCHAR));
+ _column_types.back() = bounded;
+ EXPECT_EQ(push(TExprOpcode::EQ, slot_ref("s", bounded), string_literal("abc")).get(), nullptr);
+ EXPECT_EQ(push_expr(runtime_in_filter(in_predicate(
+ false, {slot_ref("s", bounded), string_literal("abc")})))
+ .get(),
+ nullptr);
+}
+
+TEST_F(PaimonRustPredicateConverterTest, ConversionCountersExcludeResidualsAndReset) {
+ auto rf = runtime_in_filter(in_predicate(false, {slot_ref("a"), int_literal(1)}));
+ auto timestamp =
+ in_predicate(false, {slot_ref("ts", datetimev2_type()), datetimev2_literal(123456)});
+ PaimonRustPredicateConverter converter(_column_names, _column_types, _table.get());
+ predicate_ptr predicate(converter.build(
+ {VExprContext::create_shared(rf), VExprContext::create_shared(timestamp)}));
+ ASSERT_NE(predicate.get(), nullptr);
+ EXPECT_EQ(converter.converted_conjuncts(), 1);
+ EXPECT_EQ(converter.converted_runtime_filters(), 1);
+ predicate.reset(converter.build({VExprContext::create_shared(timestamp)}));
+ EXPECT_EQ(predicate.get(), nullptr);
+ EXPECT_EQ(converter.converted_conjuncts(), 0);
+ EXPECT_EQ(converter.converted_runtime_filters(), 0);
+}
+
+} // namespace doris
diff --git a/be/test/format_v2/table/paimon_rust_table_reader_test.cpp b/be/test/format_v2/table/paimon_rust_table_reader_test.cpp
new file mode 100644
index 00000000000..72094f3e020
--- /dev/null
+++ b/be/test/format_v2/table/paimon_rust_table_reader_test.cpp
@@ -0,0 +1,712 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <gtest/gtest.h>
+
+#include <chrono>
+#include <filesystem>
+#include <fstream>
+#include <iterator>
+#include <map>
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "cctz/time_zone.h"
+#include "core/block/block.h"
+#include "core/column/column_const.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_string.h"
+#include "core/field.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_data.h"
+#include "gen_cpp/PlanNodes_types.h"
+#include "runtime/runtime_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/url_coding.h"
+
+namespace doris::format::paimon {
+namespace {
+
+ColumnDefinition make_column(const std::string& name, const DataTypePtr& type,
+ bool is_partition_key = false) {
+ ColumnDefinition column;
+ column.name = name;
+ column.type = type->is_nullable() ? type : make_nullable(type);
+ column.is_partition_key = is_partition_key;
+ return column;
+}
+
+TFileRangeDesc make_rust_range() {
+ TFileRangeDesc range;
+ TTableFormatFileDesc table_format_params;
+ table_format_params.__set_table_format_type("paimon");
+ TPaimonFileDesc paimon_params;
+ paimon_params.__set_reader_type(TPaimonReaderType::PAIMON_RUST);
+ std::string encoded;
+ base64_encode("dummy-split-bytes", &encoded);
+ paimon_params.__set_paimon_split(encoded);
+ paimon_params.__set_paimon_table("/paimon/warehouse/db.db/t");
+ paimon_params.__set_db_name("db");
+ paimon_params.__set_table_name("t");
+ paimon_params.__set_paimon_table_schema_json("{}");
+ table_format_params.__set_paimon_params(paimon_params);
+ range.__set_table_format_params(table_format_params);
+ return range;
+}
+
+// A range whose table path carries the given URI scheme — the production
+// shape FE ships in paimon_table (PaimonScanNode sets it from the catalog's
+// table location).
+TFileRangeDesc make_rust_range_with_table_path(const std::string& table_path) {
+ auto range = make_rust_range();
+ range.table_format_params.paimon_params.__set_paimon_table(table_path);
+ return range;
+}
+
+} // namespace
+
+class PaimonRustTableReaderTest : public testing::Test {
+protected:
+ void SetUp() override {
+ _query_options.__set_batch_size(3);
+ _runtime_state = RuntimeState::create_unique(_query_options, _query_globals);
+ }
+
+ Status init_reader_with_count(PaimonRustTableReader* reader,
+ std::vector<GlobalIndex> count_columns) {
+ return reader->init({.projected_columns = {_projected_column},
+ .conjuncts = {},
+ .format = FileFormat::JNI,
+ .scan_params = nullptr,
+ .io_ctx = nullptr,
+ .runtime_state = _runtime_state.get(),
+ .scanner_profile = nullptr,
+ .push_down_agg_type = TPushAggOp::type::COUNT,
+ .push_down_count_columns = std::move(count_columns)});
+ }
+
+ TQueryOptions _query_options;
+ TQueryGlobals _query_globals;
+ std::unique_ptr<RuntimeState> _runtime_state;
+ ColumnDefinition _projected_column = make_column("k", std::make_shared<DataTypeInt32>());
+};
+
+TEST_F(PaimonRustTableReaderTest, ValidatesRustSplit) {
+ PaimonRustTableReader reader;
+
+ // Missing paimon_split.
+ auto range = make_rust_range();
+ range.table_format_params.paimon_params.__isset.paimon_split = false;
+ auto status = reader.TEST_validate_rust_split(range);
+ EXPECT_FALSE(status.ok());
+ EXPECT_NE(status.to_string().find("missing paimon_split"), std::string::npos) << status;
+
+ // Missing paimon_table (table path).
+ range = make_rust_range();
+ range.table_format_params.paimon_params.__isset.paimon_table = false;
+ status = reader.TEST_validate_rust_split(range);
+ EXPECT_FALSE(status.ok());
+ EXPECT_NE(status.to_string().find("missing paimon_table"), std::string::npos) << status;
+
+ // Missing db_name.
+ range = make_rust_range();
+ range.table_format_params.paimon_params.__isset.db_name = false;
+ status = reader.TEST_validate_rust_split(range);
+ EXPECT_FALSE(status.ok());
+ EXPECT_NE(status.to_string().find("missing db_name"), std::string::npos) << status;
+
+ // Missing table_name.
+ range = make_rust_range();
+ range.table_format_params.paimon_params.__isset.table_name = false;
+ status = reader.TEST_validate_rust_split(range);
+ EXPECT_FALSE(status.ok());
+ EXPECT_NE(status.to_string().find("missing table_name"), std::string::npos) << status;
+
+ // Missing paimon_table_schema_json.
+ range = make_rust_range();
+ range.table_format_params.paimon_params.__isset.paimon_table_schema_json = false;
+ status = reader.TEST_validate_rust_split(range);
+ EXPECT_FALSE(status.ok());
+ EXPECT_NE(status.to_string().find("missing paimon_table_schema_json"), std::string::npos)
+ << status;
+
+ // A mismatched reader_type is a protocol error.
+ range = make_rust_range();
+ range.table_format_params.paimon_params.__set_reader_type(TPaimonReaderType::PAIMON_JNI);
+ status = reader.TEST_validate_rust_split(range);
+ EXPECT_FALSE(status.ok());
+ EXPECT_NE(status.to_string().find("invalid reader_type"), std::string::npos) << status;
+
+ // A complete range validates cleanly.
+ EXPECT_TRUE(reader.TEST_validate_rust_split(make_rust_range()).ok());
+}
+
+TEST_F(PaimonRustTableReaderTest, AdaptiveBatchSizeBoundsWideStringReads) {
+ const auto fixture = std::filesystem::path(__FILE__).parent_path() /
+ "../../exec/test_data/paimon_rust_batch_size";
+ const auto read_file = [](const std::filesystem::path& path) {
+ std::ifstream input(path, std::ios::binary);
+ return std::string(std::istreambuf_iterator<char>(input), std::istreambuf_iterator<char>());
+ };
+ auto split = read_file(fixture / "split.bin");
+ const auto placeholder = split.find("__BUCKET__");
+ ASSERT_NE(placeholder, std::string::npos);
+ ASSERT_GE(placeholder, 2);
+ const auto bucket = std::filesystem::canonical(fixture / "bucket-0").string();
+ ASSERT_LT(bucket.size(), 65536);
+ // The Java DataSplit fixture uses a neutral bucket path, encoded with writeUTF.
+ const std::string length = {static_cast<char>(bucket.size() >> 8),
+ static_cast<char>(bucket.size() & 0xff)};
+ split.replace(placeholder - 2, 2 + std::string("__BUCKET__").size(), length + bucket);
+ std::string encoded;
+ base64_encode(split, &encoded);
+ SplitReadOptions options;
+ options.current_range = make_rust_range();
+ options.current_split_format = FileFormat::JNI;
+ options.all_runtime_filters_applied = true;
+ auto& params = options.current_range.table_format_params.paimon_params;
+ params.__set_paimon_table(std::filesystem::canonical(fixture).string());
+ params.__set_paimon_table_schema_json(read_file(fixture / "schema/schema-0"));
+ params.__set_paimon_split(encoded);
+
+ const auto value_column = make_column("v", std::make_shared<DataTypeString>());
+ PaimonRustTableReader reader;
+ ASSERT_TRUE(reader.init({.projected_columns = {_projected_column, value_column},
+ .conjuncts = {},
+ .format = FileFormat::JNI,
+ .scan_params = nullptr,
+ .io_ctx = nullptr,
+ .runtime_state = _runtime_state.get(),
+ .scanner_profile = nullptr})
+ .ok());
+ // Wide values must be bounded before Arrow decoding, including when reusing a table handle.
+ for (const size_t requested : {0, 32, 7}) {
+ if (requested != 0) {
+ reader.set_batch_size(requested);
+ }
+ const size_t expected = requested == 0 ? 3 : requested;
+ auto status = reader.prepare_split(options);
+ ASSERT_TRUE(status.ok()) << status;
+ Block block({ColumnWithTypeAndName(_projected_column.type->create_column(),
+ _projected_column.type, "k"),
+ ColumnWithTypeAndName(value_column.type->create_column(), value_column.type,
+ "v")});
+ size_t total = 0;
+ bool eos = false;
+ while (!eos) {
+ status = reader.get_block(&block, &eos);
+ ASSERT_TRUE(status.ok()) << status;
+ ASSERT_LE(block.rows(), expected);
+ const auto& keys = assert_cast<const ColumnNullable&>(*block.get_by_position(0).column);
+ const auto& values =
+ assert_cast<const ColumnNullable&>(*block.get_by_position(1).column);
+ for (size_t row = 0; row < block.rows(); ++row) {
+ ASSERT_FALSE(keys.is_null_at(row));
+ ASSERT_FALSE(values.is_null_at(row));
+ EXPECT_EQ(keys.get_nested_column().get_int(row), total + row);
+ EXPECT_EQ(values.get_nested_column().get_data_at(row).to_string(),
+ std::string(128 * 1024, 'x'));
+ }
+ total += block.rows();
+ }
+ EXPECT_EQ(total, 65);
+ ASSERT_TRUE(reader.abort_split().ok());
+ }
+ ASSERT_TRUE(reader.close().ok());
+}
+
+TEST_F(PaimonRustTableReaderTest, TableLevelCountEmitsSyntheticRows) {
+ // COUNT(*) with a table-level row count takes the base-class metadata path:
+ // prepare_split never opens the rust pipeline and get_block emits synthetic rows.
+ PaimonRustTableReader reader;
+ ASSERT_TRUE(init_reader_with_count(&reader, std::vector<GlobalIndex> {}).ok());
+
+ SplitReadOptions options;
+ options.current_range = make_rust_range();
+ options.current_split_format = FileFormat::JNI;
+ options.all_runtime_filters_applied = true;
+ options.current_range.table_format_params.__set_table_level_row_count(5);
+ ASSERT_TRUE(reader.prepare_split(options).ok());
+ EXPECT_TRUE(reader.current_split_uses_metadata_count());
+
+ Block block = Block({ColumnWithTypeAndName(_projected_column.type->create_column(),
+ _projected_column.type, _projected_column.name)});
+ bool eos = false;
+ // The base-class count contract emits batches until a call finds remaining==0:
+ // batch_size(3) splits 5 rows into 3 + 2, and only the following call reports eos.
+ ASSERT_TRUE(reader.get_block(&block, &eos).ok());
+ EXPECT_EQ(block.rows(), 3);
+ EXPECT_FALSE(eos);
+
+ ASSERT_TRUE(reader.get_block(&block, &eos).ok());
+ EXPECT_EQ(block.rows(), 2);
+ EXPECT_FALSE(eos);
+
+ ASSERT_TRUE(reader.get_block(&block, &eos).ok());
+ EXPECT_EQ(block.rows(), 0);
+ EXPECT_TRUE(eos);
+}
+
+TEST_F(PaimonRustTableReaderTest, TableLevelCountDisabledByConjuncts) {
+ // A row predicate makes the metadata shortcut unsafe: the split must not report a
+ // metadata count. It proceeds to the rust pipeline instead, which fails on the dummy
+ // split bytes of this test range rather than emitting synthetic count rows.
+ PaimonRustTableReader reader;
+ ASSERT_TRUE(init_reader_with_count(&reader, std::vector<GlobalIndex> {}).ok());
+
+ SplitReadOptions options;
+ options.current_range = make_rust_range();
+ options.current_split_format = FileFormat::JNI;
+ options.all_runtime_filters_applied = true;
+ options.current_range.table_format_params.__set_table_level_row_count(5);
+ options.conjuncts = VExprContextSPtrs {};
+ options.conjuncts->push_back(VExprContext::create_shared(VLiteral::create_shared(
+ std::make_shared<DataTypeInt32>(), Field::create_field<TYPE_INT>(1))));
+
+ const auto status = reader.prepare_split(options);
+ EXPECT_FALSE(status.ok());
+ EXPECT_FALSE(reader.current_split_uses_metadata_count());
+
+ Block block = Block({ColumnWithTypeAndName(_projected_column.type->create_column(),
+ _projected_column.type, _projected_column.name)});
+ bool eos = false;
+ const auto get_block_status = reader.get_block(&block, &eos);
+ EXPECT_FALSE(get_block_status.ok());
+ EXPECT_NE(get_block_status.to_string().find("paimon-rust reader is not initialized"),
+ std::string::npos)
+ << get_block_status;
+}
+
+TEST_F(PaimonRustTableReaderTest, FailedSplitOpenIsIncludedInRustTotalTime) {
+ RuntimeProfile profile("scanner");
+ PaimonRustTableReader reader;
+ ASSERT_TRUE(reader.init({.projected_columns = {_projected_column},
+ .conjuncts = {},
+ .format = FileFormat::JNI,
+ .scan_params = nullptr,
+ .io_ctx = nullptr,
+ .runtime_state = _runtime_state.get(),
+ .scanner_profile = &profile})
+ .ok());
+ SplitReadOptions options;
+ options.current_range = make_rust_range();
+ options.current_split_format = FileFormat::JNI;
+ // The invalid split fails during open, before get_block can contribute any time.
+ ASSERT_FALSE(reader.prepare_split(options).ok());
+ auto* total = profile.get_counter("PaimonRustReader");
+ auto* open = profile.get_counter("OpenSplitTime");
+ ASSERT_NE(total, nullptr);
+ ASSERT_NE(open, nullptr);
+ EXPECT_GT(open->value(), 0);
+ EXPECT_GE(total->value(), open->value());
+}
+
+TEST_F(PaimonRustTableReaderTest, FillsPartitionConstantsForMissingArrowColumns) {
+ PaimonRustTableReader reader;
+ const auto data_type = make_nullable(std::make_shared<DataTypeInt32>());
+ const auto partition_type = make_nullable(std::make_shared<DataTypeString>());
+ // The output block matches the projected columns exactly (get_block contract):
+ // position 0 is the data column k (absent from the arrow batch -> default fill),
+ // position 1 is the partition key dt (materialized from split metadata).
+ reader.TEST_set_projected_columns(
+ {make_column("k", std::make_shared<DataTypeInt32>()),
+ make_column("dt", std::make_shared<DataTypeString>(), /*is_partition_key=*/true)});
+ std::map<std::string, Field> partition_values;
+ partition_values.emplace("dt", Field::create_field<TYPE_STRING>("2024-01-01"));
+ reader.TEST_set_partition_values(std::move(partition_values));
+
+ Block block =
+ Block({ColumnWithTypeAndName(data_type->create_column(), data_type, "k"),
+ ColumnWithTypeAndName(partition_type->create_column(), partition_type, "dt")});
+ const size_t rows = 4;
+ ASSERT_TRUE(reader.TEST_fill_non_arrow_columns(&block, rows).ok());
+
+ // The data column is absent from both the arrow batch and split metadata: filled
+ // with defaults.
+ EXPECT_EQ(block.get_by_position(0).column->size(), rows);
+
+ // The partition position is a constant column with the split value broadcast
+ // to every row.
+ const auto& column_with_type = block.get_by_position(1);
+ EXPECT_EQ(column_with_type.column->size(), rows);
+ const auto* const_column = check_and_get_column<ColumnConst>(*column_with_type.column);
+ ASSERT_NE(const_column, nullptr);
+ EXPECT_EQ(const_column->size(), rows);
+ const auto value_field = const_column->get_field();
+ const auto& value = value_field.get<TYPE_STRING>();
+ EXPECT_EQ(std::string(value.data(), value.size()), "2024-01-01");
+}
+
+TEST_F(PaimonRustTableReaderTest, TruncatesNullablePartitionAndDefaultConstants) {
+ _query_options.__set_truncate_char_or_varchar_columns(true);
+ _runtime_state = RuntimeState::create_unique(_query_options, _query_globals);
+ for (const auto primitive : {PrimitiveType::TYPE_CHAR, PrimitiveType::TYPE_VARCHAR}) {
+ for (bool is_null : {false, true}) {
+ PaimonRustTableReader reader;
+ ASSERT_TRUE(init_reader_with_count(&reader, {}).ok());
+ const auto type = make_nullable(std::make_shared<DataTypeString>(3, primitive));
+ auto partition = make_column("partition_key", type, true);
+ auto default_column = make_column("default_value", type);
+ const auto value = is_null ? Field() : Field::create_field<TYPE_STRING>("abcdef");
+ default_column.default_expr =
+ VExprContext::create_shared(VLiteral::create_shared(type, value));
+ reader.TEST_set_projected_columns({partition, default_column});
+ reader.TEST_set_partition_values({{"partition_key", value}});
+ Block block({ColumnWithTypeAndName(type->create_column(), type, "partition_key"),
+ ColumnWithTypeAndName(type->create_column(), type, "default_value")});
+ ASSERT_TRUE(reader.TEST_fill_non_arrow_columns(&block, 3).ok());
+ ASSERT_TRUE(is_column_const(*block.get_by_position(0).column));
+ // Split constants wrap nullable data; truncation must preserve both the row count
+ // and null map when it materializes that outer wrapper.
+ ASSERT_TRUE(reader.TEST_truncate_char_or_varchar_columns(&block).ok());
+ ASSERT_EQ(block.rows(), 3);
+ for (size_t column = 0; column < 2; ++column) {
+ for (size_t row = 0; row < 3; ++row) {
+ Field actual;
+ block.get_by_position(column).column->get(row, actual);
+ if (is_null) {
+ EXPECT_TRUE(actual.is_null());
+ } else {
+ const auto& text = actual.get<TYPE_STRING>();
+ EXPECT_EQ(std::string(text.data(), text.size()), "abc");
+ }
+ }
+ }
+ }
+ }
+}
+
+TEST_F(PaimonRustTableReaderTest, DirectPathTruncatesBoundedStringColumns) {
+ // The pinned rust reader maps paimon CHAR(n)/VARCHAR(n) to lengthless Arrow
+ // Utf8, so the direct Arrow path must enforce truncate_char_or_varchar_
+ // columns itself: a value written under an old wider schema and read into
+ // a column narrowed to VARCHAR(3) truncates to "abc" — the semantics
+ // TableReader::finalize_chunk applies on the normal path, which this
+ // direct path bypasses. Unbounded STRING columns and non-string columns
+ // stay untouched, and with the option off nothing truncates.
+ const auto fill_block = [] {
+ const auto varchar3 =
+ make_nullable(std::make_shared<DataTypeString>(3, PrimitiveType::TYPE_VARCHAR));
+ const auto string_type = make_nullable(std::make_shared<DataTypeString>());
+ const auto int_type = std::make_shared<DataTypeInt32>();
+ Block block = Block({ColumnWithTypeAndName(varchar3->create_column(), varchar3, "v"),
+ ColumnWithTypeAndName(string_type->create_column(), string_type, "s"),
+ ColumnWithTypeAndName(int_type->create_column(), int_type, "k")});
+ for (auto [idx, field] : std::initializer_list<std::pair<size_t, Field>> {
+ {0, Field::create_field<TYPE_STRING>("abcdefghij")},
+ {1, Field::create_field<TYPE_STRING>("0123456789ABCDEF")},
+ {2, Field::create_field<TYPE_INT>(1)},
+ {0, Field::create_field<TYPE_STRING>("ab")},
+ {1, Field::create_field<TYPE_STRING>("xyz")},
+ {2, Field::create_field<TYPE_INT>(2)}}) {
+ auto column = IColumn::mutate(block.get_by_position(idx).column);
+ column->insert(field);
+ block.get_by_position(idx).column = std::move(column);
+ }
+ return block;
+ };
+ const auto value_at = [](const Block& block, size_t idx, size_t row) {
+ Field field;
+ block.get_by_position(idx).column->get(row, field);
+ const auto& value = field.get<TYPE_STRING>();
+ return std::string(value.data(), value.size());
+ };
+
+ // Option off (the SetUp default): historical values return unchanged.
+ {
+ PaimonRustTableReader reader;
+ ASSERT_TRUE(init_reader_with_count(&reader, std::vector<GlobalIndex> {}).ok());
+ Block block = fill_block();
+ ASSERT_TRUE(reader.TEST_truncate_char_or_varchar_columns(&block).ok());
+ EXPECT_EQ(value_at(block, 0, 0), "abcdefghij");
+ }
+
+ // Option on: the VARCHAR(3) column truncates to its declared length; the
+ // unbounded STRING column and the INT column stay untouched.
+ _query_options.__set_truncate_char_or_varchar_columns(true);
+ _runtime_state = RuntimeState::create_unique(_query_options, _query_globals);
+ {
+ PaimonRustTableReader reader;
+ ASSERT_TRUE(init_reader_with_count(&reader, std::vector<GlobalIndex> {}).ok());
+ Block block = fill_block();
+ ASSERT_TRUE(reader.TEST_truncate_char_or_varchar_columns(&block).ok());
+ EXPECT_EQ(value_at(block, 0, 0), "abc");
+ EXPECT_EQ(value_at(block, 0, 1), "ab");
+ EXPECT_EQ(value_at(block, 1, 0), "0123456789ABCDEF");
+ EXPECT_EQ(value_at(block, 1, 1), "xyz");
+ Field int_field;
+ block.get_by_position(2).column->get(0, int_field);
+ EXPECT_EQ(int_field.get<TYPE_INT>(), 1);
+ }
+}
+
+TEST_F(PaimonRustTableReaderTest, MaterializesInSessionTimezone) {
+ // TIMESTAMP_LTZ values materialize as session-local civil times: the
+ // materialization timezone must come from the session, not a fixed default
+ // (epoch 0 reads as 08:00 in a +08:00 session and 00:00 in UTC).
+ const auto hour_of_epoch_zero = [](const cctz::time_zone& tz) {
+ return tz.lookup(cctz::time_point<cctz::seconds>(std::chrono::seconds(0))).cs.hour();
+ };
+
+ PaimonRustTableReader reader;
+ _runtime_state->set_timezone("+08:00");
+ ASSERT_TRUE(init_reader_with_count(&reader, std::vector<GlobalIndex> {}).ok());
+ EXPECT_EQ(hour_of_epoch_zero(reader.TEST_ctz()), 8);
+
+ PaimonRustTableReader utc_reader;
+ _runtime_state->set_timezone("UTC");
+ ASSERT_TRUE(init_reader_with_count(&utc_reader, std::vector<GlobalIndex> {}).ok());
+ EXPECT_EQ(hour_of_epoch_zero(utc_reader.TEST_ctz()), 0);
+}
+
+TEST_F(PaimonRustTableReaderTest, OptionLogRendersKeysOnly) {
+ // Storage-option values must never reach the log: credential keys arrive
+ // under many spellings and cases (AWS_SECRET_KEY, AWS_TOKEN,
+ // fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name blocklist
+ // that misses one alias leaks the value, so the diagnostics rendering
+ // prints key names only.
+ const std::map<std::string, std::string> options {
+ {"AWS_ACCESS_KEY", "admin"},
+ {"AWS_SECRET_KEY", "leak-if-logged-1"},
+ {"AWS_TOKEN", "leak-if-logged-2"},
+ {"fs.oss.accessKeySecret", "leak-if-logged-3"},
+ {"s3.access-key", "leak-if-logged-4"},
+ {"s3.secret-key", "leak-if-logged-5"},
+ {"s3.endpoint", "http://leak-if-logged-6:19001"},
+ };
+ const std::string rendered = PaimonRustTableReader::TEST_format_options(options);
+
+ // Every key is rendered, no value and no '=' separator ever is.
+ EXPECT_NE(rendered.find("AWS_SECRET_KEY"), std::string::npos);
+ EXPECT_NE(rendered.find("fs.oss.accessKeySecret"), std::string::npos);
+ EXPECT_NE(rendered.find("s3.secret-key"), std::string::npos);
+ EXPECT_EQ(rendered.find('='), std::string::npos);
+ for (int i = 1; i <= 6; ++i) {
+ EXPECT_EQ(rendered.find("leak-if-logged-" + std::to_string(i)), std::string::npos)
+ << rendered;
+ }
+ EXPECT_EQ(rendered.find("admin"), std::string::npos) << rendered;
+}
+
+TEST_F(PaimonRustTableReaderTest, MapsAnonymousAndAssumeRoleProviderModes) {
+ // The FE storage-properties channel marks anonymous access with
+ // AWS_CREDENTIALS_PROVIDER_TYPE=ANONYMOUS and assume-role with
+ // AWS_ROLE_ARN / AWS_EXTERNAL_ID; the pinned rust S3 parser reads
+ // s3.anonymous (skip_signature) and the s3.assumed.role.* family, so the
+ // bridge must map both — without the mapping, anonymous catalogs would
+ // consult the ambient credential chain and role-only catalogs would
+ // never assume the requested role. The ambient JVM provider modes (ENV,
+ // SYSTEM_PROPERTIES, WEB_IDENTITY, CONTAINER, INSTANCE_PROFILE) have no
+ // rust equivalent and are gated away from the rust reader on the FE.
+ TFileScanRangeParams params;
+ params.properties["AWS_CREDENTIALS_PROVIDER_TYPE"] = "ANONYMOUS";
+ params.properties["AWS_ROLE_ARN"] = "arn:aws:iam::123:role/reader";
+ params.properties["AWS_EXTERNAL_ID"] = "external-123";
+ params.properties["AWS_ACCESS_KEY"] = "admin";
+ params.properties["AWS_SECRET_KEY"] = "password";
+ params.properties["AWS_ENDPOINT"] = "http://127.0.0.1:19001";
+ params.properties["AWS_REGION"] = "us-east-1";
+ params.properties["use_path_style"] = "true";
+ params.__isset.properties = true;
+
+ PaimonRustTableReader reader;
+ const auto options = reader.TEST_build_options(
+ ¶ms, make_rust_range_with_table_path("s3://bucket/wh/db.db/t"));
+ EXPECT_EQ(options.at("s3.anonymous"), "true");
+ EXPECT_EQ(options.at("s3.assumed.role.arn"), "arn:aws:iam::123:role/reader");
+ EXPECT_EQ(options.at("s3.assumed.role.externalId"), "external-123");
+ // Static credentials and connection settings keep mapping (regression
+ // for the base remap).
+ EXPECT_EQ(options.at("s3.access-key"), "admin");
+ EXPECT_EQ(options.at("s3.secret-key"), "password");
+ EXPECT_EQ(options.at("s3.endpoint"), "http://127.0.0.1:19001");
+ EXPECT_EQ(options.at("s3.region"), "us-east-1");
+ EXPECT_EQ(options.at("s3.path-style-access"), "true");
+}
+
+TEST_F(PaimonRustTableReaderTest, StaticCredentialsMapWithoutProviderMode) {
+ // Default shape: no provider-mode marker, plain static credentials.
+ TFileScanRangeParams params;
+ params.properties["AWS_ACCESS_KEY"] = "admin";
+ params.properties["AWS_SECRET_KEY"] = "password";
+ params.__isset.properties = true;
+
+ PaimonRustTableReader reader;
+ const auto options = reader.TEST_build_options(
+ ¶ms, make_rust_range_with_table_path("s3://bucket/wh/db.db/t"));
+ EXPECT_EQ(options.at("s3.access-key"), "admin");
+ EXPECT_EQ(options.at("s3.secret-key"), "password");
+ EXPECT_EQ(options.count("s3.anonymous"), 0);
+}
+
+TEST_F(PaimonRustTableReaderTest, OssSchemeMapsToOssFileIOKeys) {
+ // The pinned storage dispatcher selects the parser from the table path's
+ // URI scheme: oss:// tables read the OSS parser, which requires
+ // fs.oss.endpoint / fs.oss.accessKeyId / fs.oss.accessKeySecret (plus
+ // optional fs.oss.securityToken for STS). A production-shaped FE map
+ // (OSSProperties emits the AWS_* aliases plus the connection settings)
+ // must therefore map to the fs.oss.* family — mapping everything to
+ // s3.* leaves OSS catalogs failing to open ("Missing required OSS
+ // config: fs.oss.endpoint").
+ TFileRangeDesc range;
+ TTableFormatFileDesc table_format_params;
+ TPaimonFileDesc paimon_params;
+ paimon_params.__set_paimon_table("oss://bucket/wh/db.db/t");
+ table_format_params.__set_paimon_params(paimon_params);
+ range.__set_table_format_params(table_format_params);
+
+ TFileScanRangeParams params;
+ params.properties["AWS_ENDPOINT"] = "http://oss.internal:8080";
+ params.properties["AWS_ACCESS_KEY"] = "oss-admin";
+ params.properties["AWS_SECRET_KEY"] = "oss-password";
+ params.properties["AWS_TOKEN"] = "oss-sts-token";
+ params.properties["AWS_REGION"] = "cn-beijing";
+ params.properties["use_path_style"] = "true";
+ params.__isset.properties = true;
+
+ PaimonRustTableReader reader;
+ const auto options = reader.TEST_build_options(¶ms, range);
+ EXPECT_EQ(options.at("fs.oss.endpoint"), "http://oss.internal:8080");
+ EXPECT_EQ(options.at("fs.oss.accessKeyId"), "oss-admin");
+ EXPECT_EQ(options.at("fs.oss.accessKeySecret"), "oss-password");
+ EXPECT_EQ(options.at("fs.oss.securityToken"), "oss-sts-token");
+ // The OSS parser reads no s3.* keys; nothing but the fs.oss.* family may
+ // be synthesized for this scheme.
+ EXPECT_EQ(options.count("s3.access-key"), 0);
+ EXPECT_EQ(options.count("s3.endpoint"), 0);
+ EXPECT_EQ(options.count("s3.path-style-access"), 0);
+}
+
+TEST_F(PaimonRustTableReaderTest, OssSchemeKeepsNativeFsOssKeysUnmapped) {
+ // Native fs.oss.* options (delivered by configs that already speak the
+ // paimon-java OSS dialect) must pass through untouched instead of being
+ // remapped into the s3.* family.
+ TFileRangeDesc range;
+ TTableFormatFileDesc table_format_params;
+ TPaimonFileDesc paimon_params;
+ paimon_params.__set_paimon_table("oss://bucket/wh/db.db/t");
+ table_format_params.__set_paimon_params(paimon_params);
+ range.__set_table_format_params(table_format_params);
+
+ TFileScanRangeParams params;
+ params.properties["fs.oss.endpoint"] = "http://oss.internal:8080";
+ params.properties["fs.oss.accessKeyId"] = "oss-admin";
+ params.properties["fs.oss.accessKeySecret"] = "oss-password";
+ params.__isset.properties = true;
+
+ PaimonRustTableReader reader;
+ const auto options = reader.TEST_build_options(¶ms, range);
+ EXPECT_EQ(options.at("fs.oss.endpoint"), "http://oss.internal:8080");
+ EXPECT_EQ(options.at("fs.oss.accessKeyId"), "oss-admin");
+ EXPECT_EQ(options.at("fs.oss.accessKeySecret"), "oss-password");
+ EXPECT_EQ(options.count("s3.access-key"), 0);
+}
+
+TEST_F(PaimonRustTableReaderTest, S3aSchemeMapsToS3FileIOKeys) {
+ // The crate dispatches both s3:// and s3a:// to the S3 parser, so the
+ // legacy Hadoop-style spelling must translate to the s3.* family too.
+ TFileScanRangeParams params;
+ params.properties["AWS_ACCESS_KEY"] = "admin";
+ params.properties["AWS_SECRET_KEY"] = "password";
+ params.properties["AWS_ENDPOINT"] = "http://127.0.0.1:19001";
+ params.__isset.properties = true;
+
+ PaimonRustTableReader reader;
+ const auto options = reader.TEST_build_options(
+ ¶ms, make_rust_range_with_table_path("s3a://bucket/wh/db.db/t"));
+ EXPECT_EQ(options.at("s3.access-key"), "admin");
+ EXPECT_EQ(options.at("s3.secret-key"), "password");
+ EXPECT_EQ(options.at("s3.endpoint"), "http://127.0.0.1:19001");
+}
+
+TEST_F(PaimonRustTableReaderTest, UnverifiedSchemesPassThroughWithoutS3Aliases) {
+ // libpaimon_c.a compiles in separate COS, OBS, GCS and Azdls parsers,
+ // each reading its own key family (fs.cosn.userinfo.* / fs.obs.* /
+ // gcs.* / azure.*), while Doris's FE normalizes those object stores'
+ // credentials into the AWS_* aliases (the *Properties storage classes).
+ // The bridge only implements the s3.* / fs.oss.* translations, and the
+ // FE gates every other scheme to JNI before the split is encoded — so
+ // here nothing may be synthesized for them: a production-shaped AWS_*
+ // map must pass through untouched instead of being silently translated
+ // into s3.* keys its scheme's parser ignores (which would fail the open
+ // with a misleading auth error, e.g. "Missing required COS config").
+ const std::vector<std::string> gated_schemes {
+ "cosn://bucket/wh/db.db/t", "cos://bucket/wh/db.db/t", "obs://bucket/wh/db.db/t",
+ "gs://bucket/wh/db.db/t", "gcs://bucket/wh/db.db/t", "abfs://bucket/wh/db.db/t",
+ "abfss://bucket/wh/db.db/t", "az://bucket/wh/db.db/t", "azure://bucket/wh/db.db/t"};
+ for (const auto& table_path : gated_schemes) {
+ TFileScanRangeParams params;
+ params.properties["AWS_CREDENTIALS_PROVIDER_TYPE"] = "DEFAULT";
+ params.properties["AWS_ACCESS_KEY"] = "admin";
+ params.properties["AWS_SECRET_KEY"] = "password";
+ params.properties["AWS_ENDPOINT"] = "http://127.0.0.1:19001";
+ params.properties["AWS_REGION"] = "us-east-1";
+ params.properties["AWS_TOKEN"] = "sts-token";
+ params.properties["use_path_style"] = "true";
+ params.__isset.properties = true;
+
+ PaimonRustTableReader reader;
+ const auto options =
+ reader.TEST_build_options(¶ms, make_rust_range_with_table_path(table_path));
+ // No key of the s3.* / fs.oss.* families is synthesized for a scheme
+ // whose parser reads neither.
+ for (const char* key :
+ {"s3.access-key", "s3.secret-key", "s3.session.token", "s3.endpoint", "s3.region",
+ "s3.path-style-access", "s3.anonymous", "s3.assumed.role.arn",
+ "s3.assumed.role.externalId", "fs.oss.endpoint", "fs.oss.accessKeyId",
+ "fs.oss.accessKeySecret"}) {
+ EXPECT_EQ(options.count(key), 0) << "scheme " << table_path << " synthesized " << key;
+ }
+ // The FE-supplied property map passes through untouched.
+ EXPECT_EQ(options.at("AWS_ACCESS_KEY"), "admin");
+ EXPECT_EQ(options.at("AWS_SECRET_KEY"), "password");
+ EXPECT_EQ(options.at("AWS_ENDPOINT"), "http://127.0.0.1:19001");
+ }
+}
+
+TEST_F(PaimonRustTableReaderTest, HdfsAndLocalSchemesPassThroughHadoopConf) {
+ // hdfs:// tables read their hadoop conf as delivered (the native HDFS
+ // parser has its own config keys) and credential-free local paths need
+ // nothing at all: no AWS_* -> s3.* aliasing may be synthesized for
+ // either, even if an S3-shaped property map is present.
+ const std::vector<std::string> passthrough_schemes {"hdfs://nn/wh/db.db/t",
+ "/paimon/warehouse/db.db/t"};
+ for (const auto& table_path : passthrough_schemes) {
+ TFileScanRangeParams params;
+ params.properties["AWS_ACCESS_KEY"] = "admin";
+ params.properties["AWS_SECRET_KEY"] = "password";
+ params.properties["AWS_ENDPOINT"] = "http://127.0.0.1:19001";
+ params.__isset.properties = true;
+
+ PaimonRustTableReader reader;
+ const auto options =
+ reader.TEST_build_options(¶ms, make_rust_range_with_table_path(table_path));
+ EXPECT_EQ(options.count("s3.access-key"), 0) << table_path;
+ EXPECT_EQ(options.count("s3.secret-key"), 0) << table_path;
+ EXPECT_EQ(options.count("s3.endpoint"), 0) << table_path;
+ EXPECT_EQ(options.count("fs.oss.accessKeyId"), 0) << table_path;
+ EXPECT_EQ(options.at("AWS_ACCESS_KEY"), "admin");
+ }
+}
+
+} // namespace doris::format::paimon
diff --git a/build.sh b/build.sh
index 270c64f11d0..8449eab271a 100755
--- a/build.sh
+++ b/build.sh
@@ -447,7 +447,22 @@ else
LAST_THIRDPARTY_LIB='hadoop_hdfs/native/libhdfs.a'
fi
-if [[ ! -f "${DORIS_THIRDPARTY}/installed/lib/${LAST_THIRDPARTY_LIB}" ]]; then
+# Old installs can contain the sentinel but predate Paimon. Rebuild the whole
+# dependency set so Lance and Paimon also use the same selected Rust toolchain.
+# An interrupted header/archive publication must be repaired before either file is reused.
+if [[ ! -f "${DORIS_THIRDPARTY}/installed/lib/${LAST_THIRDPARTY_LIB}" ||
+ ! -f "${DORIS_THIRDPARTY}/installed/lib64/liblance_c.a" ||
+ ! -f "${DORIS_THIRDPARTY}/installed/lib64/libpaimon_c.a" ||
+ ! -f "${DORIS_THIRDPARTY}/installed/include/paimon_rust/paimon.h" ||
+ ! -s "${DORIS_THIRDPARTY}/installed/lib64/libpaimon_c.a" ||
+ ! -s "${DORIS_THIRDPARTY}/installed/include/paimon_rust/paimon.h" ||
+ -e "${DORIS_THIRDPARTY}/installed/lib64/.paimon-installing" ]]; then
+ # Compilation images may contain only installed artifacts; never erase them without a rebuild source.
+ if [[ ! -f "${DORIS_THIRDPARTY}/build-thirdparty.sh" ]]; then
+ echo "Third-party dependencies require a rebuild, but build-thirdparty.sh is missing." >&2
+ echo "Refresh the compilation image or set DORIS_THIRDPARTY to a complete third-party source tree." >&2
+ exit 1
+ fi
echo "Thirdparty libraries need to be build ..."
# need remove all installed pkgs because some lib like lz4 will throw error if its lib alreay exists
rm -rf "${DORIS_THIRDPARTY}/installed"
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/profile/ExecutionProfile.java b/fe/fe-core/src/main/java/org/apache/doris/common/profile/ExecutionProfile.java
index 2f6b0992ef6..185fce52d35 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/profile/ExecutionProfile.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/profile/ExecutionProfile.java
@@ -35,9 +35,11 @@ import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import java.util.ArrayList;
+import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
+import java.util.Set;
import java.util.concurrent.locks.ReentrantReadWriteLock;
/**
@@ -71,6 +73,7 @@ public class ExecutionProfile {
// use to merge profile from multi be
private Map<Integer, Map<TNetworkAddress, List<RuntimeProfile>>> multiBeProfile = null;
+ private Map<Integer, Set<TNetworkAddress>> fragmentIdDoneBackends;
private ReentrantReadWriteLock multiBeProfileLock = new ReentrantReadWriteLock();
// Not serialize this property, it is only used to get profile id.
@@ -88,6 +91,7 @@ public class ExecutionProfile {
root.addChild(fragmentsProfile, true);
fragmentProfiles = Maps.newHashMap();
multiBeProfile = Maps.newHashMap();
+ fragmentIdDoneBackends = Maps.newHashMap();
fragmentIdBeNum = Maps.newHashMap();
seqNoToFragmentId = Maps.newHashMap();
int i = 0;
@@ -96,6 +100,7 @@ public class ExecutionProfile {
fragmentProfiles.put(fragmentId, runtimeProfile);
fragmentsProfile.addChild(runtimeProfile, true);
multiBeProfile.put(fragmentId, Maps.newHashMap());
+ fragmentIdDoneBackends.put(fragmentId, new HashSet<>());
fragmentIdBeNum.put(fragmentId, 0);
seqNoToFragmentId.put(i, fragmentId);
++i;
@@ -147,7 +152,7 @@ public class ExecutionProfile {
}
protected void setMultiBeProfile(int fragmentId, TNetworkAddress backendHBAddress,
- List<RuntimeProfile> taskProfile) {
+ List<RuntimeProfile> taskProfile) {
multiBeProfileLock.writeLock().lock();
try {
multiBeProfile.get(fragmentId).put(backendHBAddress, taskProfile);
@@ -156,6 +161,40 @@ public class ExecutionProfile {
}
}
+ private boolean updateMultiBeProfile(int fragmentId, TNetworkAddress backendHBAddress,
+ List<RuntimeProfile> taskProfile, List<RuntimeProfile> profileNodes, boolean isDone) {
+ multiBeProfileLock.writeLock().lock();
+ try {
+ if (!isDone && fragmentIdDoneBackends.get(fragmentId).contains(backendHBAddress)) {
+ return false;
+ }
+ multiBeProfile.get(fragmentId).put(backendHBAddress, taskProfile);
+ // Publish both rendered forms before completion becomes visible to profile readers.
+ for (RuntimeProfile profileNode : profileNodes) {
+ fragmentProfiles.get(fragmentId).addChild(profileNode, true);
+ }
+ if (isDone) {
+ fragmentIdDoneBackends.get(fragmentId).add(backendHBAddress);
+ }
+ return true;
+ } finally {
+ multiBeProfileLock.writeLock().unlock();
+ }
+ }
+
+ private boolean areAllBackendProfilesDone(int fragmentId, int expectedBackendNum) {
+ if (expectedBackendNum == 0) {
+ return false;
+ }
+
+ multiBeProfileLock.readLock().lock();
+ try {
+ return fragmentIdDoneBackends.get(fragmentId).size() >= expectedBackendNum;
+ } finally {
+ multiBeProfileLock.readLock().unlock();
+ }
+ }
+
protected RuntimeProfile getPipelineAggregatedProfile(Map<Integer, String> planNodeMap) {
RuntimeProfile fragmentsProfile = new RuntimeProfile("Fragments");
for (int i = 0; i < fragmentProfiles.size(); ++i) {
@@ -233,6 +272,7 @@ public class ExecutionProfile {
int fragmentId = entry.getKey();
List<TDetailedReportParams> fragmentProfile = entry.getValue();
int pipelineIdx = 0;
+ List<RuntimeProfile> profileNodes = Lists.newArrayList();
List<RuntimeProfile> taskProfile = Lists.newArrayList();
String suffix = "(host=" + backendHBAddress + ")";
for (TDetailedReportParams pipelineProfile : fragmentProfile) {
@@ -259,9 +299,11 @@ public class ExecutionProfile {
profileNode.update(pipelineProfile.profile);
profileNode.setIsDone(isDone);
- fragmentProfiles.get(fragmentId).addChild(profileNode, true);
+ profileNodes.add(profileNode);
+ }
+ if (!updateMultiBeProfile(fragmentId, backendHBAddress, taskProfile, profileNodes, isDone)) {
+ continue;
}
- setMultiBeProfile(fragmentId, backendHBAddress, taskProfile);
}
LOG.info("Profile update finished query: {} fragments: {} isDone: {}",
@@ -277,29 +319,24 @@ public class ExecutionProfile {
}
public synchronized void addFragmentBackend(PlanFragmentId fragmentId, Long backendId) {
- fragmentIdBeNum.put(fragmentId.asInt(), fragmentIdBeNum.get(fragmentId.asInt()) + 1);
+ addFragmentBackend(fragmentId.asInt(), backendId);
+ }
+
+ public synchronized void addFragmentBackend(int fragmentId, Long backendId) {
+ fragmentIdBeNum.put(fragmentId, fragmentIdBeNum.get(fragmentId) + 1);
}
public TUniqueId getQueryId() {
return queryId;
}
- // Check all fragments's child, if all finished, then this execution profile is finished
+ // Check all fragments' backend reports. A backend can contribute multiple profile nodes.
public boolean isCompleted() {
for (Entry<Integer, RuntimeProfile> element : fragmentProfiles.entrySet()) {
- RuntimeProfile fragmentProfile = element.getValue();
- // If any fragment is empty, it means BE does not report the profile, then the total
- // execution profile is not completed.
- if (fragmentProfile.isEmpty()
- || fragmentProfile.getChildList().size() < fragmentIdBeNum.get(element.getKey())) {
+ int fragmentId = element.getKey();
+ if (!areAllBackendProfilesDone(fragmentId, fragmentIdBeNum.get(fragmentId))) {
return false;
}
- for (Pair<RuntimeProfile, Boolean> runtimeProfile : fragmentProfile.getChildList()) {
- // If any child instance profile is not ready, then return false.
- if (!(runtimeProfile.first.getIsDone() || runtimeProfile.first.getIsCancel())) {
- return false;
- }
- }
}
return true;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/profile/Profile.java b/fe/fe-core/src/main/java/org/apache/doris/common/profile/Profile.java
index f8e5b7884a3..a7e84f78487 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/profile/Profile.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/profile/Profile.java
@@ -117,6 +117,8 @@ public class Profile {
// 1. profile is stored to storage
// 2. or profile is loaded from storage
private String profileStoragePath = "";
+ private volatile String storageProfileCompletionState;
+ private volatile boolean executionProfilesReleased;
// isQueryFinished means the coordinator or stmt executor is finished.
// does not mean the profile report has finished, since the report is async.
// finish of collection of profile is marked by isCompleted of ExecutionProfiles.
@@ -346,6 +348,7 @@ public class Profile {
LOG.info("DebugPoint:Profile.profileSizeLimit, MAX_PROFILE_SIZE = {}", maxProfileSize);
}
// add summary to builder
+ updateProfileCompletionStateForDisplay();
summaryProfile.prettyPrint(builder);
if (!builder.isTruncated()) {
getChangedSessionVars(builder);
@@ -557,6 +560,9 @@ public class Profile {
}
public void releaseMemory() {
+ // The manager also releases reports after a failed spill. An empty list then means
+ // lost reports, not successful collection; retain that distinction across retries.
+ executionProfilesReleased = true;
this.executionProfiles.clear();
this.changedSessionVarCache = "";
this.physicalPlan = null;
@@ -619,6 +625,66 @@ public class Profile {
return !Strings.isNullOrEmpty(profileStoragePath);
}
+ // Query completion precedes asynchronous BE reports; persisted profiles must retain
+ // their terminal state even after the in-memory reports have been released.
+ public String getProfileCompletionState() {
+ String storageState = storageProfileCompletionState;
+ if (storageState != null) {
+ return storageState;
+ }
+ if (profileHasBeenStored()) {
+ String storedState = summaryProfile.getSummary().getInfoString(SummaryProfile.PROFILE_COMPLETION_STATE);
+ if (!Strings.isNullOrEmpty(storedState)) {
+ return storedState;
+ }
+ return SummaryProfile.PROFILE_COMPLETION_STATE_UNKNOWN;
+ }
+
+ if (executionProfilesReleased) {
+ return SummaryProfile.PROFILE_COMPLETION_STATE_INCOMPLETE;
+ }
+
+ if (!isQueryFinished) {
+ return SummaryProfile.PROFILE_COMPLETION_STATE_RUNNING;
+ }
+
+ for (int i = 0; i < executionProfiles.size(); i++) {
+ ExecutionProfile executionProfile = executionProfiles.get(i);
+ if (!executionProfile.isCompleted()) {
+ return SummaryProfile.PROFILE_COMPLETION_STATE_COLLECTING;
+ }
+ }
+ return SummaryProfile.PROFILE_COMPLETION_STATE_COMPLETE;
+ }
+
+ private String getProfileCompletionStateForStorage() {
+ if (executionProfilesReleased) {
+ return SummaryProfile.PROFILE_COMPLETION_STATE_INCOMPLETE;
+ }
+ if (!isQueryFinished) {
+ return SummaryProfile.PROFILE_COMPLETION_STATE_RUNNING;
+ }
+
+ for (int i = 0; i < executionProfiles.size(); i++) {
+ ExecutionProfile executionProfile = executionProfiles.get(i);
+ if (!executionProfile.isCompleted()) {
+ return SummaryProfile.PROFILE_COMPLETION_STATE_INCOMPLETE;
+ }
+ }
+ return SummaryProfile.PROFILE_COMPLETION_STATE_COMPLETE;
+ }
+
+ private synchronized void updateProfileCompletionStateForStorage() {
+ // Freeze the terminal state before serialization. Concurrent rendering must not
+ // replace INCOMPLETE with COLLECTING while the storage path is still unpublished.
+ storageProfileCompletionState = getProfileCompletionStateForStorage();
+ summaryProfile.setProfileCompletionState(storageProfileCompletionState);
+ }
+
+ private synchronized void updateProfileCompletionStateForDisplay() {
+ summaryProfile.setProfileCompletionState(getProfileCompletionState());
+ }
+
// Profile IO threads races with Coordinator threads.
public void markQueryFinished() {
try {
@@ -668,6 +734,7 @@ public class Profile {
DataOutputStream memoryDataStream = new DataOutputStream(memoryStream);
// Write summary profile and execution profile content to memory
+ updateProfileCompletionStateForStorage();
this.summaryProfile.write(memoryDataStream);
SafeStringBuilder builder = new SafeStringBuilder();
@@ -688,6 +755,7 @@ public class Profile {
this.profileStoragePath = profileFilePath;
} catch (Exception e) {
+ storageProfileCompletionState = null;
LOG.error("write {} summary profile failed", getId(), e);
return;
} finally {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java b/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java
index 15e5503bced..fe703ddde0f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java
@@ -62,6 +62,12 @@ public class SummaryProfile {
public static final String END_TIME = "End Time";
public static final String TOTAL_TIME = "Total";
public static final String TASK_STATE = "Task State";
+ public static final String PROFILE_COMPLETION_STATE = "Profile Completion State";
+ public static final String PROFILE_COMPLETION_STATE_RUNNING = "RUNNING";
+ public static final String PROFILE_COMPLETION_STATE_COLLECTING = "COLLECTING";
+ public static final String PROFILE_COMPLETION_STATE_COMPLETE = "COMPLETE";
+ public static final String PROFILE_COMPLETION_STATE_INCOMPLETE = "INCOMPLETE";
+ public static final String PROFILE_COMPLETION_STATE_UNKNOWN = "UNKNOWN";
public static final String USER = "User";
public static final String DEFAULT_CATALOG = "Default Catalog";
public static final String DEFAULT_DB = "Default Db";
@@ -470,6 +476,10 @@ public class SummaryProfile {
return summaryProfile;
}
+ public void setProfileCompletionState(String profileCompletionState) {
+ summaryProfile.addInfoString(PROFILE_COMPLETION_STATE, profileCompletionState);
+ }
+
public RuntimeProfile getExecutionSummary() {
return executionSummaryProfile;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitor.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitor.java
index 9eca19a1584..464a1c02ea0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitor.java
@@ -28,6 +28,7 @@ import org.apache.doris.datasource.DorisTypeVisitor;
import org.apache.paimon.types.BigIntType;
import org.apache.paimon.types.BooleanType;
+import org.apache.paimon.types.CharType;
import org.apache.paimon.types.DataField;
import org.apache.paimon.types.DataType;
import org.apache.paimon.types.DateType;
@@ -98,17 +99,21 @@ public class DorisToPaimonTypeVisitor extends DorisTypeVisitor<DataType> {
return new FloatType();
} else if (primitiveType.equals(PrimitiveType.DOUBLE)) {
return new DoubleType();
- } else if (primitiveType.isCharFamily()) {
+ } else if (primitiveType.equals(PrimitiveType.CHAR)) {
+ // DDL must retain bounds: widening to STRING changes historical narrowing casts.
+ return new CharType(((ScalarType) atomic).getLength());
+ } else if (primitiveType.equals(PrimitiveType.VARCHAR)) {
+ return new VarCharType(((ScalarType) atomic).getLength());
+ } else if (primitiveType.equals(PrimitiveType.STRING)) {
return new VarCharType(VarCharType.MAX_LENGTH);
} else if (primitiveType.equals(PrimitiveType.DATE) || primitiveType.equals(PrimitiveType.DATEV2)) {
return new DateType();
} else if (primitiveType.equals(PrimitiveType.DECIMALV2) || primitiveType.isDecimalV3Type()) {
return new DecimalType(((ScalarType) atomic).getScalarPrecision(), ((ScalarType) atomic).getScalarScale());
- } else if (primitiveType.equals(PrimitiveType.DATETIME)) {
- return new TimestampType();
- } else if (primitiveType.equals(PrimitiveType.DATETIMEV2)) {
- // Preserve wall-clock semantics and fractional precision in the Paimon schema.
- return new TimestampType(((ScalarType) atomic).getScalarScale());
+ } else if (primitiveType.equals(PrimitiveType.DATETIME) || primitiveType.equals(PrimitiveType.DATETIMEV2)) {
+ // Paimon interprets numeric timestamp strings in units derived from this precision.
+ return new TimestampType(primitiveType.equals(PrimitiveType.DATETIMEV2)
+ ? ((ScalarType) atomic).getScalarScale() : 0);
} else if (primitiveType.equals(PrimitiveType.TIMESTAMPTZ)) {
// TIMESTAMPTZ represents an instant, so map it to Paimon's local-zoned timestamp.
return new LocalZonedTimestampType(((ScalarType) atomic).getScalarScale());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonScanParams.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonScanParams.java
index 7a1b9718c00..2905952aa79 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonScanParams.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonScanParams.java
@@ -24,6 +24,7 @@ import org.apache.paimon.CoreOptions;
import org.apache.paimon.Snapshot;
import org.apache.paimon.options.ConfigOption;
import org.apache.paimon.options.FallbackKey;
+import org.apache.paimon.schema.TableSchema;
import org.apache.paimon.table.FileStoreTable;
import org.apache.paimon.table.Table;
import org.apache.paimon.table.source.snapshot.FullCompactedStartingScanner;
@@ -74,6 +75,28 @@ public final class PaimonScanParams {
private static final Set<String> INHERITED_READ_STATE_KEYS = inheritedReadStateKeys();
+ // The selectors the pinned paimon-rust reader re-resolves in copy_with_time_travel; the
+ // inverse of isolateSnapshotRead for schema transport (see withoutTimeTravelSelectors).
+ private static final Set<String> TIME_TRAVEL_SELECTOR_KEYS = ImmutableSet.of(
+ CoreOptions.SCAN_TIMESTAMP_MILLIS.key(),
+ CoreOptions.SCAN_WATERMARK.key(),
+ CoreOptions.SCAN_VERSION.key(),
+ CoreOptions.SCAN_SNAPSHOT_ID.key(),
+ CoreOptions.SCAN_TAG_NAME.key());
+
+ // Paimon's copyInternal materializes the derived scan mode into the copied schema's
+ // options when the merged options carry a selector — a statement fence pinning
+ // scan.snapshot-id leaves a bare "scan.mode=from-snapshot" in schema.options(). Each
+ // of these modes requires a selector (the rust ReadBuilder rejects them otherwise),
+ // so withoutTimeTravelSelectors drops them together with the selector they describe.
+ private static final Set<String> SELECTOR_DEPENDENT_SCAN_MODES = ImmutableSet.of(
+ "from-snapshot",
+ "from-snapshot-full",
+ "from-timestamp",
+ "from-timestamp-full",
+ "from-file-creation-time",
+ "from-creation-timestamp");
+
// FilesScan enumerates the latest partitions before applying its range-aware per-partition scan,
// so it cannot safely read a range when a partition in that range has since been dropped.
private static final Set<String> INCREMENTAL_SYSTEM_TABLES = ImmutableSet.of(
@@ -403,6 +426,42 @@ public final class PaimonScanParams {
return isolatedOptions;
}
+ /**
+ * Strips the time-travel planning selectors — and any derived scan mode Paimon
+ * materialized for them — from a resolved schema's options, keeping every
+ * field and every other option intact. This is the transport-side inverse of
+ * {@link #isolateSnapshotRead(long)}: a statement fence pins the data snapshot by merging
+ * {@code scan.snapshot-id} into the schema options, but the pinned paimon-rust reader
+ * re-resolves a transported selector inside {@code copy_with_time_travel} and swaps the
+ * shipped fields for the pinned snapshot's older schema — a column added after the last
+ * data commit then fails projection before per-file schema evolution can fill it. The rust
+ * reader instead pins data through the serialized DataSplit, so these selectors are
+ * planning-only state and must not be transported.
+ */
+ public static TableSchema withoutTimeTravelSelectors(TableSchema schema) {
+ if (TIME_TRAVEL_SELECTOR_KEYS.stream().noneMatch(schema.options()::containsKey)) {
+ return schema;
+ }
+ Map<String, String> options = new HashMap<>(schema.options());
+ TIME_TRAVEL_SELECTOR_KEYS.forEach(options.keySet()::remove);
+ // Paimon's copyInternal may have materialized the derived scan mode for the
+ // stripped selector; without the selector the rust ReadBuilder validation
+ // ("from-snapshot requires one of scan.snapshot-id, ... to be set") rejects
+ // the open. The reader instead pins data through the serialized DataSplit.
+ // Paimon accepts enum option values case-insensitively but preserves the
+ // original spelling in the options map, so a table can persist
+ // scan.mode=FROM-SNAPSHOT; the mode is normalized before the membership
+ // check because the rust reader recognizes the value case-insensitively and
+ // would reject the now-bare uppercase mode for lacking its selector, while
+ // JNI accepts the original option.
+ String scanMode = options.get(CoreOptions.SCAN_MODE.key());
+ if (scanMode != null && SELECTOR_DEPENDENT_SCAN_MODES.contains(
+ scanMode.trim().toLowerCase(Locale.ROOT))) {
+ options.remove(CoreOptions.SCAN_MODE.key());
+ }
+ return schema.copy(options);
+ }
+
private static boolean isCompatibleStartupMode(String position, String mode) {
if ("default".equals(mode)) {
return true;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java
index 649efdd3e31..a3506783707 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java
@@ -54,6 +54,7 @@ import org.apache.paimon.data.BinaryRow;
import org.apache.paimon.data.InternalRow;
import org.apache.paimon.data.Timestamp;
import org.apache.paimon.data.serializer.InternalRowSerializer;
+import org.apache.paimon.io.DataOutputViewStreamWrapper;
import org.apache.paimon.manifest.PartitionEntry;
import org.apache.paimon.options.ConfigOption;
import org.apache.paimon.partition.Partition;
@@ -64,6 +65,7 @@ import org.apache.paimon.table.DataTable;
import org.apache.paimon.table.FileStoreTable;
import org.apache.paimon.table.SpecialFields;
import org.apache.paimon.table.Table;
+import org.apache.paimon.table.source.DataSplit;
import org.apache.paimon.table.source.ReadBuilder;
import org.apache.paimon.tag.Tag;
import org.apache.paimon.types.ArrayType;
@@ -81,6 +83,7 @@ import org.apache.paimon.types.VarCharType;
import org.apache.paimon.utils.DateTimeUtils;
import org.apache.paimon.utils.InstantiationUtil;
import org.apache.paimon.utils.InternalRowPartitionComputer;
+import org.apache.paimon.utils.JsonSerdeUtil;
import org.apache.paimon.utils.Pair;
import org.apache.paimon.utils.PartitionPathUtils;
import org.apache.paimon.utils.Projection;
@@ -728,6 +731,36 @@ public class PaimonUtil {
private static final class SerializationSizeLimitException extends IOException {
}
+ /**
+ * Serialize DataSplit using Paimon's native binary format.
+ * This format is compatible with paimon-cpp reader.
+ * Uses standard Base64 encoding (not URL-safe) for BE compatibility.
+ */
+ public static String encodeDataSplitToString(DataSplit split) {
+ try {
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ DataOutputViewStreamWrapper out = new DataOutputViewStreamWrapper(baos);
+ split.serialize(out);
+ byte[] bytes = baos.toByteArray();
+ return Base64.getEncoder().encodeToString(bytes);
+ } catch (IOException e) {
+ throw new RuntimeException("Failed to serialize DataSplit using Paimon native format", e);
+ }
+ }
+
+ /**
+ * Serialize a paimon {@link TableSchema} to JSON via Paimon's registered
+ * serde. Output is identical to an on-disk schema/schema-N file and is
+ * consumed by the paimon-rust BE reader through paimon_table_from_schema_json.
+ */
+ public static String encodeTableSchemaToJson(TableSchema tableSchema) {
+ try {
+ return JsonSerdeUtil.toJson(tableSchema);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to serialize paimon TableSchema to JSON", e);
+ }
+ }
+
public static Map<String, String> getPartitionInfoMap(Table table, BinaryRow partitionValues, String timeZone) {
return getPartitionInfoMap(table, partitionValues, timeZone, false);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonRustReaderCapabilities.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonRustReaderCapabilities.java
new file mode 100644
index 00000000000..d45ff4ac70c
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonRustReaderCapabilities.java
@@ -0,0 +1,282 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon.source;
+
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+
+import com.google.common.collect.ImmutableSet;
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.types.ArrayType;
+import org.apache.paimon.types.DataField;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.DataTypeChecks;
+import org.apache.paimon.types.DataTypeRoot;
+import org.apache.paimon.types.DecimalType;
+import org.apache.paimon.types.MapType;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.types.VarCharType;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/** Compatibility checks for the pinned paimon-rust reader, beyond storage capabilities. */
+final class PaimonRustReaderCapabilities {
+ private static final Set<String> SUPPORTED_AGGREGATE_NAMES = ImmutableSet.of(
+ "sum", "product", "min", "max", "last_value", "first_value", "last_non_null_value",
+ "first_non_null_value", "first_not_null_value", "bool_and", "bool_or", "listagg");
+ private final FileStoreTable table;
+ private final TableSchema schema;
+ private final boolean tableCompatible;
+ private final Map<Long, Boolean> compatibleFileSchemas = new ConcurrentHashMap<>();
+
+ PaimonRustReaderCapabilities(FileStoreTable table, TupleDescriptor tuple) {
+ this.table = table;
+ this.schema = table.schema();
+ this.tableCompatible = schema != null && hasCompatibleSchemaVersion(schema)
+ && hasFullNestedProjection(tuple) && hasCompatibleMergeEngine(schema)
+ && hasCompatibleAggregates(schema);
+ }
+
+ boolean canRead(DataSplit split) {
+ if (!tableCompatible) {
+ return false;
+ }
+ // The pinned merge reader retains losing input batches until an output batch fills.
+ // Neither zero deletes nor a small read.batch-size bounds this across multiple files.
+ if (!schema.primaryKeys().isEmpty() && split.dataFiles().size() > 1) {
+ return false;
+ }
+ for (DataFileMeta file : split.dataFiles()) {
+ // Sort-merge retains consumed batches until it emits enough rows. Retracts can
+ // produce an unbounded zero-output prefix; aggregation also rejects retracts.
+ // Unknown counts must stay on JNI, including old files without this statistic.
+ if (!schema.primaryKeys().isEmpty() && file.deleteRowCount().orElse(-1L) != 0L) {
+ return false;
+ }
+ if (file.schemaId() != schema.id() && !compatibleFileSchemas.computeIfAbsent(
+ file.schemaId(), this::hasCompatibleFileSchema)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ private static boolean hasFullNestedProjection(TupleDescriptor tuple) {
+ for (SlotDescriptor slot : tuple.getSlots()) {
+ // The ABI projects only root names, while Arrow struct SerDes bind by ordinal.
+ // A pruned slot must use JNI's recursive read type, even inside arrays or maps.
+ if (slot.getType().isComplexType() && slot.getColumn() != null
+ && !slot.getType().equals(slot.getColumn().getType())) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ private boolean hasCompatibleFileSchema(long id) {
+ try {
+ TableSchema fileSchema = table.schemaManager().schema(id);
+ // Match historical types by ID: renames and added fields do not require value casts.
+ return fileSchema != null && hasCompatibleSchemaVersion(fileSchema)
+ && !hasIncompatibleEvolution(fileSchema.fields(), schema.fields());
+ } catch (RuntimeException e) {
+ // Failure to establish compatibility must not opt a historical file into Rust.
+ return false;
+ }
+ }
+
+ private static boolean hasCompatibleSchemaVersion(TableSchema schema) {
+ // Java defaults a missing JSON version to 1; Rust requires that field when opening
+ // both current and historical schemas. Metadata cannot distinguish an explicit 1.
+ return schema.version() > TableSchema.PAIMON_07_VERSION;
+ }
+
+ private static boolean hasCompatibleMergeEngine(TableSchema schema) {
+ CoreOptions.MergeEngine engine = new CoreOptions(schema.options()).mergeEngine();
+ // Older partial-update/aggregate writers retained repeated INSERT keys within one
+ // file. Zero deletes cannot bound Rust's retained losing batches for those producers.
+ return schema.primaryKeys().isEmpty()
+ || (engine != CoreOptions.MergeEngine.PARTIAL_UPDATE && engine != CoreOptions.MergeEngine.AGGREGATE);
+ }
+
+ private static boolean hasIncompatibleEvolution(List<DataField> oldFields, List<DataField> newFields) {
+ Map<Integer, DataType> oldTypes = new HashMap<>();
+ for (DataField field : oldFields) {
+ oldTypes.put(field.id(), field.type());
+ }
+ for (DataField field : newFields) {
+ DataType oldType = oldTypes.get(field.id());
+ if (oldType != null && hasIncompatibleEvolution(oldType, field.type())) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static boolean hasIncompatibleEvolution(DataType oldType, DataType newType) {
+ if (oldType.equalsIgnoreNullable(newType)) {
+ return false;
+ }
+ if (oldType instanceof RowType && newType instanceof RowType) {
+ return hasIncompatibleEvolution(((RowType) oldType).getFields(), ((RowType) newType).getFields());
+ }
+ if (oldType instanceof ArrayType && newType instanceof ArrayType) {
+ return hasIncompatibleEvolution(((ArrayType) oldType).getElementType(),
+ ((ArrayType) newType).getElementType());
+ }
+ if (oldType instanceof MapType && newType instanceof MapType) {
+ MapType oldMap = (MapType) oldType;
+ MapType newMap = (MapType) newType;
+ return hasIncompatibleEvolution(oldMap.getKeyType(), newMap.getKeyType())
+ || hasIncompatibleEvolution(oldMap.getValueType(), newMap.getValueType());
+ }
+ DataTypeRoot oldRoot = oldType.getTypeRoot();
+ DataTypeRoot newRoot = newType.getTypeRoot();
+ int oldWidth = integerWidth(oldRoot);
+ int newWidth = integerWidth(newRoot);
+ if (oldWidth > 0) {
+ return !((newWidth >= oldWidth) || newRoot == DataTypeRoot.FLOAT
+ || newRoot == DataTypeRoot.DOUBLE || newRoot == DataTypeRoot.DECIMAL);
+ }
+ if (oldRoot == DataTypeRoot.FLOAT && newRoot == DataTypeRoot.DOUBLE) {
+ return false;
+ }
+ if (oldRoot == DataTypeRoot.DECIMAL && newRoot == DataTypeRoot.DECIMAL) {
+ return false;
+ }
+ if ((oldRoot == DataTypeRoot.CHAR || oldRoot == DataTypeRoot.VARCHAR)
+ && newRoot == DataTypeRoot.VARCHAR) {
+ return DataTypeChecks.getLength(oldType) > DataTypeChecks.getLength(newType);
+ }
+ if (oldRoot == newRoot && (newRoot == DataTypeRoot.TIMESTAMP_WITHOUT_TIME_ZONE
+ || newRoot == DataTypeRoot.TIMESTAMP_WITH_LOCAL_TIME_ZONE)) {
+ int oldPrecision = DataTypeChecks.getPrecision(oldType);
+ int newPrecision = DataTypeChecks.getPrecision(newType);
+ // Arrow's nanosecond range is narrower than historical millis/micros; Java
+ // preserves out-of-range dates while Rust's checked cast turns them into NULL.
+ return oldPrecision > newPrecision || (oldPrecision <= 6 && newPrecision > 6);
+ }
+ // Only established cast equivalence may use Rust. Java's temporal casts use the
+ // default zone and truncate negative epochs; its string casts format/pad values.
+ // Arrow differs there, in numeric narrowing/decimal rounding, and in supported
+ // constructed casts. This allowlist applies at every nested leaf and map key.
+ return true;
+ }
+
+ private static int integerWidth(DataTypeRoot type) {
+ switch (type) {
+ case TINYINT:
+ return 8;
+ case SMALLINT:
+ return 16;
+ case INTEGER:
+ return 32;
+ case BIGINT:
+ return 64;
+ default:
+ return 0;
+ }
+ }
+
+ private static boolean hasCompatibleAggregates(TableSchema schema) {
+ Map<String, String> options = schema.options();
+ CoreOptions.MergeEngine engine = new CoreOptions(options).mergeEngine();
+ if (engine != CoreOptions.MergeEngine.AGGREGATE && engine != CoreOptions.MergeEngine.PARTIAL_UPDATE) {
+ return true;
+ }
+ // Rust validates every configured name before resolving overrides or excluding keys
+ // and sequence fields. Java may accept an unused SPI function that Rust cannot open.
+ for (Map.Entry<String, String> option : options.entrySet()) {
+ String key = option.getKey();
+ if (("fields.default-aggregate-function".equals(key)
+ || (key.startsWith("fields.") && key.endsWith(".aggregate-function")))
+ && !SUPPORTED_AGGREGATE_NAMES.contains(option.getValue())) {
+ return false;
+ }
+ }
+ Set<String> sequenceFields = new HashSet<>();
+ if (engine == CoreOptions.MergeEngine.PARTIAL_UPDATE) {
+ for (String key : options.keySet()) {
+ if (key.startsWith("fields.") && key.endsWith(".sequence-group")) {
+ sequenceFields.addAll(Arrays.asList(key.substring("fields.".length(),
+ key.length() - ".sequence-group".length()).split(",")));
+ }
+ }
+ }
+ // Validate values and types, not just option keys. Java's SPI includes functions
+ // such as collect which the pinned Rust aggregator factory does not implement.
+ for (DataField field : schema.fields()) {
+ if (schema.primaryKeys().contains(field.name()) || sequenceFields.contains(field.name())) {
+ continue;
+ }
+ String function = options.get("fields." + field.name() + ".aggregate-function");
+ // Partial-update applies the default too, except to keys and sequence fields.
+ // A field override must win before checking the effective function's type support.
+ if (function == null) {
+ function = options.get("fields.default-aggregate-function");
+ }
+ if (function != null && !supportsAggregate(function, field.type())) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ private static boolean supportsAggregate(String function, DataType type) {
+ DataTypeRoot root = type.getTypeRoot();
+ switch (function) {
+ case "sum":
+ // Java wraps integer arithmetic and retains compact-decimal intermediate
+ // sums beyond declared precision. Rust errors or resets the accumulator.
+ return root == DataTypeRoot.FLOAT || root == DataTypeRoot.DOUBLE
+ || (type instanceof DecimalType && ((DecimalType) type).getPrecision() > 18);
+ case "product":
+ return root == DataTypeRoot.FLOAT || root == DataTypeRoot.DOUBLE;
+ case "min":
+ case "max":
+ return integerWidth(root) != 0 || root == DataTypeRoot.FLOAT || root == DataTypeRoot.DOUBLE
+ || root == DataTypeRoot.DECIMAL || root == DataTypeRoot.DATE
+ || root == DataTypeRoot.TIME_WITHOUT_TIME_ZONE
+ || root == DataTypeRoot.TIMESTAMP_WITHOUT_TIME_ZONE
+ || root == DataTypeRoot.CHAR || root == DataTypeRoot.VARCHAR;
+ case "bool_and":
+ case "bool_or":
+ return root == DataTypeRoot.BOOLEAN;
+ case "listagg":
+ return type instanceof VarCharType && ((VarCharType) type).getLength() == Integer.MAX_VALUE;
+ case "last_value":
+ case "first_value":
+ case "last_non_null_value":
+ case "first_non_null_value":
+ case "first_not_null_value":
+ return true;
+ default:
+ return false;
+ }
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java
index 7b5e0e56a63..111bbc905c0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java
@@ -53,6 +53,7 @@ import org.apache.doris.planner.ScanContext;
import org.apache.doris.qe.SessionVariable;
import org.apache.doris.spi.Split;
import org.apache.doris.statistics.StatisticalType;
+import org.apache.doris.system.Backend;
import org.apache.doris.thrift.TExplainLevel;
import org.apache.doris.thrift.TFileFormatType;
import org.apache.doris.thrift.TFileRangeDesc;
@@ -63,16 +64,21 @@ import org.apache.doris.thrift.TTableFormatFileDesc;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
+import org.apache.commons.lang3.StringUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.paimon.CoreOptions;
+import org.apache.paimon.catalog.Identifier;
import org.apache.paimon.data.BinaryRow;
import org.apache.paimon.format.OrcOptions;
+import org.apache.paimon.io.DataFileMeta;
import org.apache.paimon.options.Options;
import org.apache.paimon.predicate.Predicate;
+import org.apache.paimon.rest.RESTTokenFileIO;
import org.apache.paimon.schema.TableSchema;
import org.apache.paimon.table.BucketMode;
import org.apache.paimon.table.DataTable;
+import org.apache.paimon.table.FallbackReadFileStoreTable;
import org.apache.paimon.table.FileStoreTable;
import org.apache.paimon.table.Table;
import org.apache.paimon.table.source.DataSplit;
@@ -85,7 +91,7 @@ import org.apache.paimon.table.source.TableScan;
import org.apache.paimon.table.source.snapshot.SnapshotReader;
import org.apache.paimon.types.ArrayType;
import org.apache.paimon.types.DataType;
-import org.apache.paimon.types.LocalZonedTimestampType;
+import org.apache.paimon.types.DataTypeRoot;
import org.apache.paimon.types.MapType;
import org.apache.paimon.types.MultisetType;
import org.apache.paimon.types.RowType;
@@ -95,6 +101,7 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
+import java.util.HashSet;
import java.util.List;
import java.util.Locale;
import java.util.Map;
@@ -102,6 +109,7 @@ import java.util.Objects;
import java.util.Optional;
import java.util.OptionalInt;
import java.util.OptionalLong;
+import java.util.Set;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
import java.util.stream.Collectors;
@@ -111,6 +119,7 @@ public class PaimonScanNode extends FileQueryScanNode {
private static final long COUNT_WITH_PARALLEL_SPLITS = 10000;
private static final long MAX_RETAINED_SERIALIZED_TASK_BYTES = 16L * 1024 * 1024;
+ private volatile PaimonRustReaderCapabilities rustReaderCapabilities;
private long maxRetainedSerializedTaskBytes = MAX_RETAINED_SERIALIZED_TASK_BYTES;
// The keys of incremental read params for Paimon SDK
private static final String PAIMON_INCREMENTAL_BETWEEN = "incremental-between";
@@ -131,11 +140,39 @@ public class PaimonScanNode extends FileQueryScanNode {
"doris.scan.manifest.parallelism-cap";
private static final String DORIS_SERIALIZED_SYSTEM_SOURCE = "doris.serialized-system-source";
private static final String DORIS_SYSTEM_TABLE_TYPE = "doris.system-table-type";
+ // Same key as the rust CoreOptions DELETION_VECTORS_MERGE_ON_READ_OPTION: paimon
+ // 1.4 exposes no Java accessor for it, so the raw TableSchema option is read.
+ private static final String DELETION_VECTORS_MERGE_ON_READ = "deletion-vectors.merge-on-read";
private static final List<String> BACKEND_PAIMON_OPTIONS = Arrays.asList(
DORIS_ENABLE_JNI_IO_MANAGER,
DORIS_JNI_IO_MANAGER_TMP_DIR,
DORIS_JNI_IO_MANAGER_IMPL_CLASS,
DORIS_ENABLE_FILE_READER_ASYNC);
+ // The table-location URI schemes whose property translation into the
+ // paimon-rust FileIO key families is implemented (BE bridge) and open
+ // tested: s3 / s3a -> the s3.* family, oss -> the fs.oss.* family, plus
+ // the credential-free hdfs and local-filesystem parsers (hadoop conf and
+ // local paths pass through untouched). See isRustVerifiedLocationScheme.
+ // An hdfs:// location additionally requires the credential-free backend
+ // shape of isRustVerifiedHdfsBackend: the same storage properties carry a
+ // Kerberized catalog's principal / keytab, its proxy user and its HA
+ // nameservice config, none of which the rust HDFS parser can honor.
+ private static final Set<String> RUST_VERIFIED_LOCATION_SCHEMES =
+ new HashSet<>(Arrays.asList("s3", "s3a", "oss", "hdfs", "file"));
+
+ // The dfs./hadoop./fs. keys a credential-free HDFS catalog may transport
+ // without changing what the pinned rust reader does: the warehouse path
+ // identity (fs.defaultFS, always shipped from the location), the
+ // authentication-type markers HdfsProperties always writes (validated to
+ // simple above), and the always-written fallback flag whose semantics only
+ // matter under Kerberos (already rejected above). Every other client
+ // setting is dropped by the rust parser and would diverge from JNI.
+ private static final Set<String> RUST_VERIFIED_HDFS_OPTION_KEYS =
+ new HashSet<>(Arrays.asList(
+ "fs.defaultFS",
+ "hadoop.security.authentication",
+ "hdfs.security.authentication",
+ "ipc.client.fallback-to-simple-auth-allowed"));
private enum SplitReadType {
JNI,
@@ -348,6 +385,283 @@ public class PaimonScanNode extends FileQueryScanNode {
}
}
+ /**
+ * Whether the table location's URI scheme is served by a paimon-rust FileIO parser whose
+ * property translation the FE/BE bridge implements (the s3.* / fs.oss.* key families for
+ * s3 / s3a / oss) or that needs no credentials at all (hdfs hadoop conf and local
+ * filesystem paths). Every other scheme the pinned crate dispatches to its own parser
+ * (cosn / obs / gs / abfs and friends) must fall back to the JNI reader because Doris
+ * delivers those credentials only as AWS_* aliases that those parsers do not read.
+ * A null location cannot be verified (and cannot ship paimon_table either), so it is
+ * not rust-eligible.
+ *
+ * <p>The scheme must also appear in the exact lowercase form the pinned crate consumes.
+ * URI schemes are case-insensitive, but the crate lowercases only its storage
+ * dispatch: its object-store path extraction strips a lowercase {@code s3://} prefix
+ * from the original string, and the hdfs / file helpers likewise match only lowercase
+ * prefixes. A {@code S3://} or {@code Hdfs://} warehouse also produces DataSplit file
+ * paths in that original casing (serialized by the paimon SDK before the FE sees
+ * them), so the mixed-case shape fails the rust open beyond the transported location.
+ * Those valid URI variants must therefore route to JNI, whose Java stack is
+ * case-insensitive everywhere.
+ */
+ @VisibleForTesting
+ static boolean isRustVerifiedLocationScheme(String location) {
+ if (location == null) {
+ return false;
+ }
+ int sep = location.indexOf("://");
+ if (sep <= 0) {
+ // No URI scheme: a plain local path reads through the crate's local-filesystem
+ // parser, which needs no credentials.
+ return true;
+ }
+ return RUST_VERIFIED_LOCATION_SCHEMES.contains(location.substring(0, sep));
+ }
+
+ // Whether the location is an hdfs:// table (the only HDFS-family scheme in
+ // RUST_VERIFIED_LOCATION_SCHEMES; viewfs / jfs never pass it).
+ private static boolean isHdfsLocationScheme(String location) {
+ if (location == null) {
+ return false;
+ }
+ int sep = location.indexOf("://");
+ return sep > 0 && "hdfs".equalsIgnoreCase(location.substring(0, sep));
+ }
+
+ // Whether any member file of the split is ORC. Paimon allows per-level
+ // file.format, so one DataSplit can mix Parquet and ORC files; the split
+ // path's suffix (the first file) cannot speak for the whole split, and the
+ // shifted ORC LTZ decode applies to whichever ORC members rust reads.
+ @VisibleForTesting
+ static boolean splitHasOrcFile(DataSplit dataSplit) {
+ if (dataSplit == null) {
+ return false;
+ }
+ for (DataFileMeta fileMeta : dataSplit.dataFiles()) {
+ String format = fileMeta.fileFormat();
+ if (format != null && "orc".equalsIgnoreCase(format)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ // Mirrors the pinned paimon-rust read-mode option matrix
+ // (PartialUpdateConfig::read_unsupported_option_keys and
+ // AggregationConfig's runtime-unsupported keys). Both validate option-key
+ // PRESENCE, not values, so a table carrying the key with an off value is
+ // still rejected by the rust merge construction while Java reads it — the
+ // gate must mirror presence exactly.
+ //
+ // Partial-update reads support basic mode, sequence groups and field
+ // aggregation; unsupported keys are the remove-record-on-delete family,
+ // per-field ignore-delete / ignore-retract / distinct / nested-key /
+ // count-limit options.
+ private static boolean isRustUnsupportedPartialUpdateReadOption(String key) {
+ return (key.endsWith(".ignore-delete")
+ && !"ignore-delete".equals(key)
+ && !"partial-update.ignore-delete".equals(key))
+ || "partial-update.remove-record-on-delete".equals(key)
+ || "partial-update.remove-record-on-sequence-group".equals(key)
+ || hasFieldOptionSuffix(key, ".ignore-retract")
+ || hasFieldOptionSuffix(key, ".distinct")
+ || hasFieldOptionSuffix(key, ".nested-key")
+ || hasFieldOptionSuffix(key, ".count-limit");
+ }
+
+ // Aggregation reads support the per-field aggregate-function /
+ // list-agg-delimiter / default-aggregate-function matrix; unsupported keys
+ // are the remove-record-on-delete family, every ignore-delete spelling
+ // (including the bare one), and per-field sequence-group / ignore-retract /
+ // distinct / nested-key / count-limit options.
+ private static boolean isRustUnsupportedAggregationRuntimeOption(String key) {
+ return "ignore-delete".equals(key)
+ || key.endsWith(".ignore-delete")
+ || "aggregation.remove-record-on-delete".equals(key)
+ || hasFieldOptionSuffix(key, ".sequence-group")
+ || hasFieldOptionSuffix(key, ".ignore-retract")
+ || hasFieldOptionSuffix(key, ".distinct")
+ || hasFieldOptionSuffix(key, ".nested-key")
+ || hasFieldOptionSuffix(key, ".count-limit");
+ }
+
+ private static boolean hasFieldOptionSuffix(String key, String suffix) {
+ return key.startsWith("fields.") && key.endsWith(suffix);
+ }
+
+ // Whether the schema options carry any option key the pinned paimon-rust
+ // read rejects for this merge engine.
+ @VisibleForTesting
+ static boolean hasRustUnsupportedMergeOption(Map<String, String> options,
+ CoreOptions.MergeEngine mergeEngine) {
+ boolean partialUpdate = mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE;
+ for (String key : options.keySet()) {
+ if (key == null) {
+ continue;
+ }
+ if (partialUpdate ? isRustUnsupportedPartialUpdateReadOption(key)
+ : isRustUnsupportedAggregationRuntimeOption(key)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ // Whether any member file of the split carries a data-file.external-paths
+ // location. Paimon can store an absolute location in each DataFileMeta, and
+ // both Java and the serialized rust split prefer it over the bucket path —
+ // but the pinned rust table builds ONE FileIO from paimon_table, whose
+ // storage enum parses every file with that warehouse-selected backend: an
+ // admitted hdfs table with an s3:// external file (or an s3 table with an
+ // oss:// file) reaches the wrong parser and fails the open, while JNI
+ // reads it. The shipped options describe only the warehouse, so any
+ // external file keeps the split on JNI.
+ @VisibleForTesting
+ static boolean splitHasExternalFiles(DataSplit dataSplit) {
+ if (dataSplit == null) {
+ return false;
+ }
+ for (DataFileMeta fileMeta : dataSplit.dataFiles()) {
+ if (fileMeta.externalPath().isPresent()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ // Native and Rust ORC routing must share this recursive check: nested LTZ values
+ // require the same JVM-zone conversion, including elements of a MULTISET.
+ @VisibleForTesting
+ static boolean containsTimestampLtz(DataType type) {
+ if (type == null) {
+ return false;
+ }
+ if (type.getTypeRoot() == DataTypeRoot.TIMESTAMP_WITH_LOCAL_TIME_ZONE) {
+ return true;
+ }
+ if (type instanceof ArrayType) {
+ return containsTimestampLtz(((ArrayType) type).getElementType());
+ }
+ if (type instanceof MapType) {
+ MapType mapType = (MapType) type;
+ return containsTimestampLtz(mapType.getKeyType())
+ || containsTimestampLtz(mapType.getValueType());
+ }
+ if (type instanceof MultisetType) {
+ return containsTimestampLtz(((MultisetType) type).getElementType());
+ }
+ if (type instanceof RowType) {
+ return ((RowType) type).getFields().stream()
+ .anyMatch(field -> containsTimestampLtz(field.type()));
+ }
+ return false;
+ }
+
+ /**
+ * Whether an HDFS catalog's shipped backend properties describe a shape the
+ * pinned paimon-rust HDFS reader can serve identically to JNI. The rust
+ * storage_hdfs parser reads only the hdfs.name-node / hdfs.enable-append
+ * keys — no kerberos, no proxy user, no HA resolution, and no Hadoop client
+ * option map (HdfsNativeConfig.options stays empty) — so a catalog whose
+ * settings need any of those would open as the BE process's ambient
+ * identity, resolve the wrong DataNode, or miss the catalog's configured
+ * access while the same query works through JNI. Only the open-tested
+ * credential-free shape stays rust-eligible: simple (or unset)
+ * authentication, no principal / keytab / proxy user, no HA nameservice
+ * resolution, and no client option beyond the always-shipped inert keys of
+ * {@link #RUST_VERIFIED_HDFS_OPTION_KEYS} — everything else in the
+ * dfs./hadoop./fs. namespaces is dropped by the rust parser and keeps the
+ * catalog on JNI (dfs.client.use.datanode.hostname=true is the canonical
+ * simple-auth example: hdfs-native defaults to false and connects to the
+ * DataNode's advertised IP instead of its hostname, which commonly fails
+ * behind containers/NAT).
+ */
+ @VisibleForTesting
+ static boolean isRustVerifiedHdfsBackend(Map<String, String> backendStorageProperties) {
+ if (backendStorageProperties == null) {
+ return true;
+ }
+ // Authentication type: "simple" (or unset) authenticates as the same
+ // ambient OS user on both readers; kerberos (or anything else) needs the
+ // channel the rust parser does not read.
+ for (String authKey : new String[] {"hadoop.security.authentication",
+ "hdfs.security.authentication"}) {
+ String value = backendStorageProperties.get(authKey);
+ if (value != null && !"simple".equalsIgnoreCase(value.trim())) {
+ return false;
+ }
+ }
+ // Kerberos identity and the proxy user: any of these configured means
+ // the open must carry a specific identity, which only JNI can honor.
+ for (String identityKey : new String[] {"hadoop.kerberos.principal",
+ "hadoop.kerberos.keytab", "hadoop.username"}) {
+ String value = backendStorageProperties.get(identityKey);
+ if (value != null && !value.trim().isEmpty()) {
+ return false;
+ }
+ }
+ // HA nameservice resolution: the rust parser receives no dfs.* config,
+ // so a nameservice-authority location (dfs.nameservices / dfs.ha.*)
+ // cannot be resolved; the proven shape is a single name-node URI.
+ String nameServices = backendStorageProperties.get("dfs.nameservices");
+ if (nameServices != null && !nameServices.trim().isEmpty()) {
+ return false;
+ }
+ for (Map.Entry<String, String> entry : backendStorageProperties.entrySet()) {
+ if (entry.getKey() != null && entry.getKey().startsWith("dfs.ha.")
+ && entry.getValue() != null && !entry.getValue().trim().isEmpty()) {
+ return false;
+ }
+ }
+ // Client options: the pinned rust storage_hdfs parser reads only the
+ // hdfs.name-node / hdfs.enable-append keys and leaves
+ // HdfsNativeConfig.options empty, so any other transported dfs./hadoop./fs.
+ // setting is silently dropped and hdfs-native runs with its own
+ // defaults. dfs.client.use.datanode.hostname=true is the canonical
+ // simple-auth example: valid without Kerberos, but hdfs-native
+ // defaults to false and connects to the DataNode's advertised IP
+ // instead of its hostname, which commonly fails behind containers/NAT
+ // while JNI succeeds. Only the keys HdfsProperties always writes for a
+ // credential-free catalog (or already validated above) are proven
+ // inert; anything else in these namespaces keeps the catalog on JNI.
+ for (Map.Entry<String, String> entry : backendStorageProperties.entrySet()) {
+ String key = entry.getKey();
+ // A blank value is the unset case (the producer null-filters; a
+ // blank site-config entry is inert), matching the identity and HA
+ // checks above.
+ if (key == null || entry.getValue() == null || entry.getValue().trim().isEmpty()) {
+ continue;
+ }
+ if ((key.startsWith("dfs.") || key.startsWith("hadoop.") || key.startsWith("fs."))
+ && !RUST_VERIFIED_HDFS_OPTION_KEYS.contains(key)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Mirrors the pinned paimon-rust {@code DataSplit::is_fully_materialized_pk_dv}: a
+ * primary-key split is safe to read raw under deletion vectors only when scan planning
+ * marked it raw convertible AND every data file is compacted (level != 0) and known to
+ * carry no retract rows (delete_row_count == Some(0)). The rust read_pk path requires
+ * this for partial-update / aggregation DV tables; anything weaker must stay on JNI.
+ */
+ @VisibleForTesting
+ static boolean isFullyMaterializedPkDvSplit(DataSplit dataSplit) {
+ if (!dataSplit.rawConvertible()) {
+ return false;
+ }
+ for (DataFileMeta fileMeta : dataSplit.dataFiles()) {
+ Optional<Long> deleteRowCount = fileMeta.deleteRowCount();
+ if (fileMeta.level() == 0 || !deleteRowCount.isPresent() || deleteRowCount.get() != 0L) {
+ return false;
+ }
+ }
+ return true;
+ }
+
private List<String> getOrderedPathPartitionKeys() {
if (source == null) {
return Collections.emptyList();
@@ -432,10 +746,348 @@ public class PaimonScanNode extends FileQueryScanNode {
String fileFormat = getFileFormat(paimonSplit.getPathString());
if (split != null) {
+ // use jni reader / paimon-cpp reader / paimon-rust reader
rangeDesc.setFormatType(TFileFormatType.FORMAT_JNI);
- // A logical DataSplit may span multiple files, so keep it intact for the JNI reader.
- fileDesc.setReaderType(TPaimonReaderType.PAIMON_JNI);
- fileDesc.setPaimonSplit(PaimonUtil.encodeObjectToString(split));
+ // paimon-cpp and paimon-rust both consume Paimon native binary serialization,
+ // which only supports DataSplit. Any other split type falls back to JNI.
+ boolean nativeSplit = split instanceof DataSplit;
+ // Fallback-read splits stay on JNI: FallbackDataSplit extends
+ // DataSplit, so the instanceof above passes, but its serializer
+ // appends an isFallback byte after the ordinary split that the
+ // pinned rust decoder rejects outright ("trailing bytes after
+ // DataSplit" — it requires full-buffer consumption), and even a
+ // permissive decode would still lack the second table identity
+ // needed to honor the fallback-side discriminator. Both sides of a
+ // FallbackReadFileStoreTable wrap their splits, so the table
+ // wrapper is gated as a whole (any split from it routes to JNI)
+ // until the rust ABI represents both sides; the FallbackSplit
+ // interface also catches a wrapper split regardless of how the
+ // table was resolved here.
+ boolean fallbackRead = split instanceof FallbackReadFileStoreTable.FallbackSplit
+ || processedTable instanceof FallbackReadFileStoreTable;
+ // Serialize the same effective table that planning and the JNI reader use.
+ // Relation options such as t@options('read.batch-size'='1') are applied by
+ // getProcessedTable() (doInitialize caches it in processedTable), and the
+ // rust reader derives its read batch size from the schema options — the raw
+ // cached table would silently drop the override. Copies, delegates and
+ // fallback wrappers of getProcessedTable() are still FileStoreTable, so the
+ // instanceof gate keeps its semantics.
+ Table paimonTable = processedTable;
+ FileStoreTable paimonFileStoreTable =
+ paimonTable instanceof FileStoreTable ? (FileStoreTable) paimonTable : null;
+ // query-auth.enabled tables stay on JNI: when catalog authorization
+ // succeeds with no row filter or column mask, Paimon still leaves an
+ // ordinary DataSplit (restricted results use QueryAuthSplit and are
+ // already handled by the nativeSplit gate above), so this table shape
+ // passes the compound gate — but the shipped schema keeps
+ // query-auth.enabled=true and the pinned rust ReadBuilder rejects
+ // every such table (its CoreOptions::ensure_read_authorized fails
+ // closed because the client cannot enforce the row filter / column
+ // masking), turning a valid authorized scan into a BE-open failure.
+ // Until the authorization result can be transported and enforced by
+ // the rust ABI, these tables route to JNI.
+ boolean queryAuthTable = false;
+ // REST-token tables stay on JNI: doInitialize snapshots
+ // RESTTokenFileIO.validToken().token() into the backend storage
+ // properties, discarding expireAtMillis and the REST refresh
+ // context, so the shipped credentials look static — but the
+ // pinned rust table reuses one option map with no refresh
+ // callback, while paimon 1.4.2's JNI RESTTokenFileIO checks
+ // expiry before each file operation and obtains a replacement
+ // token. A queued or long scan that crosses the token TTL would
+ // start on rust and later fail authentication. Gate until the
+ // rust ABI can refresh and atomically update credentials.
+ boolean restTokenTable = false;
+ // Partial-update / aggregation tables with deletion vectors only pass
+ // the rust reader in the fully materialized shape: the pinned rust
+ // read_pk rejects merge-engine=partial-update/aggregation with
+ // deletion-vectors.merge-on-read=true outright, and otherwise requires
+ // every split to be compacted and known free of retract rows
+ // (DataSplit::is_fully_materialized_pk_dv). Their ordinary DataSplits
+ // sail through the compound gate above, so without this check a valid
+ // Java/JNI scan reaches BE and the rust open fails. Deduplicate stays
+ // rust-eligible: its read_pk routes uncompacted splits to the KV
+ // reader, which applies the attached per-file DVs. merge-on-read=true
+ // is a table option, so the whole table routes to JNI;
+ // non-materialized splits are gated per split below.
+ boolean puAggDeletionVectors = false;
+ boolean dvMergeOnRead = false;
+ boolean deduplicateIgnoreDelete = false;
+ boolean rustUnsupportedMergeOption = false;
+ if (paimonFileStoreTable != null) {
+ // A renewable REST token reached the shipped properties as a
+ // plain value; only the table's FileIO type reveals it expires.
+ // Null-safe: a table handle whose FileIO is not resolved stays
+ // rust-eligible, mirroring the CoreOptions null-safety below.
+ restTokenTable = paimonFileStoreTable.fileIO() instanceof RESTTokenFileIO;
+ CoreOptions resolvedCoreOptions = paimonFileStoreTable.coreOptions();
+ // Null-safe: a table handle whose CoreOptions is not resolved
+ // (e.g. some wrapper shapes) stays rust-eligible rather than
+ // failing the scan here — the rust open itself rejects such a
+ // table if the option is really set.
+ if (resolvedCoreOptions != null) {
+ queryAuthTable = resolvedCoreOptions.queryAuthEnabled();
+ CoreOptions.MergeEngine mergeEngine = resolvedCoreOptions.mergeEngine();
+ if (resolvedCoreOptions.deletionVectorsEnabled()
+ && (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+ || mergeEngine == CoreOptions.MergeEngine.AGGREGATE)) {
+ puAggDeletionVectors = true;
+ // The merge-engine and deletion-vectors.enabled checks
+ // above resolve through the Java CoreOptions accessors,
+ // which the table builds from this same schema options
+ // map — the one the BE rust reader deserializes from
+ // the shipped schema JSON — so they cannot diverge from
+ // what BE sees. merge-on-read has no Java accessor in
+ // paimon 1.4, so it is read raw from the map, with the
+ // rust parsing semantics (any case-insensitive "true"
+ // is on, default false).
+ TableSchema dvSchema = paimonFileStoreTable.schema();
+ Map<String, String> dvOptions = dvSchema == null ? null : dvSchema.options();
+ String mergeOnRead = dvOptions == null
+ ? null : dvOptions.get(DELETION_VECTORS_MERGE_ON_READ);
+ dvMergeOnRead = "true".equalsIgnoreCase(mergeOnRead);
+ }
+ // deduplicate.ignore-delete=true tables stay on JNI:
+ // Java's DeduplicateMergeFunction skips retract records
+ // when the option is set — including old, uncompacted
+ // files that still contain them — but the pinned rust
+ // read_pk does not pass table options into its
+ // deduplicate merge: it picks the latest row and omits
+ // the key when that row is DELETE/UPDATE_BEFORE. An
+ // uncompacted insert followed by a delete therefore
+ // returns the insert through JNI but silently disappears
+ // through rust. Gate the option until the rust merge
+ // implements it.
+ if (mergeEngine == CoreOptions.MergeEngine.DEDUPLICATE
+ && resolvedCoreOptions.ignoreDelete()) {
+ deduplicateIgnoreDelete = true;
+ }
+ // Non-DV merge options the pinned rust read rejects: Java
+ // supports partial-update.remove-record-on-delete /
+ // aggregation.remove-record-on-delete and the wider
+ // per-field retract matrix, but the rust
+ // PartialUpdateConfig / AggregationConfig validations
+ // return Unsupported for them — and the DV-derived gates
+ // above only cover deletion-vector tables, so an ordinary
+ // non-DV DataSplit with one of these options would pass the
+ // compound gate and fail during the rust merge
+ // construction. Mirror the exact rust key matrix (presence,
+ // not values) against the same schema options map BE
+ // deserializes.
+ if (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+ || mergeEngine == CoreOptions.MergeEngine.AGGREGATE) {
+ TableSchema mergeSchema = paimonFileStoreTable.schema();
+ Map<String, String> mergeOptions =
+ mergeSchema == null ? null : mergeSchema.options();
+ rustUnsupportedMergeOption = mergeOptions != null
+ && hasRustUnsupportedMergeOption(mergeOptions, mergeEngine);
+ }
+ }
+ }
+ // paimon-rust additionally requires (a) FileScannerV2: the V1 FileScanner
+ // explicitly rejects PAIMON_RUST, so with enable_file_scanner_v2 disabled
+ // the split falls back to JNI instead of encoding a rust request that the
+ // selected scanner cannot consume, and (b) a FileStoreTable: BE opens the
+ // table via paimon_table_from_schema_json, which needs the resolved
+ // TableSchema that only FileStoreTable exposes via schema(). If the table
+ // is not a FileStoreTable (e.g. a sys table backed by DataSplit), we cannot
+ // ship a schema JSON, so fall back to CPP / JNI rather than sending an
+ // incomplete PAIMON_RUST request that BE would reject.
+ //
+ // The paimon-rust S3 bridge maps static credentials, anonymous
+ // access (AWS_CREDENTIALS_PROVIDER_TYPE=ANONYMOUS -> s3.anonymous)
+ // and assume-role (AWS_ROLE_ARN / AWS_EXTERNAL_ID ->
+ // s3.assumed.role.*), but the remaining credential-provider modes
+ // are ambient JVM provider chains (ENV, SYSTEM_PROPERTIES,
+ // WEB_IDENTITY, CONTAINER, INSTANCE_PROFILE) with no paimon-rust
+ // equivalent — rust would silently sign with whatever the ambient
+ // chain resolves to. Gate those modes away from the rust reader
+ // here so the configured provider is honored via the JNI path.
+ boolean providerModeTranslatable = true;
+ String providerType = backendStorageProperties == null
+ ? null : backendStorageProperties.get("AWS_CREDENTIALS_PROVIDER_TYPE");
+ String mode = providerType == null ? "DEFAULT" : providerType.trim().toUpperCase(Locale.ROOT);
+ if (mode.isEmpty()) {
+ mode = "DEFAULT";
+ }
+ String location = source.getTableLocation();
+ if (location != null && (location.startsWith("s3://") || location.startsWith("s3a://"))) {
+ // Java DEFAULT may resolve JVM properties or anonymous credentials;
+ // Rust's ambient chain is different. Only explicit keys or explicit
+ // anonymous access can cross this boundary without changing identity.
+ String accessKey = backendStorageProperties == null
+ ? null : backendStorageProperties.get("AWS_ACCESS_KEY");
+ String secretKey = backendStorageProperties == null
+ ? null : backendStorageProperties.get("AWS_SECRET_KEY");
+ boolean staticKeys = accessKey != null && !accessKey.trim().isEmpty()
+ && secretKey != null && !secretKey.trim().isEmpty();
+ // Hadoop's Simple provider gives static keys precedence over anonymous,
+ // role and token settings. Rust interprets those settings differently.
+ boolean conflictingStaticSettings = staticKeys && (mode.equals("ANONYMOUS")
+ || StringUtils.isNotEmpty(backendStorageProperties.get("AWS_ROLE_ARN"))
+ || StringUtils.isNotEmpty(backendStorageProperties.get("AWS_TOKEN")));
+ providerModeTranslatable = !conflictingStaticSettings
+ && (mode.equals("ANONYMOUS") || (mode.equals("DEFAULT") && staticKeys));
+ } else if (providerType != null) {
+ providerModeTranslatable = mode.equals("DEFAULT") || mode.equals("ANONYMOUS");
+ // OSS has no anonymous FileIO mode in the pinned Rust dependency.
+ if (mode.equals("ANONYMOUS") && location != null && location.startsWith("oss://")) {
+ providerModeTranslatable = false;
+ }
+ }
+ // Incremental scans (binlog / changelog / delta / diff) must stay
+ // on the JNI path: this wire format carries only an ordinary
+ // DataSplit and the rust reader invokes TableRead::to_arrow, but
+ // paimon 1.4 marks incremental splits as streaming (which the
+ // pinned rust deserializer rejects), diff requires a separate
+ // IncrementalPlan instead of an ordinary plan, and ordinary
+ // primary-key reads can merge versions rather than return the
+ // changes — until the C ABI transports the mode and plan, the
+ // rust reader cannot express any of these.
+ TableScanParams incrementalParams = getScanParams();
+ boolean isIncremental = incrementalParams != null && incrementalParams.incrementalRead();
+ // ORC TIMESTAMP_WITH_LOCAL_TIME_ZONE schemas stay on JNI: the pinned
+ // paimon-rust ORC decoder materializes LTZ instants shifted by the
+ // writer timezone (an upstream crate limitation), so a logical ORC
+ // DataSplit that selects rust (e.g. with force_jni_scanner=true or
+ // when raw conversion is unavailable) returns a different instant
+ // than JNI — applying the session timezone in BE cannot repair an
+ // epoch already shifted during decode. Two bypasses are covered:
+ // (a) the format must come from EVERY member file — paimon allows
+ // per-level file.format, so one DataSplit can mix Parquet and ORC
+ // files and the split path's suffix (the first file) would hide
+ // the ORC members; (b) the LTZ search must recurse into nested
+ // types — an LTZ under MAP/ARRAY/ROW reaches the same shifted ORC
+ // decode through the container's field materialization. Parquet
+ // files with any LTZ, and ORC without any recursive LTZ, stay
+ // rust-eligible. nativeSplit only guards the cast — non-DataSplit
+ // splits already route to JNI.
+ boolean orcLtzSchema = paimonFileStoreTable != null
+ && nativeSplit
+ && splitHasOrcFile((DataSplit) split)
+ && paimonFileStoreTable.schema().fields().stream()
+ .anyMatch(field -> containsTimestampLtz(field.type()));
+ // data-file.external-paths splits stay on JNI (see
+ // splitHasExternalFiles): the rust table's single FileIO cannot
+ // serve an external file's backend. nativeSplit only guards the
+ // cast — non-DataSplit splits already route to JNI.
+ boolean externalFileSplit = nativeSplit && splitHasExternalFiles((DataSplit) split);
+ // Projected VARIANT columns stay on JNI: the rust leaf feeds its
+ // Arrow arrays to the slot serdes, and DataTypeVariantV2SerDe::
+ // read_column_from_arrow unconditionally returns
+ // NOT_IMPLEMENTED_ERROR — a nested Variant (ARRAY / MAP / STRUCT
+ // containing one) reaches the same decoder through the container
+ // serdes. desc carries only the slots this query projects, so a
+ // table whose VARIANT column is not projected still scans on
+ // rust. Gate until the rust leaf has a Variant Arrow decoder.
+ boolean projectedVariant = desc.getSlots().stream()
+ .anyMatch(slot -> PaimonUtil.containsVariant(slot.getType()));
+ // Scheme capability gate: the pinned paimon-rust storage
+ // dispatcher (io/storage.rs) selects the FileIO parser from the
+ // table location's URI scheme, and libpaimon_c.a compiles in
+ // separate COS, OBS, GCS and Azdls parsers besides the OSS and S3
+ // ones. Doris normalizes every object store's credentials into
+ // the AWS_* / use_path_style aliases (see the *Properties storage
+ // classes), which the BE rust bridge translates only into the
+ // fs.oss.* and s3.* key families — a cosn:// / obs:// / gs:// /
+ // abfs:// warehouse would reach its scheme's parser without the
+ // key family it reads (fs.cosn.userinfo.*, fs.obs.*, gcs.*,
+ // azure.*) and fail the open instead of using JNI. Only the
+ // schemes whose property translation is implemented and
+ // open-tested (s3 / s3a / oss, via RUST_VERIFIED_LOCATION_SCHEMES)
+ // plus the credential-free hdfs and local-filesystem parsers stay
+ // rust-eligible; every other scheme falls back to JNI. A null
+ // location also routes to JNI: the rust path needs the
+ // paimon_table that only a real location can provide (BE rejects
+ // a split without it).
+ boolean schemeCapabilityVerified = isRustVerifiedLocationScheme(source.getTableLocation());
+ // An hdfs:// location is scheme-verified only together with the credential-free
+ // backend shape: the backend storage properties that ship to BE also carry an
+ // HDFS catalog's authentication (kerberos principal / keytab, proxy user, HA
+ // nameservice config), none of which the pinned rust HDFS parser reads — the
+ // scan would open as the BE process's ambient identity instead of the
+ // catalog's configured one and fail the access JNI honors. See
+ // isRustVerifiedHdfsBackend.
+ boolean hdfsBackendVerified = !isHdfsLocationScheme(source.getTableLocation())
+ || isRustVerifiedHdfsBackend(backendStorageProperties);
+ // With merge-on-read=true the whole table already routes to JNI (dvMergeOnRead);
+ // for the remaining partial-update/aggregation DV tables, a split that is
+ // not fully materialized (uncompacted level-0 data, or retractions not
+ // known to be applied — even a split with no deletion file attached yet)
+ // fails the rust is_fully_materialized_pk_dv guard, so it falls back per
+ // split instead of turning into a BE-open failure. nativeSplit and
+ // !fallbackRead only guard the cast — those splits already route to JNI.
+ boolean splitDvNotMaterialized = puAggDeletionVectors && !dvMergeOnRead
+ && nativeSplit && !fallbackRead
+ && !isFullyMaterializedPkDvSplit((DataSplit) split);
+ boolean canUseRust = sessionVariable.isEnablePaimonRustReader()
+ && sessionVariable.enableFileScannerV2 && nativeSplit && !fallbackRead
+ && !isIncremental && providerModeTranslatable && !queryAuthTable
+ && !restTokenTable
+ && !dvMergeOnRead && !splitDvNotMaterialized
+ && !orcLtzSchema && !projectedVariant && !externalFileSplit
+ && !deduplicateIgnoreDelete && !rustUnsupportedMergeOption
+ && schemeCapabilityVerified && hdfsBackendVerified
+ && paimonFileStoreTable != null;
+ if (canUseRust) {
+ // Branch tables can retain persisted modes without a selector. Validate the
+ // transported options too: the pinned Rust builder only accepts default here.
+ String scanMode = PaimonScanParams.withoutTimeTravelSelectors(paimonFileStoreTable.schema())
+ .options().get(CoreOptions.SCAN_MODE.key());
+ canUseRust = scanMode == null || "default".equalsIgnoreCase(scanMode);
+ }
+ if (canUseRust) {
+ // Every candidate can receive this range; older BEs cannot decode reader type 3.
+ canUseRust = !backendPolicy.getBackends().isEmpty()
+ && backendPolicy.getBackends().stream().allMatch(Backend::isPaimonRustReaderSupported);
+ }
+ if (canUseRust) {
+ if (rustReaderCapabilities == null) {
+ rustReaderCapabilities = new PaimonRustReaderCapabilities(paimonFileStoreTable, desc);
+ }
+ canUseRust = rustReaderCapabilities.canRead((DataSplit) split);
+ }
+ if (canUseRust) {
+ fileDesc.setReaderType(TPaimonReaderType.PAIMON_RUST);
+ fileDesc.setPaimonSplit(PaimonUtil.encodeDataSplitToString((DataSplit) split));
+ } else {
+ // A logical DataSplit may span multiple files, so keep it intact for the JNI reader.
+ fileDesc.setReaderType(TPaimonReaderType.PAIMON_JNI);
+ fileDesc.setPaimonSplit(PaimonUtil.encodeObjectToString(split));
+ }
+ // Set table location for paimon-cpp / paimon-rust reader
+ String tableLocation = source.getTableLocation();
+ if (tableLocation != null) {
+ fileDesc.setPaimonTable(tableLocation);
+ }
+ // paimon-rust reader opens tables via paimon_table_from_schema_json:
+ // ship db/table + resolved TableSchema JSON + non-default branch.
+ if (canUseRust) {
+ ExternalTable extTable = source.getExternalTable();
+ fileDesc.setDbName(extTable.getDbName());
+ fileDesc.setTableName(extTable.getName());
+
+ // No catalog / warehouse needed. FE ships the resolved TableSchema JSON
+ // and the branch (null-if-main; matches upstream paimon commit 742da63)
+ // so BE can skip a schema-file round trip. The FileStoreTable cast is
+ // safe here because canUseRust gates on it above. The fence's time-travel
+ // selector is stripped before transport: the rust reader pins data via
+ // the serialized DataSplit, and a shipped selector (scan.snapshot-id) is
+ // re-resolved by paimon-rust's copy_with_time_travel, which would swap
+ // these resolved fields for the pinned snapshot's older schema — a
+ // column added after the last data commit would then fail projection
+ // before per-file schema evolution could fill it (JNI keeps the resolved
+ // schema, so stripping restores rust/JNI parity).
+ TableSchema tableSchema =
+ PaimonScanParams.withoutTimeTravelSelectors(
+ ((FileStoreTable) paimonTable).schema());
+ fileDesc.setPaimonTableSchemaJson(PaimonUtil.encodeTableSchemaToJson(tableSchema));
+
+ String branch = CoreOptions.branch(tableSchema.options());
+ if (!Identifier.DEFAULT_MAIN_BRANCH.equals(branch)) {
+ fileDesc.setPaimonBranch(branch);
+ }
+ }
rangeDesc.setSelfSplitWeight(paimonSplit.getSelfSplitWeight());
} else {
// use native reader
@@ -584,6 +1236,9 @@ public class PaimonScanNode extends FileQueryScanNode {
pushDownCountSplits.add(split);
pushDownCountSum += count;
} else if (!forceJniScanner && !forceJniForSystemTable && supportNativeReader(optRawFiles)) {
+ // Keep raw-file reads ahead of logical JNI/Rust dispatch: only the native file
+ // reader can use footer aggregates for COUNT(col) and MIN/MAX. Logical DataSplits
+ // may require merge/delete semantics and cannot reuse independent file statistics.
if (ignoreSplitType == SessionVariable.IgnoreSplitType.IGNORE_NATIVE) {
continue;
}
@@ -1099,26 +1754,6 @@ public class PaimonScanNode extends FileQueryScanNode {
return true;
}
- private static boolean containsTimestampLtz(DataType type) {
- if (type instanceof LocalZonedTimestampType) {
- return true;
- }
- if (type instanceof ArrayType) {
- return containsTimestampLtz(((ArrayType) type).getElementType());
- }
- if (type instanceof MapType) {
- MapType map = (MapType) type;
- return containsTimestampLtz(map.getKeyType()) || containsTimestampLtz(map.getValueType());
- }
- if (type instanceof MultisetType) {
- return containsTimestampLtz(((MultisetType) type).getElementType());
- }
- if (type instanceof RowType) {
- return ((RowType) type).getFieldTypes().stream().anyMatch(PaimonScanNode::containsTimestampLtz);
- }
- return false;
- }
-
@Override
public TFileFormatType getFileFormatType() throws DdlException, MetaNotFoundException {
return TFileFormatType.FORMAT_JNI;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index bfb5b358b99..495aa82d96b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -821,6 +821,8 @@ public class SessionVariable implements Serializable, Writable {
public static final String FORCE_JNI_SCANNER = "force_jni_scanner";
+ public static final String ENABLE_PAIMON_RUST_READER = "enable_paimon_rust_reader";
+
public static final String ENABLE_COUNT_PUSH_DOWN_FOR_EXTERNAL_TABLE = "enable_count_push_down_for_external_table";
public static final String FETCH_ALL_FE_FOR_SYSTEM_TABLE = "fetch_all_fe_for_system_table";
@@ -3147,6 +3149,12 @@ public class SessionVariable implements Serializable, Writable {
description = {"强制使用 jni 方式读取外表", "Force the use of jni mode to read external table"})
private boolean forceJniScanner = false;
+ @VariableMgr.VarAttr(name = ENABLE_PAIMON_RUST_READER,
+ fuzzy = true,
+ description = {"Paimon 非原生文件读取使用 paimon-rust,优先级高于 paimon-cpp",
+ "Use paimon-rust for non-native Paimon reads, higher priority than paimon-cpp"})
+ private boolean enablePaimonRustReader = false;
+
@VariableMgr.VarAttr(name = ENABLE_COUNT_PUSH_DOWN_FOR_EXTERNAL_TABLE,
fuzzy = true,
description = {"对外表启用 count(*) 下推优化", "enable count(*) pushdown optimization for external table"})
@@ -4076,6 +4084,8 @@ public class SessionVariable implements Serializable, Writable {
// jni
this.forceJniScanner = random.nextBoolean();
+ this.enablePaimonRustReader = random.nextBoolean();
+
// statistics
this.fetchHiveRowCountSync = random.nextBoolean();
@@ -5875,6 +5885,7 @@ public class SessionVariable implements Serializable, Writable {
tResult.setEnableOrcFilterByMinMax(enableOrcFilterByMinMax);
tResult.setEnableExprZonemapFilter(enableExprZonemapFilter);
tResult.setFilePresignedUrlTtlSeconds(filePresignedUrlTtlSeconds);
+ tResult.setEnablePaimonRustReader(enablePaimonRustReader);
tResult.setCheckOrcInitSargsSuccess(checkOrcInitSargsSuccess);
tResult.setTruncateCharOrVarcharColumns(truncateCharOrVarcharColumns);
@@ -6704,6 +6715,10 @@ public class SessionVariable implements Serializable, Writable {
return forceJniScanner;
}
+ public boolean isEnablePaimonRustReader() {
+ return enablePaimonRustReader;
+ }
+
public String getIgnoreSplitType() {
return ignoreSplitType;
}
@@ -6727,6 +6742,10 @@ public class SessionVariable implements Serializable, Writable {
forceJniScanner = force;
}
+ public void setEnablePaimonRustReader(boolean enable) {
+ enablePaimonRustReader = enable;
+ }
+
public boolean isEnableCountPushDownForExternalTable() {
return enableCountPushDownForExternalTable;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/runtime/PipelineExecutionTaskBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/qe/runtime/PipelineExecutionTaskBuilder.java
index d9503e3145a..03f6f923dc2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/runtime/PipelineExecutionTaskBuilder.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/runtime/PipelineExecutionTaskBuilder.java
@@ -18,6 +18,7 @@
package org.apache.doris.qe.runtime;
import org.apache.doris.common.Pair;
+import org.apache.doris.common.profile.ExecutionProfile;
import org.apache.doris.common.profile.SummaryProfile;
import org.apache.doris.nereids.trees.plans.distribute.worker.BackendWorker;
import org.apache.doris.nereids.trees.plans.distribute.worker.DistributedPlanWorker;
@@ -86,15 +87,16 @@ public class PipelineExecutionTaskBuilder {
backend,
backendServiceProxy,
serializeFragments,
- buildSingleFragmentPipelineTask(backend, fragmentParamsList)
+ buildSingleFragmentPipelineTask(
+ coordinatorContext.executionProfile, backend, fragmentParamsList)
)
);
}
return fragmentTasks;
}
- private Map<Integer, SingleFragmentPipelineTask> buildSingleFragmentPipelineTask(
- Backend backend, TPipelineFragmentParamsList fragmentParamsList) {
+ static Map<Integer, SingleFragmentPipelineTask> buildSingleFragmentPipelineTask(
+ ExecutionProfile executionProfile, Backend backend, TPipelineFragmentParamsList fragmentParamsList) {
Map<Integer, SingleFragmentPipelineTask> tasks = Maps.newLinkedHashMap();
for (TPipelineFragmentParams fragmentParams : fragmentParamsList.getParamsList()) {
int fragmentId = fragmentParams.getFragmentId();
@@ -103,6 +105,7 @@ public class PipelineExecutionTaskBuilder {
.map(TPipelineInstanceParams::getFragmentInstanceId)
.collect(Collectors.toSet());
tasks.put(fragmentId, new SingleFragmentPipelineTask(backend, fragmentId, instanceIds));
+ executionProfile.addFragmentBackend(fragmentId, backend.getId());
}
return tasks;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/system/Backend.java b/fe/fe-core/src/main/java/org/apache/doris/system/Backend.java
index 8e475bc07dc..adb4a57d448 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/system/Backend.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/system/Backend.java
@@ -145,6 +145,8 @@ public class Backend implements Writable {
// The physical memory available for use by BE.
@SerializedName("beMemory")
private long beMemory = 0;
+ @SerializedName("supportsPaimonRustReader")
+ private volatile boolean supportsPaimonRustReader;
// from config::pipeline_executor_size , default equal cpuCores
@SerializedName("pipelineExecutorSize")
private int pipelineExecutorSize = 1;
@@ -257,6 +259,10 @@ public class Backend implements Writable {
return version;
}
+ public boolean isPaimonRustReaderSupported() {
+ return supportsPaimonRustReader;
+ }
+
public int getBePort() {
return bePort;
}
@@ -871,6 +877,11 @@ public class Backend implements Writable {
public boolean handleHbResponse(BackendHbResponse hbResponse, boolean isReplay) {
boolean isChanged = false;
if (hbResponse.getStatus() == HbStatus.OK) {
+ // An absent capability also clears support after a downgrade or legacy heartbeat replay.
+ if (supportsPaimonRustReader != hbResponse.isPaimonRustReaderSupported()) {
+ isChanged = true;
+ supportsPaimonRustReader = hbResponse.isPaimonRustReaderSupported();
+ }
if (!this.version.equals(hbResponse.getVersion())) {
isChanged = true;
this.version = hbResponse.getVersion();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/system/BackendHbResponse.java b/fe/fe-core/src/main/java/org/apache/doris/system/BackendHbResponse.java
index 41fad9961a4..55fb0c9f456 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/system/BackendHbResponse.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/system/BackendHbResponse.java
@@ -51,6 +51,8 @@ public class BackendHbResponse extends HeartbeatResponse implements Writable {
private boolean isShutDown = false;
// The physical memory available for use by BE.
private long beMemory = 0;
+ @SerializedName("supportsPaimonRustReader")
+ private boolean supportsPaimonRustReader;
public BackendHbResponse() {
super(HeartbeatResponse.Type.BACKEND);
@@ -152,6 +154,14 @@ public class BackendHbResponse extends HeartbeatResponse implements Writable {
return beMemory;
}
+ public boolean isPaimonRustReaderSupported() {
+ return supportsPaimonRustReader;
+ }
+
+ public void setPaimonRustReaderSupported(boolean supported) {
+ this.supportsPaimonRustReader = supported;
+ }
+
@Override
public String toString() {
StringBuilder sb = new StringBuilder();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/system/HeartbeatMgr.java b/fe/fe-core/src/main/java/org/apache/doris/system/HeartbeatMgr.java
index a8906e43df1..5e13e9191af 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/system/HeartbeatMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/system/HeartbeatMgr.java
@@ -369,9 +369,12 @@ public class HeartbeatMgr extends MasterDaemon {
isShutDown = tBackendInfo.isIsShutdown();
}
long beMemory = tBackendInfo.isSetBeMem() ? tBackendInfo.getBeMem() : 0;
- return new BackendHbResponse(backendId, bePort, httpPort, brpcPort,
+ BackendHbResponse response = new BackendHbResponse(backendId, bePort, httpPort, brpcPort,
System.currentTimeMillis(), beStartTime, version, nodeRole,
fragmentNum, lastFragmentUpdateTime, isShutDown, arrowFlightSqlPort, beMemory);
+ response.setPaimonRustReaderSupported(tBackendInfo.isSetSupportsPaimonRustReader()
+ && tBackendInfo.isSupportsPaimonRustReader());
+ return response;
} else {
return new BackendHbResponse(backendId, backend.getHost(), backend.getLastUpdateMs(),
result.getStatus().getErrorMsgs().isEmpty()
diff --git a/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileCompletionStateTest.java b/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileCompletionStateTest.java
new file mode 100644
index 00000000000..f263ff0b99f
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileCompletionStateTest.java
@@ -0,0 +1,217 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.profile;
+
+import org.apache.doris.common.util.DebugUtil;
+import org.apache.doris.planner.PlanFragmentId;
+import org.apache.doris.thrift.TDetailedReportParams;
+import org.apache.doris.thrift.TNetworkAddress;
+import org.apache.doris.thrift.TQueryProfile;
+import org.apache.doris.thrift.TRuntimeProfileNode;
+import org.apache.doris.thrift.TRuntimeProfileTree;
+import org.apache.doris.thrift.TUniqueId;
+
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+
+public class ProfileCompletionStateTest {
+ @Rule
+ public TemporaryFolder temporary = new TemporaryFolder();
+
+ private Profile profile() {
+ UUID id = UUID.randomUUID();
+ TUniqueId queryId = new TUniqueId(id.getMostSignificantBits(), id.getLeastSignificantBits());
+ Profile profile = new Profile();
+ profile.getSummaryProfile().getSummary().addInfoString(SummaryProfile.PROFILE_ID, DebugUtil.printId(queryId));
+ profile.addExecutionProfile(new ExecutionProfile(queryId, Collections.singletonList(0)));
+ return profile;
+ }
+
+ private void report(ExecutionProfile execution, String host, boolean done) {
+ TRuntimeProfileNode node = new TRuntimeProfileNode();
+ node.setName("PipelineProfile");
+ node.setNumChildren(0);
+ node.setCounters(new ArrayList<>());
+ node.setMetadata(0);
+ node.setIndent(false);
+ node.setInfoStrings(new HashMap<>());
+ node.setInfoStringsDisplayOrder(new ArrayList<>());
+ node.setChildCountersMap(new HashMap<>());
+ node.setTimestamp(0);
+ TDetailedReportParams params = new TDetailedReportParams();
+ params.setProfile(new TRuntimeProfileTree(Collections.singletonList(node)));
+ params.setIsFragmentLevel(false);
+ TQueryProfile report = new TQueryProfile();
+ report.setQueryId(execution.getQueryId());
+ // Multiple pipeline nodes from one BE must not count as multiple BE reports.
+ report.putToFragmentIdToProfile(0, Arrays.asList(params, params));
+ execution.updateProfile(report, new TNetworkAddress(host, 9050), done);
+ }
+
+ @Test
+ public void waitsForEveryBackendAndRejectsStaleReports() {
+ Profile profile = profile();
+ ExecutionProfile execution = profile.getExecutionProfiles().get(0);
+ execution.addFragmentBackend(new PlanFragmentId(0), 1L);
+ execution.addFragmentBackend(new PlanFragmentId(0), 2L);
+ Assert.assertEquals("RUNNING", profile.getProfileCompletionState());
+ profile.markQueryFinished();
+ Assert.assertEquals("COLLECTING", profile.getProfileCompletionState());
+ report(execution, "127.0.0.1", true);
+ Assert.assertEquals("COLLECTING", profile.getProfileCompletionState());
+ report(execution, "127.0.0.2", true);
+ Assert.assertEquals("COMPLETE", profile.getProfileCompletionState());
+ report(execution, "127.0.0.1", false);
+ Assert.assertEquals("COMPLETE", profile.getProfileCompletionState());
+ Assert.assertTrue(profile.getProfileByLevel().contains("Profile Completion State: COMPLETE"));
+ }
+
+ private static class PausingSummaryProfile extends SummaryProfile {
+ private final transient boolean beforeSerialization;
+ private final transient CountDownLatch storagePaused;
+ private final transient CountDownLatch resumeStorage;
+
+ private PausingSummaryProfile(boolean beforeSerialization, CountDownLatch storagePaused,
+ CountDownLatch resumeStorage) {
+ this.beforeSerialization = beforeSerialization;
+ this.storagePaused = storagePaused;
+ this.resumeStorage = resumeStorage;
+ }
+
+ @Override
+ public void write(DataOutput output) throws IOException {
+ if (!beforeSerialization) {
+ super.write(output);
+ }
+ storagePaused.countDown();
+ try {
+ if (!resumeStorage.await(10, TimeUnit.SECONDS)) {
+ throw new IOException("Timed out waiting for concurrent profile rendering");
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new IOException(e);
+ }
+ if (beforeSerialization) {
+ super.write(output);
+ }
+ }
+ }
+
+ private static class FailingSummaryProfile extends SummaryProfile {
+ private transient boolean fail = true;
+
+ @Override
+ public void write(DataOutput output) throws IOException {
+ if (fail) {
+ fail = false;
+ throw new IOException("Injected storage failure");
+ }
+ super.write(output);
+ }
+ }
+
+ @Test
+ public void failedStorageDoesNotFreezeCompletionState() throws Exception {
+ Profile profile = profile();
+ SummaryProfile summary = new FailingSummaryProfile();
+ summary.getSummary().addInfoString(SummaryProfile.PROFILE_ID, profile.getId());
+ profile.setSummaryProfile(summary);
+ ExecutionProfile execution = profile.getExecutionProfiles().get(0);
+ execution.addFragmentBackend(new PlanFragmentId(0), 1L);
+ profile.markQueryFinished();
+ String directory = temporary.newFolder().getAbsolutePath();
+ profile.writeToStorage(directory);
+ Assert.assertFalse(profile.profileHasBeenStored());
+ Assert.assertEquals("COLLECTING", profile.getProfileCompletionState());
+ report(execution, "127.0.0.1", true);
+ profile.writeToStorage(directory);
+ Assert.assertTrue(profile.profileHasBeenStored());
+ Assert.assertEquals("COMPLETE", Profile.read(profile.getProfileStoragePath()).getProfileCompletionState());
+ }
+
+ @Test
+ public void renderingDuringStoragePreservesTerminalState() throws Exception {
+ for (boolean beforeSerialization : new boolean[] {true, false}) {
+ Profile profile = profile();
+ CountDownLatch storagePaused = new CountDownLatch(1);
+ CountDownLatch resumeStorage = new CountDownLatch(1);
+ SummaryProfile summary = new PausingSummaryProfile(beforeSerialization, storagePaused, resumeStorage);
+ summary.getSummary().addInfoString(SummaryProfile.PROFILE_ID, profile.getId());
+ profile.setSummaryProfile(summary);
+ profile.getExecutionProfiles().get(0).addFragmentBackend(new PlanFragmentId(0), 1L);
+ profile.markQueryFinished();
+ String directory = temporary.newFolder().getAbsolutePath();
+ ExecutorService writer = Executors.newSingleThreadExecutor();
+ try {
+ Future<?> stored = writer.submit(() -> profile.writeToStorage(directory));
+ Assert.assertTrue(storagePaused.await(10, TimeUnit.SECONDS));
+ // Render both before serialization and before path publication, while storage is paused.
+ profile.getProfileByLevel();
+ resumeStorage.countDown();
+ stored.get(10, TimeUnit.SECONDS);
+ Assert.assertTrue(profile.profileHasBeenStored());
+ Assert.assertEquals("INCOMPLETE", profile.getProfileCompletionState());
+ Profile restored = Profile.read(profile.getProfileStoragePath());
+ Assert.assertNotNull(restored);
+ Assert.assertEquals("INCOMPLETE", restored.getProfileCompletionState());
+ } finally {
+ resumeStorage.countDown();
+ writer.shutdownNow();
+ Assert.assertTrue(writer.awaitTermination(10, TimeUnit.SECONDS));
+ }
+ }
+ }
+
+ @Test
+ public void persistsTerminalStateBeforeReleasingReports() throws Exception {
+ for (boolean complete : new boolean[] {false, true}) {
+ Profile profile = profile();
+ ExecutionProfile execution = profile.getExecutionProfiles().get(0);
+ execution.addFragmentBackend(new PlanFragmentId(0), 1L);
+ profile.markQueryFinished();
+ if (complete) {
+ report(execution, "127.0.0.1", true);
+ }
+ profile.writeToStorage(temporary.newFolder().getAbsolutePath());
+ Assert.assertTrue(profile.profileHasBeenStored());
+ profile.releaseMemory();
+ String expected = complete ? "COMPLETE" : "INCOMPLETE";
+ Assert.assertEquals(expected, profile.getProfileCompletionState());
+ Profile restored = Profile.read(profile.getProfileStoragePath());
+ Assert.assertNotNull(restored);
+ Assert.assertEquals(expected, restored.getProfileCompletionState());
+ Assert.assertTrue(restored.getProfileByLevel().contains("Profile Completion State: " + expected));
+ }
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileManagerTest.java b/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileManagerTest.java
index 65778b8c512..f6d37c6d3b4 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileManagerTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileManagerTest.java
@@ -430,6 +430,35 @@ class ProfileManagerTest {
Assertions.assertEquals(30 / 2, profiles.size());
}
+ @Test
+ void testFailedSpillRemainsIncompleteAfterMemoryRelease() throws Exception {
+ UUID id = UUID.randomUUID();
+ TUniqueId queryId = new TUniqueId(id.getMostSignificantBits(), id.getLeastSignificantBits());
+ Profile profile = constructProfile(DebugUtil.printId(queryId));
+ ExecutionProfile execution = new ExecutionProfile(queryId, Lists.newArrayList(0));
+ execution.addFragmentBackend(0, 1L);
+ profile.addExecutionProfile(execution);
+ profile.markQueryFinished();
+ profile.setQueryFinishTimestamp(0);
+ profileManager.pushProfile(profile);
+ profileManager.addExecutionProfile(execution);
+ File invalidDirectory = new File(tempDir, "not_a_directory");
+ Assertions.assertTrue(invalidDirectory.createNewFile());
+ ProfileManager.PROFILE_STORAGE_PATH = invalidDirectory.getAbsolutePath();
+
+ profileManager.writeProfileToStorage();
+
+ Assertions.assertFalse(profile.profileHasBeenStored());
+ Assertions.assertTrue(profile.getExecutionProfiles().isEmpty());
+ Assertions.assertEquals("INCOMPLETE", profile.getProfileCompletionState());
+ Assertions.assertTrue(profile.getProfileByLevel().contains("Profile Completion State: INCOMPLETE"));
+ // A later successful spill cannot recreate the reports discarded after the failed write.
+ ProfileManager.PROFILE_STORAGE_PATH = tempDir.getAbsolutePath();
+ profileManager.writeProfileToStorage();
+ Assertions.assertTrue(profile.profileHasBeenStored());
+ Assertions.assertEquals("INCOMPLETE", Profile.read(profile.getProfileStoragePath()).getProfileCompletionState());
+ }
+
@Test
void testWriteProfileToStorage() throws InterruptedException {
// Create some test profile files
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitorTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitorTest.java
new file mode 100644
index 00000000000..e54a1a2ff47
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitorTest.java
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon;
+
+import org.apache.doris.catalog.ArrayType;
+import org.apache.doris.catalog.MapType;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.Type;
+import org.apache.doris.datasource.DorisTypeVisitor;
+
+import org.apache.paimon.casting.CastExecutor;
+import org.apache.paimon.casting.CastExecutors;
+import org.apache.paimon.data.BinaryString;
+import org.apache.paimon.types.CharType;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.TimestampType;
+import org.apache.paimon.types.VarCharType;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class DorisToPaimonTypeVisitorTest {
+ private DataType convert(Type type) {
+ return DorisTypeVisitor.visit(type, new DorisToPaimonTypeVisitor());
+ }
+
+ @Test
+ public void testCharacterBounds() {
+ Assert.assertEquals(new CharType(3), convert(ScalarType.createCharType(3)));
+ Assert.assertEquals(new VarCharType(3), convert(ScalarType.createVarcharType(3)));
+ Assert.assertEquals(new VarCharType(VarCharType.MAX_LENGTH), convert(Type.STRING));
+ }
+
+ @Test
+ public void testTimestampPrecision() {
+ for (int precision : new int[] {0, 3, 6}) {
+ Assert.assertEquals(new TimestampType(precision),
+ convert(ScalarType.createDatetimeV2Type(precision)));
+ }
+ Assert.assertEquals(new TimestampType(0), convert(Type.DATETIME));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testMappedHistoricalCasts() {
+ CastExecutor<BinaryString, ?> varcharCast = (CastExecutor<BinaryString, ?>) CastExecutors.resolve(
+ new VarCharType(10), convert(ScalarType.createVarcharType(3)));
+ Assert.assertNotNull(varcharCast);
+ Assert.assertEquals("abc", varcharCast.cast(BinaryString.fromString("abcdef")).toString());
+ CastExecutor<BinaryString, ?> timestampCast = (CastExecutor<BinaryString, ?>) CastExecutors.resolve(
+ new VarCharType(VarCharType.MAX_LENGTH), convert(ScalarType.createDatetimeV2Type(3)));
+ Assert.assertNotNull(timestampCast);
+ Assert.assertEquals("1970-01-01T00:00:00.042", timestampCast.cast(BinaryString.fromString("42")).toString());
+ }
+
+ @Test
+ public void testNestedBoundsAndPrecision() {
+ ArrayType array = new ArrayType(ScalarType.createVarcharType(3));
+ Assert.assertEquals(new org.apache.paimon.types.ArrayType(new VarCharType(3)), convert(array));
+ MapType map = new MapType(ScalarType.createCharType(2), ScalarType.createDatetimeV2Type(3));
+ Assert.assertEquals(new org.apache.paimon.types.MapType(new CharType(false, 2), new TimestampType(3)),
+ convert(map));
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonRustBackendCapabilityTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonRustBackendCapabilityTest.java
new file mode 100644
index 00000000000..24f0383cd87
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonRustBackendCapabilityTest.java
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon.source;
+
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.analysis.TupleId;
+import org.apache.doris.datasource.ExternalScanNode;
+import org.apache.doris.datasource.FederationBackendPolicy;
+import org.apache.doris.datasource.paimon.PaimonExternalTable;
+import org.apache.doris.persist.gson.GsonUtils;
+import org.apache.doris.planner.PlanNodeId;
+import org.apache.doris.planner.ScanContext;
+import org.apache.doris.qe.SessionVariable;
+import org.apache.doris.system.Backend;
+import org.apache.doris.thrift.TFileRangeDesc;
+import org.apache.doris.thrift.TPaimonFileDesc;
+import org.apache.doris.thrift.TPaimonReaderType;
+
+import org.apache.paimon.data.BinaryRow;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.manifest.FileSource;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.stats.SimpleStats;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.types.DataField;
+import org.apache.paimon.types.IntType;
+import org.apache.paimon.utils.InstantiationUtil;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.List;
+
+public class PaimonRustBackendCapabilityTest {
+ @Test
+ public void testUnknownAndMixedBackendsUseJni() throws Exception {
+ Backend oldBackend = new Backend(1, "127.0.0.1", 9050);
+ Backend newBackend = capableBackend(2);
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI, reader(Collections.emptyList(), true));
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI, reader(Collections.singletonList(oldBackend), true));
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI, reader(Arrays.asList(newBackend, oldBackend), true));
+ Assert.assertEquals(TPaimonReaderType.PAIMON_RUST, reader(Collections.singletonList(newBackend), true));
+ Assert.assertEquals(TPaimonReaderType.PAIMON_RUST, reader(Arrays.asList(newBackend, capableBackend(3)), true));
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI, reader(Collections.singletonList(newBackend), false));
+ }
+
+ private Backend capableBackend(long id) {
+ // Exercise the same persisted capability used when a follower plans the scan.
+ return GsonUtils.GSON.fromJson("{\"id\":" + id + ",\"supportsPaimonRustReader\":true}", Backend.class);
+ }
+
+ private TPaimonReaderType reader(List<Backend> backends, boolean enabled) throws Exception {
+ SessionVariable vars = new SessionVariable();
+ vars.setEnablePaimonRustReader(enabled);
+ vars.enableFileScannerV2 = true;
+ PaimonScanNode node = new PaimonScanNode(new PlanNodeId(0), new TupleDescriptor(new TupleId(0)),
+ false, vars, ScanContext.EMPTY);
+ Field policyField = ExternalScanNode.class.getDeclaredField("backendPolicy");
+ policyField.setAccessible(true);
+ ((FederationBackendPolicy) policyField.get(node)).replaceBackendOrder(backends);
+
+ PaimonSource source = Mockito.mock(PaimonSource.class);
+ FileStoreTable table = Mockito.mock(FileStoreTable.class);
+ Mockito.when(table.schema()).thenReturn(new TableSchema(1,
+ Collections.singletonList(new DataField(0, "id", new IntType())), 0,
+ Collections.emptyList(), Collections.emptyList(), Collections.emptyMap(), null));
+ PaimonExternalTable external = Mockito.mock(PaimonExternalTable.class);
+ Mockito.when(source.getExternalTable()).thenReturn(external);
+ Mockito.when(external.getDbName()).thenReturn("db");
+ Mockito.when(external.getName()).thenReturn("t");
+ Mockito.when(source.getTableLocation()).thenReturn("file:///warehouse/db/t");
+ node.setSource(source);
+ Field tableField = PaimonScanNode.class.getDeclaredField("processedTable");
+ tableField.setAccessible(true);
+ tableField.set(node, table);
+
+ DataFileMeta file = DataFileMeta.forAppend("data.parquet", 1024, 1, SimpleStats.EMPTY_STATS,
+ 1, 1, 1, Collections.emptyList(), null, FileSource.APPEND,
+ Collections.emptyList(), null, null, Collections.emptyList());
+ DataSplit split = DataSplit.builder().rawConvertible(true).withPartition(BinaryRow.EMPTY_ROW)
+ .withBucket(0).withBucketPath("file:///warehouse/db/t/bucket-0")
+ .withDataFiles(Collections.singletonList(file)).build();
+ TFileRangeDesc range = new TFileRangeDesc();
+ Method setParams = PaimonScanNode.class.getDeclaredMethod("setPaimonParams",
+ TFileRangeDesc.class, PaimonSplit.class);
+ setParams.setAccessible(true);
+ setParams.invoke(node, range, new PaimonSplit(split));
+ TPaimonFileDesc fileDesc = range.getTableFormatParams().getPaimonParams();
+ if (fileDesc.getReaderType() == TPaimonReaderType.PAIMON_JNI) {
+ // Legacy BEs require the Java object encoding as well as the JNI reader enum.
+ DataSplit decoded = InstantiationUtil.deserializeObject(
+ Base64.getUrlDecoder().decode(fileDesc.getPaimonSplit()), getClass().getClassLoader());
+ Assert.assertEquals("data.parquet", decoded.dataFiles().get(0).fileName());
+ }
+ return fileDesc.getReaderType();
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonRustReaderCompatibilityTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonRustReaderCompatibilityTest.java
new file mode 100644
index 00000000000..4e44aa67809
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonRustReaderCompatibilityTest.java
@@ -0,0 +1,496 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon.source;
+
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.analysis.TupleId;
+
+import com.google.common.collect.ImmutableMap;
+import org.apache.paimon.data.BinaryString;
+import org.apache.paimon.data.Decimal;
+import org.apache.paimon.data.GenericArray;
+import org.apache.paimon.data.GenericRow;
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.data.Timestamp;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.fs.local.LocalFileIO;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.schema.Schema;
+import org.apache.paimon.schema.SchemaChange;
+import org.apache.paimon.schema.SchemaManager;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.FileStoreTableFactory;
+import org.apache.paimon.table.sink.BatchTableCommit;
+import org.apache.paimon.table.sink.BatchTableWrite;
+import org.apache.paimon.table.sink.BatchWriteBuilder;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.table.source.ReadBuilder;
+import org.apache.paimon.table.source.Split;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.DataTypes;
+import org.apache.paimon.types.RowKind;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.time.LocalDateTime;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TimeZone;
+import java.util.function.Function;
+
+/** Real persisted files verify the Java result contract and the split metadata used by the gate. */
+public class PaimonRustReaderCompatibilityTest {
+ @Rule
+ public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Test
+ public void testPersistedIntegerNarrowing() throws Exception {
+ FileStoreTable table = table(DataTypes.INT(), Collections.emptyMap());
+ commit(table, GenericRow.of(1, 383), GenericRow.of(2, -129), GenericRow.of(3, null));
+ assertRead(table, true, row -> row.isNullAt(1) ? null : row.getInt(1), Arrays.asList(383, -129, null));
+ table.schemaManager().commitChanges(Collections.singletonList(
+ SchemaChange.updateColumnType("v", DataTypes.TINYINT(), true)));
+ table = FileStoreTableFactory.create(table.fileIO(), table.location());
+ assertRead(table, false, row -> row.isNullAt(1) ? null : row.getByte(1),
+ Arrays.asList((byte) 127, (byte) 127, null));
+ }
+
+ @Test
+ public void testPersistedStringToTemporal() throws Exception {
+ for (boolean date : new boolean[] {true, false}) {
+ for (boolean nested : new boolean[] {false, true}) {
+ DataType sourceType = nested ? DataTypes.ROW(DataTypes.FIELD(2, "value", DataTypes.STRING()))
+ : DataTypes.STRING();
+ DataType target = date ? DataTypes.DATE() : DataTypes.TIMESTAMP(3);
+ FileStoreTable table = table(Schema.newBuilder().column("id", DataTypes.INT())
+ .column("v", sourceType), ImmutableMap.of("bucket", "-1"));
+ Object value = BinaryString.fromString("42");
+ commit(table, GenericRow.of(1, nested ? GenericRow.of(value) : value),
+ GenericRow.of(2, nested ? GenericRow.of((Object) null) : null));
+ assertRead(table, true, row -> row.getInt(0), Arrays.asList(1, 2));
+ table.schemaManager().commitChanges(Collections.singletonList(SchemaChange.updateColumnType(
+ nested ? new String[] {"v", "value"} : new String[] {"v"}, target, true)));
+ table = FileStoreTableFactory.create(table.fileIO(), table.location());
+ // Java accepts epoch-day/epoch-millisecond strings that Arrow's date parser rejects.
+ assertRead(table, false, row -> {
+ InternalRow valueRow = nested ? row.getRow(1, 1) : row;
+ int pos = nested ? 0 : 1;
+ if (valueRow.isNullAt(pos)) {
+ return null;
+ }
+ return date ? java.time.LocalDate.ofEpochDay(valueRow.getInt(pos)).toString()
+ : valueRow.getTimestamp(pos, 3).toString();
+ }, Arrays.asList(date ? "1970-02-12" : "1970-01-01T00:00:00.042", null));
+ }
+ }
+ }
+
+ @Test
+ public void testPersistedIntegerToTimestamp() throws Exception {
+ for (boolean bigint : new boolean[] {false, true}) {
+ for (int precision : new int[] {0, 3, 6}) {
+ for (boolean nested : new boolean[] {false, true}) {
+ DataType sourceType = bigint ? DataTypes.BIGINT() : DataTypes.INT();
+ FileStoreTable table = table(Schema.newBuilder().column("id", DataTypes.INT())
+ .column("v", nested ? DataTypes.ROW(DataTypes.FIELD(2, "value", sourceType)) : sourceType),
+ ImmutableMap.of("bucket", "-1"));
+ Object value = bigint ? (Object) 1700000000L : 1700000000;
+ commit(table, GenericRow.of(1, nested ? GenericRow.of(value) : value),
+ GenericRow.of(2, nested ? GenericRow.of((Object) null) : null));
+ assertRead(table, true, row -> row.getInt(0), Arrays.asList(1, 2));
+ table.schemaManager().commitChanges(Collections.singletonList(SchemaChange.updateColumnType(
+ nested ? new String[] {"v", "value"} : new String[] {"v"},
+ DataTypes.TIMESTAMP(precision), true)));
+ table = FileStoreTableFactory.create(table.fileIO(), table.location());
+ // Paimon numeric timestamps use seconds independently of the target precision.
+ assertRead(table, false, row -> {
+ InternalRow valueRow = nested ? row.getRow(1, 1) : row;
+ int pos = nested ? 0 : 1;
+ return valueRow.isNullAt(pos) ? null : valueRow.getTimestamp(pos, precision).toString();
+ }, Arrays.asList("2023-11-14T22:13:20", null));
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testPersistedTimestampToDateBeforeEpoch() throws Exception {
+ // Java divides toward zero; Arrow takes the preceding calendar date before the epoch.
+ assertPersistedEvolution(DataTypes.TIMESTAMP(3), DataTypes.DATE(), Timestamp.fromEpochMillis(-1),
+ false, row -> row.getInt(1), 0);
+ }
+
+ @Test
+ public void testPersistedTimestampZoneChanges() throws Exception {
+ TimeZone previous = TimeZone.getDefault();
+ try {
+ TimeZone.setDefault(TimeZone.getTimeZone("Asia/Shanghai"));
+ // Paimon interprets NTZ/LTZ changes in the JVM zone, unlike an Arrow timezone cast.
+ assertPersistedEvolution(DataTypes.TIMESTAMP(3), DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(3),
+ Timestamp.fromEpochMillis(0), false, row -> row.getTimestamp(1, 3).getMillisecond(),
+ -8L * 60 * 60 * 1000);
+ assertPersistedEvolution(DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(3), DataTypes.TIMESTAMP(3),
+ Timestamp.fromEpochMillis(0), false, row -> row.getTimestamp(1, 3).getMillisecond(),
+ 8L * 60 * 60 * 1000);
+ } finally {
+ TimeZone.setDefault(previous);
+ }
+ }
+
+ @Test
+ public void testPersistedTimeToTimestamp() throws Exception {
+ assertPersistedEvolution(DataTypes.TIME(3), DataTypes.TIMESTAMP(3), 1234, false,
+ row -> row.getTimestamp(1, 3).toString(), "1970-01-01T00:00:01.234");
+ }
+
+ @Test
+ public void testPersistedFormattedStrings() throws Exception {
+ assertPersistedEvolution(DataTypes.FLOAT(), DataTypes.STRING(), 0.0001f, false,
+ row -> row.getString(1).toString(), "1.0E-4");
+ assertPersistedEvolution(DataTypes.DOUBLE(), DataTypes.STRING(), 1.0e20, false,
+ row -> row.getString(1).toString(), "1.0E20");
+ assertPersistedEvolution(DataTypes.TIMESTAMP(3), DataTypes.STRING(), Timestamp.fromEpochMillis(1230),
+ false, row -> row.getString(1).toString(), "1970-01-01 00:00:01.230");
+ assertPersistedEvolution(DataTypes.TIME(3), DataTypes.STRING(), 1230, false,
+ row -> row.getString(1).toString(), "00:00:01.23");
+ }
+
+ @Test
+ public void testPersistedStringBounds() throws Exception {
+ assertPersistedEvolution(DataTypes.VARCHAR(10), DataTypes.VARCHAR(3), BinaryString.fromString("abcdef"),
+ false, row -> row.getString(1).toString(), "abc");
+ assertPersistedEvolution(DataTypes.CHAR(3), DataTypes.CHAR(10), BinaryString.fromString("x"),
+ false, row -> row.getString(1).toString(), "x ");
+ assertPersistedEvolution(DataTypes.VARCHAR(3), DataTypes.VARCHAR(10), BinaryString.fromString("abc"),
+ true, row -> row.getString(1).toString(), "abc");
+ }
+
+ @Test
+ public void testPersistedRowToString() throws Exception {
+ DataType rowType = DataTypes.ROW(DataTypes.FIELD(2, "a", DataTypes.INT()),
+ DataTypes.FIELD(3, "b", DataTypes.STRING()));
+ assertPersistedEvolution(rowType, DataTypes.STRING(), GenericRow.of(11, BinaryString.fromString("x")),
+ false, row -> row.getString(1).toString(), "{11, x}");
+ }
+
+ @Test
+ public void testPersistedTimestampEvolutionIntoNanosecondRange() throws Exception {
+ List<String> expected = Arrays.asList("2300-01-01T00:00:00.123", "1600-01-01T00:00:00.123",
+ "2024-01-01T00:00:00.123", null);
+ for (int oldPrecision : new int[] {3, 6}) {
+ for (boolean localZone : new boolean[] {false, true}) {
+ DataType source = localZone ? DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(oldPrecision)
+ : DataTypes.TIMESTAMP(oldPrecision);
+ FileStoreTable table = table(Schema.newBuilder().column("id", DataTypes.INT()).column("v", source),
+ ImmutableMap.of("bucket", "-1"));
+ commit(table, GenericRow.of(1, Timestamp.fromLocalDateTime(LocalDateTime.parse(expected.get(0)))),
+ GenericRow.of(2, Timestamp.fromLocalDateTime(LocalDateTime.parse(expected.get(1)))),
+ GenericRow.of(3, Timestamp.fromLocalDateTime(LocalDateTime.parse(expected.get(2)))),
+ GenericRow.of(4, null));
+ assertRead(table, true, row -> row.isNullAt(1) ? null
+ : row.getTimestamp(1, oldPrecision).toString(), expected);
+ DataType target = localZone ? DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(9) : DataTypes.TIMESTAMP(9);
+ table.schemaManager().commitChanges(Collections.singletonList(
+ SchemaChange.updateColumnType("v", target, true)));
+ table = FileStoreTableFactory.create(table.fileIO(), table.location());
+ // Java retains the full historical range; Arrow's nanosecond cast overflows these dates to NULL.
+ assertRead(table, false, row -> row.isNullAt(1) ? null : row.getTimestamp(1, 9).toString(), expected);
+ }
+ }
+ }
+
+ @Test
+ public void testPersistedSafeEvolutionControls() throws Exception {
+ assertPersistedEvolution(DataTypes.INT(), DataTypes.BIGINT(), 123, true, row -> row.getLong(1), 123L);
+ assertPersistedEvolution(DataTypes.INT(), DataTypes.DECIMAL(12, 2), 123, true,
+ row -> row.getDecimal(1, 12, 2).toBigDecimal().toPlainString(), "123.00");
+ assertPersistedEvolution(DataTypes.DECIMAL(10, 3), DataTypes.DECIMAL(10, 2),
+ Decimal.fromUnscaledLong(1235, 10, 3), true,
+ row -> row.getDecimal(1, 10, 2).toBigDecimal().toPlainString(), "1.24");
+ assertPersistedEvolution(DataTypes.TIMESTAMP(3), DataTypes.TIMESTAMP(6), Timestamp.fromEpochMillis(1234),
+ true, row -> row.getTimestamp(1, 6).toString(), "1970-01-01T00:00:01.234");
+ }
+
+ private <T> void assertPersistedEvolution(DataType sourceType, DataType targetType, Object value,
+ boolean rustExpected, Function<InternalRow, T> extract, T expected) throws Exception {
+ FileStoreTable table = table(Schema.newBuilder().column("id", DataTypes.INT()).column("v", sourceType),
+ ImmutableMap.of("bucket", "-1"));
+ commit(table, GenericRow.of(1, value), GenericRow.of(2, null));
+ assertRead(table, true, row -> row.getInt(0), Arrays.asList(1, 2));
+ table.schemaManager().commitChanges(Collections.singletonList(
+ SchemaChange.updateColumnType("v", targetType, true)));
+ table = FileStoreTableFactory.create(table.fileIO(), table.location());
+ assertRead(table, rustExpected, row -> row.isNullAt(1) ? null : extract.apply(row),
+ Arrays.asList(expected, null));
+ }
+
+ @Test
+ public void testPersistedFloatingToDecimalRounding() throws Exception {
+ FileStoreTable table = table(Schema.newBuilder().column("id", DataTypes.INT())
+ .column("v", DataTypes.DOUBLE()), ImmutableMap.of("bucket", "-1"));
+ commit(table, GenericRow.of(1, 1.005), GenericRow.of(2, -1.005), GenericRow.of(3, 1.25),
+ GenericRow.of(4, null));
+ assertRead(table, true, row -> row.getInt(0), Arrays.asList(1, 2, 3, 4));
+ table.schemaManager().commitChanges(Collections.singletonList(
+ SchemaChange.updateColumnType("v", DataTypes.DECIMAL(3, 2), true)));
+ table = FileStoreTableFactory.create(table.fileIO(), table.location());
+ assertRead(table, false, row -> row.isNullAt(1) ? null
+ : row.getDecimal(1, 3, 2).toBigDecimal().toPlainString(), Arrays.asList("1.01", "-1.01", "1.25", null));
+ }
+
+ @Test
+ public void testPersistedUnusedAggregateDefault() throws Exception {
+ for (String engine : Arrays.asList("aggregation", "partial-update")) {
+ for (String function : Arrays.asList("collect", "sum")) {
+ Map<String, String> options = new HashMap<>();
+ options.put("merge-engine", engine);
+ options.put("fields.default-aggregate-function", function);
+ options.put("fields.v.aggregate-function", "max");
+ Schema.Builder schema = Schema.newBuilder().column("id", DataTypes.INT().notNull())
+ .column("v", DataTypes.INT()).primaryKey("id");
+ boolean partial = "partial-update".equals(engine);
+ if (partial) {
+ schema.column("seq", DataTypes.INT());
+ options.put("fields.seq.sequence-group", "v");
+ }
+ FileStoreTable table = table(schema, options);
+ commit(table, partial ? GenericRow.of(1, 7, 1) : GenericRow.of(1, 7));
+ // An unused function override cannot establish that historical files contain unique keys.
+ assertRead(table, false, row -> row.getInt(1), Collections.singletonList(7));
+ }
+ }
+ }
+
+ @Test
+ public void testPersistedNonIntegerNarrowing() throws Exception {
+ for (DataType type : Arrays.asList(DataTypes.DOUBLE(), DataTypes.DECIMAL(10, 2))) {
+ boolean floating = type.equals(DataTypes.DOUBLE());
+ FileStoreTable table = table(type, Collections.emptyMap());
+ commit(table, GenericRow.of(1, floating ? 383.0 : Decimal.fromUnscaledLong(38300, 10, 2)),
+ GenericRow.of(2, floating ? -129.0 : Decimal.fromUnscaledLong(-12900, 10, 2)),
+ GenericRow.of(3, floating ? 12.0 : Decimal.fromUnscaledLong(1200, 10, 2)),
+ GenericRow.of(4, null));
+ assertRead(table, true, row -> row.getInt(0), Arrays.asList(1, 2, 3, 4));
+ table.schemaManager().commitChanges(Collections.singletonList(
+ SchemaChange.updateColumnType("v", DataTypes.TINYINT(), true)));
+ table = FileStoreTableFactory.create(table.fileIO(), table.location());
+ assertRead(table, false, row -> row.isNullAt(1) ? null : row.getByte(1),
+ Arrays.asList((byte) 127, (byte) 127, (byte) 12, null));
+ }
+ }
+
+ @Test
+ public void testPersistedPartialUpdateDefaultAggregate() throws Exception {
+ for (boolean decimal : new boolean[] {false, true}) {
+ DataType type = decimal ? DataTypes.DECIMAL(2, 0) : DataTypes.TINYINT();
+ for (boolean override : new boolean[] {false, true}) {
+ Map<String, String> options = new HashMap<>();
+ options.put("merge-engine", "partial-update");
+ options.put("fields.seq.sequence-group", "v");
+ options.put("fields.default-aggregate-function", "sum");
+ if (override) {
+ options.put("fields.v.aggregate-function", "max");
+ }
+ FileStoreTable table = table(Schema.newBuilder().column("id", DataTypes.INT().notNull())
+ .column("v", type).column("seq", DataTypes.INT()).primaryKey("id"), options);
+ commit(table, GenericRow.of(1, decimal ? Decimal.fromUnscaledLong(99, 2, 0) : (byte) 127, 1));
+ // Single-file partial updates must also fall back: old writers retained repeated keys.
+ assertRead(table, false, row -> decimal
+ ? row.getDecimal(1, 2, 0).toBigDecimal().toPlainString()
+ : Byte.toString(row.getByte(1)), Collections.singletonList(decimal ? "99" : "127"));
+ commit(table, GenericRow.of(1, decimal ? Decimal.fromUnscaledLong(1, 2, 0) : (byte) 1, 2));
+ if (decimal) {
+ commit(table, GenericRow.of(1, Decimal.fromUnscaledLong(-1, 2, 0), 3));
+ }
+ assertRead(table, false, row -> decimal
+ ? row.getDecimal(1, 2, 0).toBigDecimal().toPlainString()
+ : Byte.toString(row.getByte(1)), Collections.singletonList(
+ decimal ? "99" : override ? "127" : "-128"));
+ }
+ }
+ }
+
+ @Test
+ public void testPersistedInsertOnlyOverlappingRunsFallback() throws Exception {
+ FileStoreTable table = table(DataTypes.STRING(), ImmutableMap.of("read.batch-size", "1"));
+ for (int run = 0; run < 3; run++) {
+ List<GenericRow> rows = new ArrayList<>();
+ String payload = run + String.join("", Collections.nCopies(4096, "x"));
+ for (int key = 0; key < 1025; key++) {
+ rows.add(GenericRow.of(key, BinaryString.fromString(payload)));
+ }
+ commit(table, rows.toArray(new GenericRow[0]));
+ }
+ List<Split> splits = table.newReadBuilder().newScan().plan().splits();
+ Assert.assertEquals("Overlapping runs must be read in one logical split", 1, splits.size());
+ DataSplit split = (DataSplit) splits.get(0);
+ Assert.assertEquals(3, split.dataFiles().size());
+ split.dataFiles().forEach(file -> {
+ Assert.assertEquals(1025L, file.rowCount());
+ Assert.assertEquals(Long.valueOf(0), file.deleteRowCount().get());
+ });
+ assertRead(table, false, row -> {
+ String payload = row.getString(1).toString();
+ Assert.assertEquals(4097, payload.length());
+ return payload.substring(0, 1);
+ }, Collections.nCopies(1025, "2"));
+ }
+
+ @Test
+ public void testPersistedAppendOnlyMultiFileControl() throws Exception {
+ FileStoreTable table = table(Schema.newBuilder().column("id", DataTypes.INT()).column("v", DataTypes.INT()),
+ ImmutableMap.of("bucket", "-1"));
+ commit(table, GenericRow.of(1, 11));
+ commit(table, GenericRow.of(2, 22));
+ Assert.assertTrue(table.newReadBuilder().newScan().plan().splits().stream()
+ .anyMatch(split -> ((DataSplit) split).dataFiles().size() > 1));
+ assertRead(table, true, row -> row.getInt(1), Arrays.asList(11, 22));
+ }
+
+ @Test
+ public void testPersistedIntegerSumAndProductOverflow() throws Exception {
+ DataType[] types = {DataTypes.TINYINT(), DataTypes.SMALLINT(), DataTypes.INT(), DataTypes.BIGINT()};
+ Object[] maximums = {(byte) 127, (short) 32767, Integer.MAX_VALUE, Long.MAX_VALUE};
+ Object[] ones = {(byte) 1, (short) 1, 1, 1L};
+ Object[] halves = {(byte) 64, (short) 16384, 1073741824, 4611686018427387904L};
+ Object[] twos = {(byte) 2, (short) 2, 2, 2L};
+ for (String function : Arrays.asList("sum", "product")) {
+ for (int i = 0; i < types.length; i++) {
+ FileStoreTable table = table(types[i], aggregate(function));
+ commit(table, GenericRow.of(1, "sum".equals(function) ? maximums[i] : halves[i]));
+ commit(table, GenericRow.of(1, "sum".equals(function) ? ones[i] : twos[i]));
+ InternalRow.FieldGetter getter = InternalRow.createFieldGetter(types[i], 1);
+ Object minimum = i == 0 ? (byte) -128 : i == 1 ? (short) -32768
+ : i == 2 ? Integer.MIN_VALUE : Long.MIN_VALUE;
+ // Compare decimal strings to avoid Java's conditional numeric promotion.
+ assertRead(table, false, row -> getter.getFieldOrNull(row).toString(),
+ Collections.singletonList(minimum.toString()));
+ }
+ }
+ }
+
+ @Test
+ public void testPersistedCompactDecimalIntermediateOverflow() throws Exception {
+ FileStoreTable table = table(DataTypes.DECIMAL(2, 0), aggregate("sum"));
+ for (long value : new long[] {99, 1, -1}) {
+ commit(table, GenericRow.of(1, Decimal.fromUnscaledLong(value, 2, 0)));
+ }
+ assertRead(table, false, row -> row.getDecimal(1, 2, 0).toBigDecimal().toPlainString(),
+ Collections.singletonList("99"));
+ }
+
+ @Test
+ public void testPersistedCollect() throws Exception {
+ FileStoreTable table = table(DataTypes.ARRAY(DataTypes.INT()), aggregate("collect"));
+ commit(table, GenericRow.of(1, new GenericArray(new Integer[] {11})));
+ commit(table, GenericRow.of(1, new GenericArray(new Integer[] {22})));
+ assertRead(table, false, row -> {
+ List<Integer> values = Arrays.asList(row.getArray(1).getInt(0), row.getArray(1).getInt(1));
+ Collections.sort(values);
+ return values;
+ }, Collections.singletonList(Arrays.asList(11, 22)));
+ }
+
+ @Test
+ public void testPersistedAggregateInsertsAndRetractsFallback() throws Exception {
+ for (RowKind kind : Arrays.asList(RowKind.DELETE, RowKind.UPDATE_BEFORE)) {
+ FileStoreTable table = table(DataTypes.DOUBLE(), aggregate("sum"));
+ commit(table, GenericRow.of(1, 10.0));
+ assertRead(table, false, row -> row.getDouble(1), Collections.singletonList(10.0));
+ commit(table, GenericRow.of(1, 3.0));
+ assertRead(table, false, row -> row.getDouble(1), Collections.singletonList(13.0));
+ commit(table, GenericRow.ofKind(kind, 1, 3.0));
+ assertRead(table, false, row -> row.getDouble(1), Collections.singletonList(10.0));
+ }
+ }
+
+ @Test
+ public void testPersistedZeroAndSparseOutputMerge() throws Exception {
+ for (boolean survivor : new boolean[] {false, true}) {
+ FileStoreTable table = table(DataTypes.INT(), Collections.emptyMap());
+ List<GenericRow> inserts = new ArrayList<>();
+ List<GenericRow> deletes = new ArrayList<>();
+ for (int i = 0; i < 2048; i++) {
+ inserts.add(GenericRow.of(i, i));
+ deletes.add(GenericRow.ofKind(RowKind.DELETE, i, i));
+ }
+ if (survivor) {
+ inserts.add(GenericRow.of(2048, 2048));
+ }
+ commit(table, inserts.toArray(new GenericRow[0]));
+ commit(table, deletes.toArray(new GenericRow[0]));
+ assertRead(table, false, row -> row.getInt(1), survivor
+ ? Collections.singletonList(2048) : Collections.emptyList());
+ }
+ }
+
+ private static Map<String, String> aggregate(String function) {
+ return ImmutableMap.of("merge-engine", "aggregation", "fields.v.aggregate-function", function);
+ }
+
+ private FileStoreTable table(DataType type, Map<String, String> options) throws Exception {
+ return table(Schema.newBuilder().column("id", DataTypes.INT().notNull()).column("v", type)
+ .primaryKey("id"), options);
+ }
+
+ private FileStoreTable table(Schema.Builder schema, Map<String, String> options) throws Exception {
+ Path path = new Path(temporaryFolder.newFolder().toURI());
+ LocalFileIO fileIO = LocalFileIO.create();
+ schema.option("bucket", "1").option("write-only", "true")
+ .option("file.format", "parquet").option("deletion-vectors.enabled", "false")
+ .option("read.batch-size", "16").option("scan.manifest.parallelism", "1");
+ options.forEach(schema::option);
+ new SchemaManager(fileIO, path).createTable(schema.build());
+ return FileStoreTableFactory.create(fileIO, path);
+ }
+
+ private static void commit(FileStoreTable table, GenericRow... rows) throws Exception {
+ // Separate writers force persisted, uncompacted inputs instead of a write-time reduction.
+ BatchWriteBuilder builder = table.newBatchWriteBuilder();
+ try (BatchTableWrite write = builder.newWrite(); BatchTableCommit commit = builder.newCommit()) {
+ for (GenericRow row : rows) {
+ write.write(row);
+ }
+ commit.commit(write.prepareCommit());
+ }
+ }
+
+ private static <T> void assertRead(FileStoreTable table, boolean rustExpected,
+ Function<InternalRow, T> extract, List<T> expected) throws Exception {
+ ReadBuilder builder = table.newReadBuilder();
+ List<Split> splits = builder.newScan().plan().splits();
+ Assert.assertFalse("Persisted files must reach scan planning", splits.isEmpty());
+ PaimonRustReaderCapabilities capabilities = new PaimonRustReaderCapabilities(table,
+ new TupleDescriptor(new TupleId(0)));
+ List<T> actual = new ArrayList<>();
+ for (Split split : splits) {
+ Assert.assertEquals("reader eligibility for persisted input", rustExpected,
+ capabilities.canRead((DataSplit) split));
+ try (RecordReader<InternalRow> reader = builder.newRead().createReader(split)) {
+ reader.forEachRemaining(row -> actual.add(extract.apply(row)));
+ }
+ }
+ Assert.assertEquals(expected, actual);
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonRustReaderEvolutionTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonRustReaderEvolutionTest.java
new file mode 100644
index 00000000000..e3ee311e38c
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonRustReaderEvolutionTest.java
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon.source;
+
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.analysis.TupleId;
+
+import org.apache.paimon.data.BinaryRow;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.manifest.FileSource;
+import org.apache.paimon.schema.SchemaManager;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.stats.SimpleStats;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.types.DataField;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.DataTypes;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Optional;
+
+public class PaimonRustReaderEvolutionTest {
+ @Test
+ public void testPrimaryKeyMergeEnginesWithOneInsertOnlyFile() {
+ for (String engine : Arrays.asList("deduplicate", "first-row", "partial-update", "aggregation")) {
+ FileStoreTable table = Mockito.mock(FileStoreTable.class);
+ TableSchema schema = new TableSchema(2, Arrays.asList(new DataField(0, "id", DataTypes.INT()),
+ new DataField(1, "v", DataTypes.DOUBLE())), 1, Collections.emptyList(),
+ Collections.singletonList("id"), Collections.singletonMap("merge-engine", engine), null);
+ Mockito.when(table.schema()).thenReturn(schema);
+ DataFileMeta file = Mockito.mock(DataFileMeta.class);
+ Mockito.when(file.schemaId()).thenReturn(2L);
+ Mockito.when(file.deleteRowCount()).thenReturn(Optional.of(0L));
+ DataSplit split = DataSplit.builder().withPartition(BinaryRow.EMPTY_ROW).withBucket(0)
+ .withBucketPath("file:///warehouse/table/bucket-0")
+ .withDataFiles(Collections.singletonList(file)).build();
+ // Older writers can keep repeated inserts inside one aggregate/partial-update file.
+ Assert.assertEquals(engine, "deduplicate".equals(engine) || "first-row".equals(engine),
+ new PaimonRustReaderCapabilities(table, new TupleDescriptor(new TupleId(0))).canRead(split));
+ }
+ }
+
+ @Test
+ public void testTimestampEvolutionIntoNanosecondRangeFallback() {
+ for (int oldPrecision : new int[] {0, 3, 6}) {
+ for (int newPrecision : new int[] {7, 8, 9}) {
+ assertNestedEvolution(DataTypes.TIMESTAMP(oldPrecision), DataTypes.TIMESTAMP(newPrecision), false);
+ assertNestedEvolution(DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(oldPrecision),
+ DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(newPrecision), false);
+ }
+ }
+ assertNestedEvolution(DataTypes.TIMESTAMP(7), DataTypes.TIMESTAMP(9), true);
+ }
+
+ @Test
+ public void testTimestampToDateFallback() {
+ assertNestedEvolution(DataTypes.TIMESTAMP(3), DataTypes.DATE(), false);
+ assertNestedEvolution(DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(3), DataTypes.DATE(), false);
+ }
+
+ @Test
+ public void testTimestampZoneChangeFallback() {
+ assertNestedEvolution(DataTypes.TIMESTAMP(3), DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(3), false);
+ assertNestedEvolution(DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(3), DataTypes.TIMESTAMP(3), false);
+ }
+
+ @Test
+ public void testTimeToTimestampFallback() {
+ assertNestedEvolution(DataTypes.TIME(3), DataTypes.TIMESTAMP(3), false);
+ assertNestedEvolution(DataTypes.TIME(3), DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(3), false);
+ }
+
+ @Test
+ public void testFormattedStringFallback() {
+ for (DataType source : Arrays.asList(DataTypes.FLOAT(), DataTypes.DOUBLE(), DataTypes.TIME(3),
+ DataTypes.TIMESTAMP(3), DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(3))) {
+ for (DataType target : Arrays.asList(DataTypes.STRING(), DataTypes.VARCHAR(30), DataTypes.CHAR(30))) {
+ assertNestedEvolution(source, target, false);
+ }
+ }
+ }
+
+ @Test
+ public void testConstructedStringFallback() {
+ for (DataType source : Arrays.asList(DataTypes.ROW(DataTypes.FIELD(2, "a", DataTypes.INT())),
+ DataTypes.ARRAY(DataTypes.INT()), DataTypes.MAP(DataTypes.INT(), DataTypes.STRING()))) {
+ assertNestedEvolution(source, DataTypes.STRING(), false);
+ }
+ }
+
+ @Test
+ public void testUnprovenScalarCastsFallback() {
+ assertNestedEvolution(DataTypes.STRING(), DataTypes.TIME(3), false);
+ assertNestedEvolution(DataTypes.DATE(), DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(3), false);
+ assertNestedEvolution(DataTypes.TIMESTAMP(6), DataTypes.TIMESTAMP(3), false);
+ assertNestedEvolution(DataTypes.VARCHAR(10), DataTypes.VARCHAR(3), false);
+ assertNestedEvolution(DataTypes.CHAR(3), DataTypes.CHAR(10), false);
+ }
+
+ @Test
+ public void testSafeEvolutionControls() {
+ DataType[][] pairs = {
+ {DataTypes.TINYINT(), DataTypes.SMALLINT()},
+ {DataTypes.INT(), DataTypes.BIGINT()},
+ {DataTypes.INT(), DataTypes.DOUBLE()},
+ {DataTypes.INT(), DataTypes.DECIMAL(12, 2)},
+ {DataTypes.FLOAT(), DataTypes.DOUBLE()},
+ {DataTypes.DECIMAL(10, 3), DataTypes.DECIMAL(10, 2)},
+ {DataTypes.VARCHAR(10), DataTypes.VARCHAR(30)},
+ {DataTypes.CHAR(3), DataTypes.VARCHAR(10)},
+ {DataTypes.TIMESTAMP(3), DataTypes.TIMESTAMP(6)},
+ {DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(3), DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(6)},
+ {DataTypes.DATE(), DataTypes.DATE()}
+ };
+ for (DataType[] pair : pairs) {
+ assertNestedEvolution(pair[0], pair[1], true);
+ }
+ }
+
+ @Test
+ public void testNestedFieldsMatchById() {
+ DataType oldType = DataTypes.ROW(DataTypes.FIELD(2, "removed", DataTypes.TIMESTAMP(3)),
+ DataTypes.FIELD(3, "old_name", DataTypes.INT()));
+ DataType newType = DataTypes.ROW(DataTypes.FIELD(3, "renamed", DataTypes.BIGINT()),
+ DataTypes.FIELD(4, "added", DataTypes.STRING()));
+ assertNestedEvolution(oldType, newType, true);
+ }
+
+ private void assertNestedEvolution(DataType source, DataType target, boolean expected) {
+ DataType[][] shapes = {
+ {source, target},
+ {DataTypes.ROW(new DataField(10, "old_name", source)),
+ DataTypes.ROW(new DataField(10, "renamed", target))},
+ {DataTypes.ARRAY(source), DataTypes.ARRAY(target)},
+ {DataTypes.MAP(source, DataTypes.INT()), DataTypes.MAP(target, DataTypes.INT())},
+ {DataTypes.MAP(DataTypes.INT(), source), DataTypes.MAP(DataTypes.INT(), target)}
+ };
+ for (DataType[] shape : shapes) {
+ Assert.assertTrue("Unchanged history must remain eligible: " + shape[0], canRead(shape[0], shape[0]));
+ Assert.assertEquals(Arrays.toString(shape), expected, canRead(shape[0], shape[1]));
+ }
+ }
+
+ private boolean canRead(DataType source, DataType target) {
+ FileStoreTable table = Mockito.mock(FileStoreTable.class);
+ SchemaManager schemas = Mockito.mock(SchemaManager.class);
+ Mockito.when(table.schema()).thenReturn(schema(2, target));
+ Mockito.when(table.schemaManager()).thenReturn(schemas);
+ Mockito.when(schemas.schema(1)).thenReturn(schema(1, source));
+ DataFileMeta file = DataFileMeta.forAppend("historical.parquet", 100L, 1L, SimpleStats.EMPTY_STATS,
+ 0L, 0L, 1L, Collections.emptyList(), null, FileSource.APPEND,
+ Collections.emptyList(), null, null, Collections.emptyList());
+ DataSplit split = DataSplit.builder().withPartition(BinaryRow.EMPTY_ROW).withBucket(0)
+ .withBucketPath("file:///warehouse/table/bucket-0")
+ .withDataFiles(Collections.singletonList(file)).build();
+ return new PaimonRustReaderCapabilities(table, new TupleDescriptor(new TupleId(0))).canRead(split);
+ }
+
+ private TableSchema schema(long id, DataType valueType) {
+ return new TableSchema(id, Arrays.asList(new DataField(0, "id", DataTypes.INT()),
+ new DataField(1, "v", valueType)), 10, Collections.emptyList(), Collections.emptyList(),
+ Collections.emptyMap(), null);
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonRustReaderLegacySchemaTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonRustReaderLegacySchemaTest.java
new file mode 100644
index 00000000000..1ed35e94007
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonRustReaderLegacySchemaTest.java
@@ -0,0 +1,101 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon.source;
+
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.analysis.TupleId;
+
+import org.apache.paimon.data.GenericRow;
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.fs.local.LocalFileIO;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.schema.Schema;
+import org.apache.paimon.schema.SchemaChange;
+import org.apache.paimon.schema.SchemaManager;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.FileStoreTableFactory;
+import org.apache.paimon.table.sink.BatchTableCommit;
+import org.apache.paimon.table.sink.BatchTableWrite;
+import org.apache.paimon.table.sink.BatchWriteBuilder;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.table.source.Split;
+import org.apache.paimon.types.DataTypes;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+public class PaimonRustReaderLegacySchemaTest {
+ @Rule
+ public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Test
+ public void testPersistedLegacySchemaFallsBackBeforeAndAfterEvolution() throws Exception {
+ java.nio.file.Path directory = temporaryFolder.newFolder().toPath();
+ Path location = new Path(directory.toUri());
+ LocalFileIO fileIO = LocalFileIO.create();
+ new SchemaManager(fileIO, location).createTable(Schema.newBuilder()
+ .column("id", DataTypes.INT()).option("bucket", "-1")
+ .option("file.format", "parquet").build());
+ FileStoreTable table = FileStoreTableFactory.create(fileIO, location);
+ BatchWriteBuilder builder = table.newBatchWriteBuilder();
+ try (BatchTableWrite write = builder.newWrite(); BatchTableCommit commit = builder.newCommit()) {
+ write.write(GenericRow.of(7));
+ commit.commit(write.prepareCommit());
+ }
+ assertReadable(table, true);
+ // Older producers omitted version. Java accepts that schema, but Rust requires it
+ // even when reopening a historical schema for otherwise compatible field evolution.
+ java.nio.file.Path schemaFile = directory.resolve("schema/schema-0");
+ // Fixture I/O must also work with the FE test compilation target of Java 8.
+ String json = new String(Files.readAllBytes(schemaFile), StandardCharsets.UTF_8);
+ String legacy = json.replaceFirst("\"version\"\\s*:\\s*\\d+\\s*,", "");
+ Assert.assertNotEquals(json, legacy);
+ Files.write(schemaFile, legacy.getBytes(StandardCharsets.UTF_8));
+ table = FileStoreTableFactory.create(fileIO, location);
+ Assert.assertEquals(1, table.schema().version());
+ assertReadable(table, false);
+ table.schemaManager().commitChanges(Collections.singletonList(
+ SchemaChange.addColumn("extra", DataTypes.STRING())));
+ table = FileStoreTableFactory.create(fileIO, location);
+ Assert.assertTrue(table.schema().version() > 0);
+ assertReadable(table, false);
+ }
+
+ private static void assertReadable(FileStoreTable table, boolean rustExpected) throws Exception {
+ List<Split> splits = table.newScan().plan().splits();
+ Assert.assertFalse(splits.isEmpty());
+ PaimonRustReaderCapabilities capabilities = new PaimonRustReaderCapabilities(table,
+ new TupleDescriptor(new TupleId(0)));
+ List<Integer> actual = new ArrayList<>();
+ for (Split split : splits) {
+ Assert.assertEquals(rustExpected, capabilities.canRead((DataSplit) split));
+ try (RecordReader<InternalRow> reader = table.newRead().createReader(split)) {
+ reader.forEachRemaining(row -> actual.add(row.getInt(0)));
+ }
+ }
+ Assert.assertEquals(Collections.singletonList(7), actual);
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonScanNodeTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonScanNodeTest.java
index d744973a566..2508f3c7bbe 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonScanNodeTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonScanNodeTest.java
@@ -35,6 +35,7 @@ import org.apache.doris.common.ExceptionChecker;
import org.apache.doris.common.UserException;
import org.apache.doris.datasource.CatalogIf;
import org.apache.doris.datasource.CatalogProperty;
+import org.apache.doris.datasource.FederationBackendPolicy;
import org.apache.doris.datasource.FileQueryScanNode;
import org.apache.doris.datasource.FileSplitter;
import org.apache.doris.datasource.hive.HMSExternalTable;
@@ -54,10 +55,12 @@ import org.apache.doris.datasource.paimon.PaimonUtils;
import org.apache.doris.datasource.property.metastore.MetastoreProperties;
import org.apache.doris.datasource.property.metastore.PaimonJdbcMetaStoreProperties;
import org.apache.doris.nereids.StatementContext;
+import org.apache.doris.persist.gson.GsonUtils;
import org.apache.doris.planner.PlanNodeId;
import org.apache.doris.planner.ScanContext;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.SessionVariable;
+import org.apache.doris.system.Backend;
import org.apache.doris.thrift.TFileFormatType;
import org.apache.doris.thrift.TFileRangeDesc;
import org.apache.doris.thrift.TFileScanRangeParams;
@@ -65,21 +68,26 @@ import org.apache.doris.thrift.TPaimonReaderType;
import org.apache.doris.thrift.TPushAggOp;
import com.google.common.collect.ImmutableMap;
+import mockit.MockUp;
import org.apache.paimon.CoreOptions;
import org.apache.paimon.Snapshot;
import org.apache.paimon.data.BinaryRow;
import org.apache.paimon.fs.FileIO;
import org.apache.paimon.fs.Path;
import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.io.DataInputDeserializer;
+import org.apache.paimon.io.DataOutputSerializer;
import org.apache.paimon.manifest.FileSource;
import org.apache.paimon.predicate.Predicate;
import org.apache.paimon.predicate.PredicateBuilder;
+import org.apache.paimon.rest.RESTTokenFileIO;
import org.apache.paimon.schema.TableSchema;
import org.apache.paimon.stats.SimpleStats;
import org.apache.paimon.table.AppendOnlyFileStoreTable;
import org.apache.paimon.table.BucketMode;
import org.apache.paimon.table.CatalogEnvironment;
import org.apache.paimon.table.DataTable;
+import org.apache.paimon.table.FallbackReadFileStoreTable;
import org.apache.paimon.table.FileStoreTable;
import org.apache.paimon.table.Table;
import org.apache.paimon.table.source.DataSplit;
@@ -91,7 +99,9 @@ import org.apache.paimon.table.source.snapshot.SnapshotReader;
import org.apache.paimon.types.DataField;
import org.apache.paimon.types.DataTypes;
import org.apache.paimon.types.IntType;
+import org.apache.paimon.types.LocalZonedTimestampType;
import org.apache.paimon.types.RowType;
+import org.apache.paimon.types.TimestampType;
import org.apache.paimon.utils.InstantiationUtil;
import org.junit.After;
import org.junit.Assert;
@@ -110,6 +120,7 @@ import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Base64;
+import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
@@ -117,6 +128,7 @@ import java.util.Map;
import java.util.Optional;
import java.util.OptionalLong;
import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
@RunWith(MockitoJUnitRunner.class)
public class PaimonScanNodeTest {
@@ -134,6 +146,14 @@ public class PaimonScanNodeTest {
// Statement-scoped scan task reuse is on by default; the mock's field would otherwise
// read false and bypass the cache under test.
sv.enableExternalScanTaskReuse = true;
+ // These routing fixtures bypass scan initialization and use an upgraded eligible BE.
+ Backend backend = GsonUtils.GSON.fromJson("{\"supportsPaimonRustReader\":true}", Backend.class);
+ new MockUp<FederationBackendPolicy>() {
+ @mockit.Mock
+ public Collection<Backend> getBackends() {
+ return Collections.singletonList(backend);
+ }
+ };
}
@After
@@ -474,6 +494,43 @@ public class PaimonScanNodeTest {
Mockito.verify(dataSplit, Mockito.never()).mergedRowCount();
}
+ @Test
+ public void testRustEnabledKeepsNativeAggregateSplits() throws Exception {
+ SessionVariable session = new SessionVariable();
+ session.setEnablePaimonRustReader(true);
+ session.enableFileScannerV2 = true;
+ PaimonScanNode node = Mockito.spy(newTestNode(new PlanNodeId(1), new TupleId(3), session));
+ PaimonSource source = Mockito.mock(PaimonSource.class);
+ DataTable table = Mockito.mock(DataTable.class, Mockito.RETURNS_DEEP_STUBS);
+ RowType rowType = DataTypes.ROW(DataTypes.INT());
+ Mockito.when(table.partitionKeys()).thenReturn(Collections.emptyList());
+ Mockito.when(table.rowType()).thenReturn(rowType);
+ // Native ORC eligibility now inspects historical types even for file aggregates.
+ Mockito.when(table.schemaManager().schema(1L).logicalRowType()).thenReturn(rowType);
+ Mockito.when(source.getPaimonTable()).thenReturn(table);
+ Mockito.when(source.getExternalTable()).thenReturn(Mockito.mock(PaimonExternalTable.class));
+ node.setSource(source);
+ setField(FileQueryScanNode.class, node, "fileSplitter",
+ new FileSplitter(64L * 1024 * 1024, 64L * 1024 * 1024, 0));
+ setField(PaimonScanNode.class, node, "storagePropertiesMap", Collections.emptyMap());
+ DataSplit dataSplit = Mockito.spy(createDataSplit(Arrays.asList("first.parquet", "second.orc")));
+ Mockito.doReturn(Collections.singletonList(dataSplit)).when(node).getPaimonSplitFromAPI();
+
+ // File aggregates need native file readers; enabling Rust must not replace raw-file splits
+ // with logical DataSplits, which require row reads for COUNT(col) and MIN/MAX.
+ for (TPushAggOp aggregate : Arrays.asList(TPushAggOp.COUNT, TPushAggOp.MINMAX)) {
+ node.setPushDownAggNoGrouping(aggregate);
+ node.setPushDownCountSlotIds(Collections.singletonList(new SlotId(7)));
+ List<org.apache.doris.spi.Split> splits = node.getSplits(1);
+ Assert.assertEquals(2, splits.size());
+ for (org.apache.doris.spi.Split split : splits) {
+ Assert.assertNull(((PaimonSplit) split).getSplit());
+ Assert.assertFalse(((PaimonSplit) split).getRowCount().isPresent());
+ }
+ }
+ Mockito.verify(dataSplit, Mockito.never()).mergedRowCount();
+ }
+
@Test
public void testCountColumnKeepsAllSplitsWhileCountStarUsesMergedRowCount() throws UserException {
PaimonScanNode node = Mockito.spy(newTestNode(new PlanNodeId(1), new TupleId(3), sv));
@@ -1846,6 +1903,173 @@ public class PaimonScanNodeTest {
Assert.assertTrue(rangeDesc.getTableFormatParams().getPaimonParams().isSetPaimonSplit());
}
+ @Test
+ public void testRustReaderSelectionRequiresFileScannerV2() throws Exception {
+ // The V1 FileScanner explicitly rejects PAIMON_RUST, so a rust request may
+ // only be encoded when enable_file_scanner_v2 is on; with V2 disabled the
+ // split falls back to JNI so the selected scanner can consume it.
+ for (boolean scannerV2 : Arrays.asList(true, false)) {
+ SessionVariable vars = new SessionVariable();
+ vars.setEnablePaimonRustReader(true);
+ vars.enableFileScannerV2 = scannerV2;
+
+ PaimonScanNode node = new PaimonScanNode(new PlanNodeId(0),
+ new TupleDescriptor(new TupleId(0)), false, vars, ScanContext.EMPTY);
+ PaimonSource source = Mockito.mock(PaimonSource.class);
+ FileStoreTable paimonTable = Mockito.mock(FileStoreTable.class);
+ Mockito.when(paimonTable.schema()).thenReturn(new TableSchema(
+ 1, Collections.singletonList(new DataField(0, "id", new IntType())),
+ 0, Collections.emptyList(), Collections.emptyList(),
+ Collections.emptyMap(), null));
+ PaimonExternalTable externalTable = Mockito.mock(PaimonExternalTable.class);
+ Mockito.when(source.getExternalTable()).thenReturn(externalTable);
+ Mockito.when(source.getTableLocation()).thenReturn("s3://warehouse/wh/db.db/t");
+ setField(PaimonScanNode.class, node, "backendStorageProperties",
+ ImmutableMap.of("AWS_ACCESS_KEY", "example-key", "AWS_SECRET_KEY", "example-secret"));
+ Mockito.when(externalTable.getDbName()).thenReturn("db");
+ Mockito.when(externalTable.getName()).thenReturn("t");
+ node.setSource(source);
+ // The rust gate and schema serialization use doInitialize's cached
+ // getProcessedTable() result, not the raw source table.
+ setField(PaimonScanNode.class, node, "processedTable", paimonTable);
+ setField(PaimonScanNode.class, node, "storagePropertiesMap", Collections.emptyMap());
+
+ TFileRangeDesc rangeDesc = new TFileRangeDesc();
+ invokePrivateMethod(node, "setPaimonParams",
+ new Class<?>[] {TFileRangeDesc.class, PaimonSplit.class},
+ rangeDesc, new PaimonSplit(createDataSplit("rust_gate.parquet")));
+
+ Assert.assertEquals(scannerV2 ? TPaimonReaderType.PAIMON_RUST : TPaimonReaderType.PAIMON_JNI,
+ rangeDesc.getTableFormatParams().getPaimonParams().getReaderType());
+ }
+ }
+
+ @Test
+ public void testRustSchemaJsonShipsProcessedTableRelationOptions() throws Exception {
+ // Relation options such as t@options('read.batch-size'='1') are applied by
+ // getProcessedTable(); the JNI reader serializes that effective table, and the
+ // rust reader rebuilds its table from the shipped schema JSON, deriving its
+ // read batch size from the schema options. Only the processed table's schema
+ // carries the override here, and source.getPaimonTable() is deliberately left
+ // unstubbed (it returns null): if the serialization ever regresses to the raw
+ // cached table, the rust gate falls back to JNI and this test fails.
+ SessionVariable vars = new SessionVariable();
+ vars.setEnablePaimonRustReader(true);
+ vars.enableFileScannerV2 = true;
+
+ PaimonScanNode node = new PaimonScanNode(new PlanNodeId(0),
+ new TupleDescriptor(new TupleId(0)), false, vars, ScanContext.EMPTY);
+
+ // The processed table: the relation override is merged into its schema.
+ FileStoreTable processed = Mockito.mock(FileStoreTable.class);
+ Mockito.when(processed.schema()).thenReturn(new TableSchema(
+ 1, Collections.singletonList(new DataField(0, "id", new IntType())),
+ 0, Collections.emptyList(), Collections.emptyList(),
+ ImmutableMap.of("read.batch-size", "1"), null));
+
+ PaimonSource source = Mockito.mock(PaimonSource.class);
+ PaimonExternalTable externalTable = Mockito.mock(PaimonExternalTable.class);
+ Mockito.when(source.getExternalTable()).thenReturn(externalTable);
+ Mockito.when(source.getTableLocation()).thenReturn("s3://warehouse/wh/db.db/t");
+ setField(PaimonScanNode.class, node, "backendStorageProperties",
+ ImmutableMap.of("AWS_ACCESS_KEY", "example-key", "AWS_SECRET_KEY", "example-secret"));
+ Mockito.when(externalTable.getDbName()).thenReturn("db");
+ Mockito.when(externalTable.getName()).thenReturn("t");
+ node.setSource(source);
+ setField(PaimonScanNode.class, node, "processedTable", processed);
+ setField(PaimonScanNode.class, node, "storagePropertiesMap", Collections.emptyMap());
+
+ TFileRangeDesc rangeDesc = new TFileRangeDesc();
+ invokePrivateMethod(node, "setPaimonParams",
+ new Class<?>[] {TFileRangeDesc.class, PaimonSplit.class},
+ rangeDesc, new PaimonSplit(createDataSplit("relation_options.parquet")));
+
+ org.apache.doris.thrift.TPaimonFileDesc fileDesc =
+ rangeDesc.getTableFormatParams().getPaimonParams();
+ Assert.assertEquals(TPaimonReaderType.PAIMON_RUST, fileDesc.getReaderType());
+ TableSchema shipped = org.apache.paimon.utils.JsonSerdeUtil.fromJson(
+ fileDesc.getPaimonTableSchemaJson(), TableSchema.class);
+ Assert.assertEquals("1", shipped.options().get("read.batch-size"));
+ }
+
+ @Test
+ public void testRustSchemaJsonStripsTimeTravelSelectors() throws Exception {
+ // A statement fence pins the data snapshot by merging scan.snapshot-id into the
+ // schema options (isolateSnapshotRead) while the fields keep the latest schema —
+ // a schema-only ALTER after the last data commit must stay visible. The rust
+ // reader pins data via the serialized DataSplit and re-resolves a shipped
+ // selector in copy_with_time_travel, swapping the fields back to the pinned
+ // snapshot's older schema; the shipped JSON must therefore carry the resolved
+ // fields without the planning selectors, keeping every other option.
+ // The derived scan.mode is stripped with it in every spelling: paimon accepts
+ // enum option values case-insensitively but preserves the original string in
+ // the options map, so a table can persist scan.mode=FROM-SNAPSHOT — the rust
+ // reader recognizes the value case-insensitively and would reject the now-bare
+ // uppercase mode for lacking its selector, while JNI accepts the original
+ // option.
+ for (String scanMode : new String[] {"from-snapshot", "FROM-SNAPSHOT", " From-Snapshot "}) {
+ SessionVariable vars = new SessionVariable();
+ vars.setEnablePaimonRustReader(true);
+ vars.enableFileScannerV2 = true;
+
+ PaimonScanNode node = new PaimonScanNode(new PlanNodeId(0),
+ new TupleDescriptor(new TupleId(0)), false, vars, ScanContext.EMPTY);
+
+ // Resolved (latest) schema: the column added after the last data commit is a
+ // field, and the fence snapshot id is only planning state in the options.
+ FileStoreTable processed = Mockito.mock(FileStoreTable.class);
+ Mockito.when(processed.schema()).thenReturn(new TableSchema(
+ 1, Arrays.asList(
+ new DataField(0, "id", new IntType()),
+ new DataField(1, "added_after", DataTypes.STRING())),
+ 1, Collections.emptyList(), Collections.emptyList(),
+ ImmutableMap.of(
+ "scan.snapshot-id", "3",
+ "scan.tag-name", "stale-tag",
+ // Paimon's copyInternal materializes the derived scan mode for
+ // the merged selector; it dangles once the selector is stripped.
+ "scan.mode", scanMode,
+ "read.batch-size", "1"),
+ null));
+
+ PaimonSource source = Mockito.mock(PaimonSource.class);
+ PaimonExternalTable externalTable = Mockito.mock(PaimonExternalTable.class);
+ Mockito.when(source.getExternalTable()).thenReturn(externalTable);
+ Mockito.when(source.getTableLocation()).thenReturn("s3://warehouse/wh/db.db/t");
+ setField(PaimonScanNode.class, node, "backendStorageProperties",
+ ImmutableMap.of("AWS_ACCESS_KEY", "example-key", "AWS_SECRET_KEY", "example-secret"));
+ Mockito.when(externalTable.getDbName()).thenReturn("db");
+ Mockito.when(externalTable.getName()).thenReturn("t");
+ node.setSource(source);
+ setField(PaimonScanNode.class, node, "processedTable", processed);
+ setField(PaimonScanNode.class, node, "storagePropertiesMap", Collections.emptyMap());
+
+ TFileRangeDesc rangeDesc = new TFileRangeDesc();
+ invokePrivateMethod(node, "setPaimonParams",
+ new Class<?>[] {TFileRangeDesc.class, PaimonSplit.class},
+ rangeDesc, new PaimonSplit(createDataSplit("strip_selectors.parquet")));
+
+ org.apache.doris.thrift.TPaimonFileDesc fileDesc =
+ rangeDesc.getTableFormatParams().getPaimonParams();
+ Assert.assertEquals("scan.mode " + scanMode, TPaimonReaderType.PAIMON_RUST,
+ fileDesc.getReaderType());
+ TableSchema shipped = org.apache.paimon.utils.JsonSerdeUtil.fromJson(
+ fileDesc.getPaimonTableSchemaJson(), TableSchema.class);
+ Assert.assertNull(shipped.options().get(CoreOptions.SCAN_SNAPSHOT_ID.key()));
+ Assert.assertNull(shipped.options().get(CoreOptions.SCAN_TAG_NAME.key()));
+ // The fence-materialized "from-snapshot" would otherwise dangle: the rust
+ // ReadBuilder requires a selector for it and rejects the open.
+ Assert.assertNull("scan.mode " + scanMode + " must be stripped",
+ shipped.options().get(CoreOptions.SCAN_MODE.key()));
+ Assert.assertEquals("1", shipped.options().get("read.batch-size"));
+ // The resolved fields are transported untouched, including the column added
+ // after the last data commit.
+ Assert.assertEquals(
+ Arrays.asList("id", "added_after"),
+ shipped.fields().stream().map(DataField::name).collect(Collectors.toList()));
+ }
+ }
+
@Test
public void testGetFieldIndexMatchesMixedCaseColumns() {
List<String> fieldNames = Arrays.asList("data", "mIxEd_COL", "PART");
@@ -2085,15 +2309,21 @@ public class PaimonScanNodeTest {
}
private DataSplit createDataSplit(String fileName) {
- DataFileMeta dataFileMeta = DataFileMeta.forAppend(fileName, 64L * 1024 * 1024, 1L, SimpleStats.EMPTY_STATS,
- 1L, 1L, 1L, Collections.<String>emptyList(), null, FileSource.APPEND,
- Collections.<String>emptyList(), null, null, Collections.<String>emptyList());
+ return createDataSplit(Collections.singletonList(fileName));
+ }
+
+ private DataSplit createDataSplit(List<String> fileNames) {
+ List<DataFileMeta> dataFileMetas = fileNames.stream().map(fileName ->
+ DataFileMeta.forAppend(fileName, 64L * 1024 * 1024, 1L, SimpleStats.EMPTY_STATS,
+ 1L, 1L, 1L, Collections.<String>emptyList(), null, FileSource.APPEND,
+ Collections.<String>emptyList(), null, null, Collections.<String>emptyList()))
+ .collect(Collectors.toList());
return DataSplit.builder()
.rawConvertible(true)
.withPartition(BinaryRow.singleColumn(1))
.withBucket(1)
.withBucketPath("file://b1")
- .withDataFiles(Collections.singletonList(dataFileMeta))
+ .withDataFiles(dataFileMetas)
.build();
}
@@ -2111,4 +2341,1311 @@ public class PaimonScanNodeTest {
Mockito.when(dataSplit.deletionFiles()).thenReturn(Optional.empty());
return dataSplit;
}
+
+ @Test
+ public void testRustReaderSelectionRejectsFallbackSplits() throws Exception {
+ RustRoutingFixture f = new RustRoutingFixture();
+ DataOutputSerializer out = new DataOutputSerializer(1024);
+ f.split.serialize(out);
+ out.writeBoolean(true);
+ DataInputDeserializer in = new DataInputDeserializer();
+ in.setBuffer(out.getSharedBuffer(), 0, out.length());
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI,
+ f.reader(FallbackReadFileStoreTable.FallbackDataSplit.deserialize(in)));
+ setField(PaimonScanNode.class, f.node, "processedTable", Mockito.mock(FallbackReadFileStoreTable.class));
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI, f.reader(f.split));
+ }
+
+ @Test
+ public void testRustReaderSelectionRejectsQueryAuthTables() throws Exception {
+ RustRoutingFixture f = new RustRoutingFixture();
+ f.schema(1, new IntType(), ImmutableMap.of("query-auth.enabled", "true"), false);
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI, f.reader(f.split));
+ }
+
+ @Test
+ public void testRustReaderSelectionRejectsUnsupportedPkDvModes() throws Exception {
+ // The pinned rust read_pk supports partial-update / aggregation tables
+ // with deletion vectors only in the fully materialized shape: it rejects
+ // deletion-vectors.merge-on-read=true outright, and otherwise requires
+ // every split to be compacted (level != 0) and known free of retract rows
+ // (DataSplit::is_fully_materialized_pk_dv). All other shapes are valid
+ // Java/JNI scans, so they must fall back here instead of failing the
+ // BE open. Deduplicate stays rust-eligible: its read_pk routes
+ // uncompacted DV splits to the KV reader, which applies the DVs.
+ SessionVariable vars = new SessionVariable();
+ vars.setEnablePaimonRustReader(true);
+ vars.enableFileScannerV2 = true;
+
+ // createDataSplit builds a level-0 file — the ordinary uncompacted shape.
+ DataSplit levelZeroSplit = createDataSplit("dv_level0.parquet");
+ DataSplit unknownDeleteCountSplit = dvSplitOverFiles(
+ dvCompactedFile("dv_unknown.parquet", null));
+ DataSplit retractsSplit = dvSplitOverFiles(
+ dvCompactedFile("dv_retracts.parquet", 1L));
+ DataSplit materializedSplit = dvSplitOverFiles(
+ dvCompactedFile("dv_materialized.parquet", 0L));
+
+ for (CoreOptions.MergeEngine engine : Arrays.asList(
+ CoreOptions.MergeEngine.PARTIAL_UPDATE, CoreOptions.MergeEngine.AGGREGATE)) {
+ // deletion-vectors.merge-on-read=true: rejected table-wide.
+ Assert.assertEquals("merge-on-read " + engine, TPaimonReaderType.PAIMON_JNI,
+ dvReaderTypeOf(vars, engine, true,
+ ImmutableMap.of("deletion-vectors.merge-on-read", "true"),
+ materializedSplit));
+ // merge-on-read=false but the split still needs per-key merge work:
+ // level-0 data, unknown delete count, or known retracts — all fall
+ // back per split.
+ Assert.assertEquals("level-0 " + engine, TPaimonReaderType.PAIMON_JNI,
+ dvReaderTypeOf(vars, engine, true, Collections.emptyMap(), levelZeroSplit));
+ Assert.assertEquals("unknown delete count " + engine, TPaimonReaderType.PAIMON_JNI,
+ dvReaderTypeOf(vars, engine, true, Collections.emptyMap(), unknownDeleteCountSplit));
+ Assert.assertEquals("retracts " + engine, TPaimonReaderType.PAIMON_JNI,
+ dvReaderTypeOf(vars, engine, true, Collections.emptyMap(), retractsSplit));
+ // A fully materialized compacted split reads raw on rust, and without
+ // deletion vectors the KV reader handles any split.
+ Assert.assertEquals("materialized " + engine, TPaimonReaderType.PAIMON_RUST,
+ dvReaderTypeOf(vars, engine, true, Collections.emptyMap(), materializedSplit));
+ Assert.assertEquals("no dv " + engine, TPaimonReaderType.PAIMON_RUST,
+ dvReaderTypeOf(vars, engine, false, Collections.emptyMap(), levelZeroSplit));
+ }
+
+ // Deduplicate keeps rust eligibility for uncompacted DV splits and for
+ // merge-on-read=true: the KV reader applies the attached per-file DVs.
+ Assert.assertEquals(TPaimonReaderType.PAIMON_RUST,
+ dvReaderTypeOf(vars, CoreOptions.MergeEngine.DEDUPLICATE, true,
+ ImmutableMap.of("deletion-vectors.merge-on-read", "true"), levelZeroSplit));
+ }
+
+ // Builds a node whose processed table is a primary-key table with the given
+ // merge engine and deletion-vector options (all other gates open: no
+ // query-auth, v2 on, ordinary DataSplit) and returns the reader type chosen
+ // for the given split.
+ private TPaimonReaderType dvReaderTypeOf(SessionVariable vars, CoreOptions.MergeEngine mergeEngine,
+ boolean deletionVectorsEnabled, Map<String, String> schemaOptions, DataSplit split)
+ throws Exception {
+ PaimonScanNode node = new PaimonScanNode(new PlanNodeId(0),
+ new TupleDescriptor(new TupleId(0)), false, vars, ScanContext.EMPTY);
+ PaimonSource source = Mockito.mock(PaimonSource.class);
+ FileStoreTable paimonTable = Mockito.mock(FileStoreTable.class);
+ // lenient: the schema / external-table stubs are only consumed when the
+ // split stays rust-eligible.
+ Mockito.lenient().when(paimonTable.schema()).thenReturn(new TableSchema(
+ 1, Collections.singletonList(new DataField(0, "id", new IntType())),
+ 0, Collections.emptyList(), Collections.emptyList(), schemaOptions, null));
+ CoreOptions options = Mockito.mock(CoreOptions.class);
+ Mockito.when(paimonTable.coreOptions()).thenReturn(options);
+ Mockito.when(options.queryAuthEnabled()).thenReturn(false);
+ Mockito.when(options.mergeEngine()).thenReturn(mergeEngine);
+ Mockito.when(options.deletionVectorsEnabled()).thenReturn(deletionVectorsEnabled);
+ PaimonExternalTable externalTable = Mockito.mock(PaimonExternalTable.class);
+ Mockito.lenient().when(source.getExternalTable()).thenReturn(externalTable);
+ Mockito.when(source.getTableLocation()).thenReturn("s3://warehouse/wh/db.db/t");
+ setField(PaimonScanNode.class, node, "backendStorageProperties",
+ ImmutableMap.of("AWS_ACCESS_KEY", "example-key", "AWS_SECRET_KEY", "example-secret"));
+ Mockito.lenient().when(externalTable.getDbName()).thenReturn("db");
+ Mockito.lenient().when(externalTable.getName()).thenReturn("t");
+ node.setSource(source);
+ setField(PaimonScanNode.class, node, "processedTable", paimonTable);
+
+ TFileRangeDesc rangeDesc = new TFileRangeDesc();
+ invokePrivateMethod(node, "setPaimonParams",
+ new Class<?>[] {TFileRangeDesc.class, PaimonSplit.class},
+ rangeDesc, new PaimonSplit(split));
+ return rangeDesc.getTableFormatParams().getPaimonParams().getReaderType();
+ }
+
+ // A compacted (level != 0) data file with an explicit retract-row count:
+ // null = unknown, 0 = fully materialized, >0 = rows still to retract.
+ private static DataFileMeta dvCompactedFile(String fileName, Long deleteRowCount) {
+ return DataFileMeta.create(fileName, 64L * 1024 * 1024, 1L,
+ DataFileMeta.EMPTY_MIN_KEY, DataFileMeta.EMPTY_MAX_KEY,
+ SimpleStats.EMPTY_STATS, SimpleStats.EMPTY_STATS,
+ 1L, 1L, 1L, 5, deleteRowCount, null,
+ FileSource.APPEND, Collections.emptyList(), null, Collections.emptyList());
+ }
+
+ private static DataSplit dvSplitOverFiles(DataFileMeta... files) {
+ return DataSplit.builder()
+ .rawConvertible(true)
+ .withPartition(BinaryRow.singleColumn(1))
+ .withBucket(1)
+ .withBucketPath("file://b1")
+ .withDataFiles(Arrays.asList(files))
+ .build();
+ }
+
+ @Test
+ public void testRustReaderSelectionRejectsOrcTimestampLtzSchemas() throws Exception {
+ RustRoutingFixture f = new RustRoutingFixture();
+ Assert.assertEquals(TPaimonReaderType.PAIMON_RUST, f.reader(createDataSplit("plain.orc")));
+ f.schema(1, new org.apache.paimon.types.LocalZonedTimestampType(6), Collections.emptyMap(), false);
+ Assert.assertEquals(TPaimonReaderType.PAIMON_RUST, f.reader(f.split));
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI, f.reader(createDataSplit("ltz.orc")));
+ }
+
+ @Test
+ public void testRustReaderSelectionRejectsMixedFormatOrcLtzSplits() throws Exception {
+ // Paimon allows per-level file.format, so one DataSplit can mix
+ // Parquet and ORC members. The old gate read only the split path's
+ // suffix (the first file), so a first-Parquet/later-ORC split with an
+ // LTZ schema passed while rust still applied the shifted ORC decode
+ // to the ORC members. The format now comes from every member file.
+ SessionVariable vars = new SessionVariable();
+ vars.setEnablePaimonRustReader(true);
+ vars.enableFileScannerV2 = true;
+ List<DataField> ltzFields = Arrays.asList(
+ new DataField(0, "id", new IntType()),
+ new DataField(1, "ts_ltz", new LocalZonedTimestampType(6)));
+
+ // First file Parquet, later member ORC: the first file's suffix no
+ // longer speaks for the split -> JNI.
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI,
+ readerTypeOf(vars, ltzFields, "first.parquet", "second.orc"));
+ // ORC first, later Parquet member (the shape the suffix check did
+ // catch) stays JNI.
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI,
+ readerTypeOf(vars, ltzFields, "first.orc", "second.parquet"));
+ // All-Parquet members with the same LTZ schema stay rust-eligible.
+ Assert.assertEquals(TPaimonReaderType.PAIMON_RUST,
+ readerTypeOf(vars, ltzFields, "a.parquet", "b.parquet"));
+ }
+
+ @Test
+ public void testRustReaderSelectionRejectsNestedTimestampLtzSchemas() throws Exception {
+ // An LTZ nested under MAP/ARRAY/ROW reaches the same shifted ORC
+ // decode through the container's field materialization; the old
+ // top-level-only type-root check missed it. The search recurses; the
+ // same containers without LTZ stay rust-eligible, and Parquet splits
+ // with nested LTZ are unaffected (only the ORC decoder diverges).
+ SessionVariable vars = new SessionVariable();
+ vars.setEnablePaimonRustReader(true);
+ vars.enableFileScannerV2 = true;
+ LocalZonedTimestampType ltz = new LocalZonedTimestampType(6);
+
+ // LTZ under each container kind, uniform-ORC split -> JNI.
+ for (DataField nested : Arrays.asList(
+ new DataField(1, "m", new org.apache.paimon.types.MapType(new IntType(), ltz)),
+ new DataField(1, "arr", new org.apache.paimon.types.ArrayType(ltz)),
+ new DataField(1, "r",
+ new RowType(Collections.singletonList(new DataField(0, "ts_ltz", ltz)))))) {
+ Assert.assertEquals("nested " + nested.name() + " under ORC",
+ TPaimonReaderType.PAIMON_JNI,
+ readerTypeOf(vars, Arrays.asList(
+ new DataField(0, "id", new IntType()), nested),
+ "nested.orc"));
+ // The same nested-LTZ schema over Parquet stays rust-eligible.
+ Assert.assertEquals("nested " + nested.name() + " over Parquet",
+ TPaimonReaderType.PAIMON_RUST,
+ readerTypeOf(vars, Arrays.asList(
+ new DataField(0, "id", new IntType()), nested),
+ "nested.parquet"));
+ }
+
+ // Containers without any LTZ over ORC stay rust-eligible.
+ Assert.assertEquals(TPaimonReaderType.PAIMON_RUST,
+ readerTypeOf(vars, Arrays.asList(
+ new DataField(0, "id", new IntType()),
+ new DataField(1, "m",
+ new org.apache.paimon.types.MapType(new IntType(), new IntType()))),
+ "map_int.orc"));
+ }
+
+ @Test
+ public void testRustReaderSelectionRejectsExternalDataFileSplits() throws Exception {
+ // data-file.external-paths: paimon can store an absolute location in
+ // each DataFileMeta, and both Java and the serialized rust split prefer
+ // it over the bucket path — but the pinned rust table builds one FileIO
+ // from paimon_table, whose storage enum parses every file with the
+ // warehouse-selected backend: an hdfs table with an s3:// external file
+ // (or an s3 table with an oss:// file) reaches the wrong parser and
+ // fails the open, while JNI reads it. Any split carrying an external
+ // file therefore routes to JNI, regardless of the external path's own
+ // scheme (the shipped options describe only the warehouse).
+ SessionVariable vars = new SessionVariable();
+ vars.setEnablePaimonRustReader(true);
+ vars.enableFileScannerV2 = true;
+ List<DataField> fields = Arrays.asList(
+ new DataField(0, "id", new IntType()),
+ new DataField(1, "ts", new TimestampType(6)));
+
+ // A split whose file carries an external path -> JNI.
+ DataFileMeta external = DataFileMeta.forAppend(
+ "external.parquet", 64L * 1024 * 1024, 1L, SimpleStats.EMPTY_STATS,
+ 1L, 1L, 1L, Collections.<String>emptyList(), null, FileSource.APPEND,
+ Collections.<String>emptyList(), null, null, Collections.<String>emptyList())
+ .newExternalPath("hdfs://other-nn/external/wh/external.parquet");
+ DataSplit externalSplit = DataSplit.builder()
+ .rawConvertible(true)
+ .withPartition(BinaryRow.singleColumn(1))
+ .withBucket(1)
+ .withBucketPath("file://b1")
+ .withDataFiles(Collections.singletonList(external))
+ .build();
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI,
+ readerTypeOfSplit(vars, fields, externalSplit));
+
+ // The same split without an external path stays rust-eligible.
+ DataSplit plainSplit = DataSplit.builder()
+ .rawConvertible(true)
+ .withPartition(BinaryRow.singleColumn(1))
+ .withBucket(1)
+ .withBucketPath("file://b1")
+ .withDataFiles(Collections.singletonList(DataFileMeta.forAppend(
+ "plain.parquet", 64L * 1024 * 1024, 1L, SimpleStats.EMPTY_STATS,
+ 1L, 1L, 1L, Collections.<String>emptyList(), null, FileSource.APPEND,
+ Collections.<String>emptyList(), null, null, Collections.<String>emptyList())))
+ .build();
+ Assert.assertEquals(TPaimonReaderType.PAIMON_RUST,
+ readerTypeOfSplit(vars, fields, plainSplit));
+ }
+
+ @Test
+ public void testRustReaderSelectionRejectsDeduplicateIgnoreDeleteTables() throws Exception {
+ // deduplicate.ignore-delete=true tables stay on JNI: Java's
+ // DeduplicateMergeFunction skips retract records when the option is set
+ // (including old, uncompacted files that still contain them), but the
+ // pinned rust read_pk does not pass table options into its deduplicate
+ // merge — it picks the latest row and omits the key when that row is
+ // DELETE/UPDATE_BEFORE. An uncompacted insert followed by a delete
+ // therefore returns the insert through JNI but silently disappears
+ // through rust.
+ for (Object[] shape : new Object[][] {
+ // DEDUPLICATE engine with the option on -> JNI.
+ {CoreOptions.MergeEngine.DEDUPLICATE, true, TPaimonReaderType.PAIMON_JNI},
+ // Option off, same engine -> rust.
+ {CoreOptions.MergeEngine.DEDUPLICATE, false, TPaimonReaderType.PAIMON_RUST},
+ // Option on under a non-deduplicate engine: meaningless there,
+ // the gate must not reject -> rust.
+ {CoreOptions.MergeEngine.PARTIAL_UPDATE, true, TPaimonReaderType.PAIMON_RUST}}) {
+ CoreOptions.MergeEngine mergeEngine = (CoreOptions.MergeEngine) shape[0];
+ boolean ignoreDelete = (Boolean) shape[1];
+ TPaimonReaderType expected = (TPaimonReaderType) shape[2];
+
+ SessionVariable vars = new SessionVariable();
+ vars.setEnablePaimonRustReader(true);
+ vars.enableFileScannerV2 = true;
+
+ PaimonScanNode node = new PaimonScanNode(new PlanNodeId(0),
+ new TupleDescriptor(new TupleId(0)), false, vars, ScanContext.EMPTY);
+ PaimonSource source = Mockito.mock(PaimonSource.class);
+ FileStoreTable paimonTable = Mockito.mock(FileStoreTable.class);
+ CoreOptions options = Mockito.mock(CoreOptions.class);
+ Mockito.when(paimonTable.coreOptions()).thenReturn(options);
+ Mockito.when(options.queryAuthEnabled()).thenReturn(false);
+ Mockito.when(options.mergeEngine()).thenReturn(mergeEngine);
+ Mockito.when(options.ignoreDelete()).thenReturn(ignoreDelete);
+ PaimonExternalTable externalTable = Mockito.mock(PaimonExternalTable.class);
+ Mockito.when(source.getExternalTable()).thenReturn(externalTable);
+ Mockito.when(source.getTableLocation()).thenReturn("s3://warehouse/wh/db.db/t");
+ setField(PaimonScanNode.class, node, "backendStorageProperties",
+ ImmutableMap.of("AWS_ACCESS_KEY", "example-key", "AWS_SECRET_KEY", "example-secret"));
+ Mockito.when(externalTable.getDbName()).thenReturn("db");
+ Mockito.when(externalTable.getName()).thenReturn("t");
+ node.setSource(source);
+ setField(PaimonScanNode.class, node, "processedTable", paimonTable);
+ if (expected == TPaimonReaderType.PAIMON_RUST) {
+ // Only the rust branch serializes the schema (a JNI-expected
+ // stub would trip Mockito's strict unused-stubbing check).
+ Mockito.when(paimonTable.schema()).thenReturn(new TableSchema(
+ 1, Collections.singletonList(new DataField(0, "id", new IntType())),
+ 0, Collections.emptyList(), Collections.emptyList(),
+ Collections.emptyMap(), null));
+ }
+
+ TFileRangeDesc rangeDesc = new TFileRangeDesc();
+ invokePrivateMethod(node, "setPaimonParams",
+ new Class<?>[] {TFileRangeDesc.class, PaimonSplit.class},
+ rangeDesc, new PaimonSplit(createDataSplit("dedup_ignore_del.parquet")));
+ Assert.assertEquals("engine " + mergeEngine + ", ignore-delete " + ignoreDelete,
+ expected, rangeDesc.getTableFormatParams().getPaimonParams().getReaderType());
+ }
+ }
+
+ @Test
+ public void testRustReaderSelectionRejectsRestTokenTables() throws Exception {
+ // doInitialize snapshots RESTTokenFileIO.validToken().token() into
+ // the backend storage properties, discarding expireAtMillis and the
+ // REST refresh context, so the shipped credentials look static — but
+ // the pinned rust table reuses one option map with no refresh
+ // callback, while paimon 1.4.2's JNI RESTTokenFileIO checks expiry
+ // before each file operation and obtains a replacement token. A scan
+ // crossing the token TTL would start on rust and later fail
+ // authentication; only the table's FileIO type reveals the token is
+ // renewable.
+ for (boolean restToken : new boolean[] {true, false}) {
+ SessionVariable vars = new SessionVariable();
+ vars.setEnablePaimonRustReader(true);
+ vars.enableFileScannerV2 = true;
+
+ PaimonScanNode node = new PaimonScanNode(new PlanNodeId(0),
+ new TupleDescriptor(new TupleId(0)), false, vars, ScanContext.EMPTY);
+ PaimonSource source = Mockito.mock(PaimonSource.class);
+ FileStoreTable paimonTable = Mockito.mock(FileStoreTable.class);
+ CoreOptions options = Mockito.mock(CoreOptions.class);
+ Mockito.when(paimonTable.coreOptions()).thenReturn(options);
+ Mockito.when(options.queryAuthEnabled()).thenReturn(false);
+ // S3-location static credentials, an ordinary shape the
+ // REST-token snapshot also produces.
+ setField(PaimonScanNode.class, node, "backendStorageProperties", ImmutableMap.of(
+ "AWS_CREDENTIALS_PROVIDER_TYPE", "DEFAULT",
+ "AWS_ACCESS_KEY", "ak",
+ "AWS_SECRET_KEY", "sk",
+ "AWS_ENDPOINT", "http://127.0.0.1:19001",
+ "AWS_REGION", "us-east-1"));
+ if (restToken) {
+ Mockito.when(paimonTable.fileIO())
+ .thenReturn(Mockito.mock(RESTTokenFileIO.class));
+ // The JNI branch never serializes the schema.
+ } else {
+ Mockito.when(paimonTable.schema()).thenReturn(new TableSchema(
+ 1, Collections.singletonList(new DataField(0, "id", new IntType())),
+ 0, Collections.emptyList(), Collections.emptyList(),
+ Collections.emptyMap(), null));
+ // A plain (or unresolved) FileIO shape stays rust-eligible.
+ Mockito.when(paimonTable.fileIO()).thenReturn(null);
+ }
+ PaimonExternalTable externalTable = Mockito.mock(PaimonExternalTable.class);
+ Mockito.when(source.getExternalTable()).thenReturn(externalTable);
+ Mockito.when(source.getTableLocation()).thenReturn("s3://warehouse/wh/db.db/t");
+ Mockito.when(externalTable.getDbName()).thenReturn("db");
+ Mockito.when(externalTable.getName()).thenReturn("t");
+ node.setSource(source);
+ setField(PaimonScanNode.class, node, "processedTable", paimonTable);
+
+ TFileRangeDesc rangeDesc = new TFileRangeDesc();
+ invokePrivateMethod(node, "setPaimonParams",
+ new Class<?>[] {TFileRangeDesc.class, PaimonSplit.class},
+ rangeDesc, new PaimonSplit(createDataSplit("rest_token.parquet")));
+ Assert.assertEquals("RESTTokenFileIO=" + restToken,
+ restToken ? TPaimonReaderType.PAIMON_JNI : TPaimonReaderType.PAIMON_RUST,
+ rangeDesc.getTableFormatParams().getPaimonParams().getReaderType());
+ }
+ }
+
+ @Test
+ public void testRustReaderSelectionRejectsRustUnsupportedMergeOptions() throws Exception {
+ // Java supports partial-update.remove-record-on-delete /
+ // aggregation.remove-record-on-delete and the wider per-field option
+ // matrix, but the pinned paimon-rust PartialUpdateConfig /
+ // AggregationConfig validations return Unsupported for them — the
+ // rust predicates check option-key PRESENCE, so the FE gate mirrors
+ // presence exactly (an off value still routes to JNI, because the
+ // rust merge construction rejects the bare key). Non-DV tables are
+ // the review's case: without this gate an ordinary DataSplit passes
+ // the compound gate and fails during the rust merge construction
+ // instead of scanning through JNI.
+ for (Object[] shape : new Object[][] {
+ // Partial-update unsupported keys -> JNI.
+ {CoreOptions.MergeEngine.PARTIAL_UPDATE, "partial-update.remove-record-on-delete",
+ TPaimonReaderType.PAIMON_JNI},
+ {CoreOptions.MergeEngine.PARTIAL_UPDATE,
+ "partial-update.remove-record-on-sequence-group",
+ TPaimonReaderType.PAIMON_JNI},
+ {CoreOptions.MergeEngine.PARTIAL_UPDATE, "fields.v.ignore-retract",
+ TPaimonReaderType.PAIMON_JNI},
+ {CoreOptions.MergeEngine.PARTIAL_UPDATE, "fields.v.distinct",
+ TPaimonReaderType.PAIMON_JNI},
+ {CoreOptions.MergeEngine.PARTIAL_UPDATE, "fields.v.ignore-delete",
+ TPaimonReaderType.PAIMON_JNI},
+ // Aggregation unsupported keys -> JNI.
+ {CoreOptions.MergeEngine.AGGREGATE, "aggregation.remove-record-on-delete",
+ TPaimonReaderType.PAIMON_JNI},
+ {CoreOptions.MergeEngine.AGGREGATE, "fields.v.sequence-group",
+ TPaimonReaderType.PAIMON_JNI},
+ {CoreOptions.MergeEngine.AGGREGATE, "ignore-delete",
+ TPaimonReaderType.PAIMON_JNI},
+ // Keys the rust read path DOES support: bare /
+ // partial-update-prefixed ignore-delete, sequence groups and
+ // field aggregation stay rust-eligible.
+ {CoreOptions.MergeEngine.PARTIAL_UPDATE, "partial-update.ignore-delete",
+ TPaimonReaderType.PAIMON_RUST},
+ {CoreOptions.MergeEngine.PARTIAL_UPDATE, "fields.v.sequence-group",
+ TPaimonReaderType.PAIMON_RUST},
+ {CoreOptions.MergeEngine.PARTIAL_UPDATE, "fields.v.aggregate-function",
+ TPaimonReaderType.PAIMON_RUST},
+ {CoreOptions.MergeEngine.AGGREGATE, "fields.v.aggregate-function",
+ TPaimonReaderType.PAIMON_RUST},
+ {CoreOptions.MergeEngine.AGGREGATE, "fields.default-aggregate-function",
+ TPaimonReaderType.PAIMON_RUST}}) {
+ CoreOptions.MergeEngine mergeEngine = (CoreOptions.MergeEngine) shape[0];
+ String optionKey = (String) shape[1];
+ TPaimonReaderType expected = (TPaimonReaderType) shape[2];
+
+ SessionVariable vars = new SessionVariable();
+ vars.setEnablePaimonRustReader(true);
+ vars.enableFileScannerV2 = true;
+
+ PaimonScanNode node = new PaimonScanNode(new PlanNodeId(0),
+ new TupleDescriptor(new TupleId(0)), false, vars, ScanContext.EMPTY);
+ PaimonSource source = Mockito.mock(PaimonSource.class);
+ FileStoreTable paimonTable = Mockito.mock(FileStoreTable.class);
+ CoreOptions options = Mockito.mock(CoreOptions.class);
+ Mockito.when(paimonTable.coreOptions()).thenReturn(options);
+ Mockito.when(options.queryAuthEnabled()).thenReturn(false);
+ Mockito.when(options.mergeEngine()).thenReturn(mergeEngine);
+ Mockito.when(paimonTable.schema()).thenReturn(new TableSchema(
+ 1, Collections.singletonList(new DataField(0, "id", new IntType())),
+ 0, Collections.emptyList(), Collections.emptyList(),
+ ImmutableMap.of(optionKey, "true"), null));
+ PaimonExternalTable externalTable = Mockito.mock(PaimonExternalTable.class);
+ Mockito.when(source.getExternalTable()).thenReturn(externalTable);
+ Mockito.when(source.getTableLocation()).thenReturn("s3://warehouse/wh/db.db/t");
+ setField(PaimonScanNode.class, node, "backendStorageProperties",
+ ImmutableMap.of("AWS_ACCESS_KEY", "example-key", "AWS_SECRET_KEY", "example-secret"));
+ Mockito.when(externalTable.getDbName()).thenReturn("db");
+ Mockito.when(externalTable.getName()).thenReturn("t");
+ node.setSource(source);
+ setField(PaimonScanNode.class, node, "processedTable", paimonTable);
+
+ TFileRangeDesc rangeDesc = new TFileRangeDesc();
+ invokePrivateMethod(node, "setPaimonParams",
+ new Class<?>[] {TFileRangeDesc.class, PaimonSplit.class},
+ rangeDesc, new PaimonSplit(createDataSplit("merge_option_gate.parquet")));
+ Assert.assertEquals("engine " + mergeEngine + ", option " + optionKey, expected,
+ rangeDesc.getTableFormatParams().getPaimonParams().getReaderType());
+ }
+ }
+
+ // Builds a node whose processed table carries the given row-type fields
+ // (all other gates open: no query-auth, v2 on, ordinary DataSplit) and
+ // returns the reader type chosen for a split over the given member files.
+ // Multiple file names build a multi-file split (paimon allows per-level
+ // file.format, so one DataSplit can mix Parquet and ORC members).
+ private TPaimonReaderType readerTypeOf(SessionVariable vars, List<DataField> fields,
+ String... fileNames) throws Exception {
+ return readerTypeOfSplit(vars, fields, createDataSplit(Arrays.asList(fileNames)));
+ }
+
+ // Same harness as readerTypeOf over an externally built DataSplit (e.g. one
+ // whose member files carry data-file.external-paths locations).
+ private TPaimonReaderType readerTypeOfSplit(SessionVariable vars, List<DataField> fields,
+ DataSplit dataSplit) throws Exception {
+ PaimonScanNode node = new PaimonScanNode(new PlanNodeId(0),
+ new TupleDescriptor(new TupleId(0)), false, vars, ScanContext.EMPTY);
+ PaimonSource source = Mockito.mock(PaimonSource.class);
+ FileStoreTable paimonTable = Mockito.mock(FileStoreTable.class);
+ Mockito.when(paimonTable.schema()).thenReturn(new TableSchema(
+ 1, fields, 0, Collections.emptyList(), Collections.emptyList(),
+ Collections.emptyMap(), null));
+ CoreOptions options = Mockito.mock(CoreOptions.class);
+ Mockito.when(paimonTable.coreOptions()).thenReturn(options);
+ Mockito.when(options.queryAuthEnabled()).thenReturn(false);
+ PaimonExternalTable externalTable = Mockito.mock(PaimonExternalTable.class);
+ Mockito.when(source.getExternalTable()).thenReturn(externalTable);
+ Mockito.when(source.getTableLocation()).thenReturn("s3://warehouse/wh/db.db/t");
+ setField(PaimonScanNode.class, node, "backendStorageProperties",
+ ImmutableMap.of("AWS_ACCESS_KEY", "example-key", "AWS_SECRET_KEY", "example-secret"));
+ Mockito.when(externalTable.getDbName()).thenReturn("db");
+ Mockito.when(externalTable.getName()).thenReturn("t");
+ node.setSource(source);
+ setField(PaimonScanNode.class, node, "processedTable", paimonTable);
+
+ TFileRangeDesc rangeDesc = new TFileRangeDesc();
+ invokePrivateMethod(node, "setPaimonParams",
+ new Class<?>[] {TFileRangeDesc.class, PaimonSplit.class},
+ rangeDesc, new PaimonSplit(dataSplit));
+ return rangeDesc.getTableFormatParams().getPaimonParams().getReaderType();
+ }
+
+ @Test
+ public void testRustReaderSelectionRejectsIncrementalScans() throws Exception {
+ for (String mode : Arrays.asList("delta", "changelog", "diff")) {
+ RustRoutingFixture f = new RustRoutingFixture();
+ f.node.setScanParams(new TableScanParams("incr", ImmutableMap.of(
+ "incremental-between", "5,10", "incremental-between-scan-mode", mode), null));
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI, f.reader(f.split));
+ }
+ }
+
+ @Test
+ public void testRustReaderSelectionRejectsAmbientCredentialProviderModes() throws Exception {
+ for (String provider : Arrays.asList("ENV", "SYSTEM_PROPERTIES", "WEB_IDENTITY",
+ "CONTAINER", "INSTANCE_PROFILE")) {
+ RustRoutingFixture f = new RustRoutingFixture();
+ f.credentials(provider);
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI, f.reader(f.split));
+ }
+ }
+
+ @Test
+ public void testRustReaderSelectionRequiresExplicitS3Credentials() throws Exception {
+ // DEFAULT without static keys may use JVM-only credentials or anonymous fallback.
+ for (String[] shape : new String[][] {
+ {"ANONYMOUS", "s3://warehouse/wh", "PAIMON_RUST", "", ""},
+ {"DEFAULT", "s3://warehouse/wh", "PAIMON_JNI", "", ""},
+ {"DEFAULT", "s3a://warehouse/wh", "PAIMON_JNI", "example-key", ""},
+ {"DEFAULT", "s3://warehouse/wh", "PAIMON_JNI", " ", " "},
+ {"DEFAULT", "s3://warehouse/wh", "PAIMON_RUST", "example-key", "example-secret"},
+ {"", "s3://warehouse/wh", "PAIMON_JNI", "", ""},
+ {"ANONYMOUS", "oss://warehouse/wh", "PAIMON_JNI", "", ""}}) {
+ String mode = shape[0];
+ String location = shape[1];
+ TPaimonReaderType expected =
+ TPaimonReaderType.valueOf(shape[2]);
+ SessionVariable vars = new SessionVariable();
+ vars.setEnablePaimonRustReader(true);
+ vars.enableFileScannerV2 = true;
+
+ PaimonScanNode node = new PaimonScanNode(new PlanNodeId(0),
+ new TupleDescriptor(new TupleId(0)), false, vars, ScanContext.EMPTY);
+ PaimonSource source = Mockito.mock(PaimonSource.class);
+ Mockito.when(source.getTableLocation()).thenReturn(location);
+ FileStoreTable paimonTable = Mockito.mock(FileStoreTable.class);
+ PaimonExternalTable externalTable = Mockito.mock(PaimonExternalTable.class);
+ // Supply both paths so an incorrect Rust selection fails the
+ // reader-type assertion rather than unrelated schema setup.
+ Mockito.lenient().when(source.getExternalTable()).thenReturn(externalTable);
+ Mockito.lenient().when(paimonTable.schema()).thenReturn(new TableSchema(
+ 1, Collections.singletonList(new DataField(0, "id", new IntType())),
+ 0, Collections.emptyList(), Collections.emptyList(),
+ Collections.emptyMap(), null));
+ Mockito.lenient().when(externalTable.getDbName()).thenReturn("db");
+ Mockito.lenient().when(externalTable.getName()).thenReturn("t");
+ node.setSource(source);
+ setField(PaimonScanNode.class, node, "processedTable", paimonTable);
+ setField(PaimonScanNode.class, node, "backendStorageProperties",
+ ImmutableMap.of("AWS_CREDENTIALS_PROVIDER_TYPE", mode,
+ "AWS_ACCESS_KEY", shape[3], "AWS_SECRET_KEY", shape[4]));
+
+ TFileRangeDesc rangeDesc = new TFileRangeDesc();
+ invokePrivateMethod(node, "setPaimonParams",
+ new Class<?>[] {TFileRangeDesc.class, PaimonSplit.class},
+ rangeDesc, new PaimonSplit(createDataSplit("provider_ok.parquet")));
+ Assert.assertEquals("mode " + mode + " on " + location, expected,
+ rangeDesc.getTableFormatParams().getPaimonParams().getReaderType());
+ }
+ }
+
+ @Test
+ public void testRustReaderRejectsConflictingStaticCredentials() throws Exception {
+ for (Map<String, String> extra : Arrays.asList(
+ ImmutableMap.of("AWS_CREDENTIALS_PROVIDER_TYPE", "ANONYMOUS"),
+ ImmutableMap.of("AWS_ROLE_ARN", "arn:aws:iam::123456789012:role/example"),
+ ImmutableMap.of("AWS_TOKEN", "expired-example-token"),
+ ImmutableMap.of("AWS_TOKEN", " "), ImmutableMap.of("AWS_ROLE_ARN", " "))) {
+ RustRoutingFixture f = new RustRoutingFixture();
+ Map<String, String> properties = new HashMap<>();
+ properties.put("AWS_ACCESS_KEY", "example-key");
+ properties.put("AWS_SECRET_KEY", "example-secret");
+ properties.putAll(extra);
+ setField(PaimonScanNode.class, f.node, "backendStorageProperties", properties);
+ // Hadoop's Simple provider ignores these settings; Rust must not change identity.
+ Assert.assertEquals(extra.toString(), TPaimonReaderType.PAIMON_JNI, f.reader(f.split));
+ }
+ }
+
+ @Test
+ public void testRustReaderRejectsUnsupportedPersistedBranchScanModes() throws Exception {
+ for (String mode : Arrays.asList("latest", "latest-full", "full", "compacted-full", "LATEST-FULL", " default ")) {
+ RustRoutingFixture f = new RustRoutingFixture();
+ f.schema(1, new IntType(), ImmutableMap.of("branch", "audit", "scan.mode", mode), false);
+ // A branch can keep its persisted mode without a time-travel selector to strip it.
+ Assert.assertEquals(mode, TPaimonReaderType.PAIMON_JNI, f.reader(f.split));
+ }
+ }
+
+ @Test
+ public void testRustReaderSelectionGatesUnverifiedLocationSchemes() throws Exception {
+ // The pinned paimon-rust storage dispatcher (io/storage.rs) selects the
+ // FileIO parser from the table location's URI scheme, and libpaimon_c.a
+ // compiles in separate COS, OBS, GCS and Azdls parsers besides OSS and
+ // S3. Doris normalizes those object stores' credentials into the AWS_*
+ // aliases, which the BE rust bridge translates only into the s3.* /
+ // fs.oss.* key families — a cosn:// / obs:// / gs:// / abfs:// table
+ // would reach its scheme's parser without the key family it reads
+ // (fs.cosn.userinfo.*, fs.obs.*, gcs.*, azure.*) and fail the open.
+ // Those schemes must fall back to JNI before the split is encoded;
+ // the verified schemes (s3 / s3a / oss, plus credential-free hdfs and
+ // local paths) stay rust-eligible.
+ for (String[] shape : new String[][] {
+ // Unverified object-store schemes -> JNI (property translation
+ // not implemented; Doris sends these as AWS_* aliases).
+ {"cosn://bucket/wh/db.db/t", "PAIMON_JNI"},
+ {"obs://bucket/wh/db.db/t", "PAIMON_JNI"},
+ {"gs://bucket/wh/db.db/t", "PAIMON_JNI"},
+ {"abfs://bucket/wh/db.db/t", "PAIMON_JNI"},
+ {"abfss://bucket@account/wh/db.db/t", "PAIMON_JNI"},
+ {"az://bucket/wh/db.db/t", "PAIMON_JNI"},
+ {"azure://bucket/wh/db.db/t", "PAIMON_JNI"},
+ // Uppercase URI-scheme variants of otherwise verified schemes:
+ // the crate strips only a lowercase prefix, and the DataSplit's
+ // file paths carry the original casing too -> JNI.
+ {"S3://bucket/wh/db.db/t", "PAIMON_JNI"},
+ {"Hdfs://nn/wh/db.db/t", "PAIMON_JNI"},
+ // A null location cannot be verified (and cannot ship
+ // paimon_table for the rust reader) -> JNI.
+ {null, "PAIMON_JNI"},
+ // Verified schemes with implemented property translation or
+ // credential-free parsers -> rust.
+ {"s3://bucket/wh/db.db/t", "PAIMON_RUST"},
+ {"s3a://bucket/wh/db.db/t", "PAIMON_RUST"},
+ {"oss://bucket/wh/db.db/t", "PAIMON_RUST"},
+ {"hdfs://nn/wh/db.db/t", "PAIMON_RUST"},
+ {"file:///paimon/wh/db.db/t", "PAIMON_RUST"},
+ {"/paimon/wh/db.db/t", "PAIMON_RUST"}}) {
+ String location = shape[0];
+ TPaimonReaderType expected = TPaimonReaderType.valueOf(shape[1]);
+ SessionVariable vars = new SessionVariable();
+ vars.setEnablePaimonRustReader(true);
+ vars.enableFileScannerV2 = true;
+
+ PaimonScanNode node = new PaimonScanNode(new PlanNodeId(0),
+ new TupleDescriptor(new TupleId(0)), false, vars, ScanContext.EMPTY);
+ PaimonSource source = Mockito.mock(PaimonSource.class);
+ if (location != null) {
+ Mockito.when(source.getTableLocation()).thenReturn(location);
+ }
+ FileStoreTable paimonTable = Mockito.mock(FileStoreTable.class);
+ PaimonExternalTable externalTable = Mockito.mock(PaimonExternalTable.class);
+ if (expected == TPaimonReaderType.PAIMON_RUST) {
+ // Only the rust path serializes the schema and the table
+ // identity; the JNI iterations must not leave unused stubs.
+ Mockito.when(source.getExternalTable()).thenReturn(externalTable);
+ Mockito.when(paimonTable.schema()).thenReturn(new TableSchema(
+ 1, Collections.singletonList(new DataField(0, "id", new IntType())),
+ 0, Collections.emptyList(), Collections.emptyList(),
+ Collections.emptyMap(), null));
+ Mockito.when(externalTable.getDbName()).thenReturn("db");
+ Mockito.when(externalTable.getName()).thenReturn("t");
+ }
+ node.setSource(source);
+ setField(PaimonScanNode.class, node, "processedTable", paimonTable);
+ // Production-shaped COS/OBS/GCS/Azure map: static credentials
+ // normalized to the AWS_* aliases (plus the S3-compatible
+ // connection settings).
+ setField(PaimonScanNode.class, node, "backendStorageProperties", ImmutableMap.of(
+ "AWS_CREDENTIALS_PROVIDER_TYPE", "DEFAULT",
+ "AWS_ACCESS_KEY", "ak",
+ "AWS_SECRET_KEY", "sk",
+ "AWS_ENDPOINT", "http://127.0.0.1:19001",
+ "AWS_REGION", "us-east-1"));
+
+ TFileRangeDesc rangeDesc = new TFileRangeDesc();
+ invokePrivateMethod(node, "setPaimonParams",
+ new Class<?>[] {TFileRangeDesc.class, PaimonSplit.class},
+ rangeDesc, new PaimonSplit(createDataSplit("scheme_gate.parquet")));
+ Assert.assertEquals("location " + location, expected,
+ rangeDesc.getTableFormatParams().getPaimonParams().getReaderType());
+ }
+ }
+
+ @Test
+ public void testIsRustVerifiedLocationSchemeClassification() {
+ // A plain path without a URI scheme is a local-filesystem location, which
+ // reads through the crate's LocalFs parser and needs no credentials. The
+ // scheme must appear in the exact lowercase form the pinned crate
+ // consumes: it lowercases only its storage dispatch, while its path
+ // extraction strips a lowercase prefix from the original string — a
+ // S3:// or Hdfs:// warehouse must route to JNI instead (the Java stack
+ // is case-insensitive everywhere).
+ Assert.assertTrue(PaimonScanNode.isRustVerifiedLocationScheme("s3://bucket/wh"));
+ Assert.assertTrue(PaimonScanNode.isRustVerifiedLocationScheme("/local/wh"));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedLocationScheme("S3://bucket/wh"));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedLocationScheme("OSS://bucket/wh"));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedLocationScheme("Hdfs://nn/wh"));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedLocationScheme("FILE:///paimon/wh"));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedLocationScheme("cosn://bucket/wh"));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedLocationScheme("COSN://bucket/wh"));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedLocationScheme("viewfs://cluster/wh"));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedLocationScheme(null));
+ }
+
+ @Test
+ public void testRustReaderSelectionRejectsAuthenticatedHdfsBackends() throws Exception {
+ // The pinned paimon-rust HDFS parser reads only the hdfs.name-node /
+ // hdfs.enable-append keys: no kerberos, no proxy user, no hadoop HA
+ // resolution. A Kerberized HDFS catalog (or one with hadoop.username /
+ // HA nameservice config) passes the location scheme gate, so the scan
+ // would open as the BE process's ambient identity and fail the access
+ // JNI honors — those shapes must fall back to JNI; the credential-free
+ // shape stays rust-eligible.
+ for (String[][] shape : new String[][][] {
+ // Production-shaped credential-free HdfsProperties output: an
+ // fs.defaultFS, the always-written simple-auth markers and
+ // pass-through tunables, none of which carry an identity.
+ {new String[] {"fs.defaultFS", "hdfs://nn:8020"},
+ new String[] {"hdfs.security.authentication", "simple"},
+ new String[] {"ipc.client.fallback-to-simple-auth-allowed", "true"},
+ new String[] {"PAIMON_RUST"}},
+ // A null / empty map is the anonymous default the parser
+ // supports (the regression suite's plain hdfs:// table).
+ {new String[] {"PAIMON_RUST"}},
+ // Kerberos: authentication type plus principal / keytab, as
+ // HdfsProperties emits them for a Kerberized catalog.
+ {new String[] {"hadoop.security.authentication", "kerberos"},
+ new String[] {"hadoop.kerberos.principal", "hdfs/_HOST@REALM"},
+ new String[] {"hadoop.kerberos.keytab", "/etc/security/hdfs.keytab"},
+ new String[] {"PAIMON_JNI"}},
+ {new String[] {"hdfs.security.authentication", "kerberos"}, new String[] {"PAIMON_JNI"}},
+ // Proxy user: hadoop.username authenticates as a specific user
+ // the rust parser would drop.
+ {new String[] {"hadoop.username", "hive"}, new String[] {"PAIMON_JNI"}},
+ // HA nameservice resolution: dfs.nameservices / dfs.ha.* config
+ // cannot be resolved without the dfs.* channel.
+ {new String[] {"dfs.nameservices", "ns1"},
+ new String[] {"dfs.ha.namenodes.ns1", "nn1,nn2"},
+ new String[] {"PAIMON_JNI"}},
+ {new String[] {"dfs.ha.namenodes.ns1", "nn1,nn2"}, new String[] {"PAIMON_JNI"}},
+ // Simple-auth client options: valid without Kerberos, but the
+ // rust storage_hdfs parser drops every dfs./hadoop./fs. key
+ // beyond its two, and hdfs-native defaults
+ // dfs.client.use.datanode.hostname to false — behind
+ // containers/NAT it connects to the advertised IP instead of
+ // the hostname and fails while JNI succeeds.
+ {new String[] {"dfs.client.use.datanode.hostname", "true"},
+ new String[] {"PAIMON_JNI"}},
+ {new String[] {"hadoop.proxyuser.kerberos", "x"},
+ new String[] {"PAIMON_JNI"}}}) {
+ Map<String, String> backendProperties = new HashMap<>();
+ for (int i = 0; i < shape.length - 1; i++) {
+ backendProperties.put(shape[i][0], shape[i][1]);
+ }
+ TPaimonReaderType expected = TPaimonReaderType.valueOf(
+ shape[shape.length - 1][0]);
+
+ SessionVariable vars = new SessionVariable();
+ vars.setEnablePaimonRustReader(true);
+ vars.enableFileScannerV2 = true;
+
+ PaimonScanNode node = new PaimonScanNode(new PlanNodeId(0),
+ new TupleDescriptor(new TupleId(0)), false, vars, ScanContext.EMPTY);
+ PaimonSource source = Mockito.mock(PaimonSource.class);
+ Mockito.when(source.getTableLocation()).thenReturn("hdfs://nn/wh/db.db/t");
+ FileStoreTable paimonTable = Mockito.mock(FileStoreTable.class);
+ PaimonExternalTable externalTable = Mockito.mock(PaimonExternalTable.class);
+ if (expected == TPaimonReaderType.PAIMON_RUST) {
+ Mockito.when(source.getExternalTable()).thenReturn(externalTable);
+ Mockito.when(paimonTable.schema()).thenReturn(new TableSchema(
+ 1, Collections.singletonList(new DataField(0, "id", new IntType())),
+ 0, Collections.emptyList(), Collections.emptyList(),
+ Collections.emptyMap(), null));
+ Mockito.when(externalTable.getDbName()).thenReturn("db");
+ Mockito.when(externalTable.getName()).thenReturn("t");
+ }
+ node.setSource(source);
+ setField(PaimonScanNode.class, node, "processedTable", paimonTable);
+ setField(PaimonScanNode.class, node, "backendStorageProperties", backendProperties);
+
+ TFileRangeDesc rangeDesc = new TFileRangeDesc();
+ invokePrivateMethod(node, "setPaimonParams",
+ new Class<?>[] {TFileRangeDesc.class, PaimonSplit.class},
+ rangeDesc, new PaimonSplit(createDataSplit("hdfs_auth_gate.parquet")));
+ Assert.assertEquals("backend " + backendProperties, expected,
+ rangeDesc.getTableFormatParams().getPaimonParams().getReaderType());
+ }
+ }
+
+ @Test
+ public void testIsRustVerifiedHdfsBackendClassification() {
+ // The anonymous / null map is the open-tested shape; the always-written
+ // simple-auth markers and fs.defaultFS carry no identity; every
+ // auth-bearing key (type, principal, keytab, proxy user) and the HA
+ // nameservice config route to JNI.
+ Assert.assertTrue(PaimonScanNode.isRustVerifiedHdfsBackend(null));
+ Assert.assertTrue(PaimonScanNode.isRustVerifiedHdfsBackend(Collections.emptyMap()));
+ Assert.assertTrue(PaimonScanNode.isRustVerifiedHdfsBackend(ImmutableMap.of(
+ "fs.defaultFS", "hdfs://nn:8020",
+ "hadoop.security.authentication", "simple")));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedHdfsBackend(ImmutableMap.of(
+ "hadoop.security.authentication", "kerberos")));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedHdfsBackend(ImmutableMap.of(
+ "hadoop.security.authentication", "KERBEROS")));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedHdfsBackend(ImmutableMap.of(
+ "hadoop.kerberos.principal", "hdfs/_HOST@REALM")));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedHdfsBackend(ImmutableMap.of(
+ "hadoop.kerberos.keytab", "/etc/security/hdfs.keytab")));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedHdfsBackend(ImmutableMap.of(
+ "hadoop.username", "hive")));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedHdfsBackend(ImmutableMap.of(
+ "dfs.nameservices", "ns1")));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedHdfsBackend(ImmutableMap.of(
+ "dfs.ha.namenodes.ns1", "nn1,nn2")));
+ // A blank value is the unset case (values are null-filtered upstream,
+ // but a blank site-config entry must not gate an otherwise
+ // credential-free catalog).
+ Assert.assertTrue(PaimonScanNode.isRustVerifiedHdfsBackend(ImmutableMap.of(
+ "hadoop.username", " ")));
+ // The always-shipped inert keys stay rust-eligible together.
+ Assert.assertTrue(PaimonScanNode.isRustVerifiedHdfsBackend(ImmutableMap.of(
+ "fs.defaultFS", "hdfs://nn:8020",
+ "hadoop.security.authentication", "simple",
+ "hdfs.security.authentication", "simple",
+ "ipc.client.fallback-to-simple-auth-allowed", "true")));
+ // Any other dfs./hadoop./fs. client option is dropped by the rust
+ // parser and would diverge from JNI -> JNI (the canonical simple-auth
+ // case is dfs.client.use.datanode.hostname).
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedHdfsBackend(ImmutableMap.of(
+ "dfs.client.use.datanode.hostname", "true")));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedHdfsBackend(ImmutableMap.of(
+ "dfs.client.socket-timeout", "60000")));
+ Assert.assertFalse(PaimonScanNode.isRustVerifiedHdfsBackend(ImmutableMap.of(
+ "fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem")));
+ // Keys outside the hadoop/dfs/fs namespaces cannot change what the
+ // rust HDFS parser does (the BE bridge passes them through untouched
+ // and the parser ignores them).
+ Assert.assertTrue(PaimonScanNode.isRustVerifiedHdfsBackend(ImmutableMap.of(
+ "warehouse", "hdfs://nn/wh", "unrelated.key", "value")));
+ }
+
+ @Test
+ public void testRustReaderSelectionRejectsProjectedVariantColumns() throws Exception {
+ // The rust leaf feeds its Arrow arrays to the slot serdes, and
+ // DataTypeVariantV2SerDe::read_column_from_arrow unconditionally returns
+ // NOT_IMPLEMENTED_ERROR — a nested Variant (ARRAY / MAP / STRUCT
+ // containing one) reaches the same decoder through the container
+ // serdes. Projections containing Variant must route to JNI; a table
+ // whose VARIANT column is not projected by this query stays on rust.
+ Type[][] shapes = new Type[][] {
+ // Plain projected columns -> rust.
+ new Type[] {Type.INT, Type.STRING},
+ // A projected VARIANT -> JNI.
+ new Type[] {Type.INT, Type.VARIANT},
+ // Nested Variant inside every container kind -> JNI.
+ new Type[] {new ArrayType(Type.VARIANT)},
+ new Type[] {new MapType(Type.STRING, Type.VARIANT)},
+ new Type[] {new StructType(new StructField("v", Type.VARIANT))},
+ // The same containers without Variant stay rust-eligible.
+ new Type[] {new ArrayType(Type.INT)},
+ new Type[] {new MapType(Type.STRING, Type.INT)}};
+ boolean[] expectRust = new boolean[] {true, false, false, false, false, true, true};
+
+ for (int i = 0; i < shapes.length; i++) {
+ TupleDescriptor desc = new TupleDescriptor(new TupleId(0));
+ for (int j = 0; j < shapes[i].length; j++) {
+ SlotDescriptor slot = new SlotDescriptor(new SlotId(j), desc);
+ slot.setType(shapes[i][j]);
+ desc.addSlot(slot);
+ }
+
+ SessionVariable vars = new SessionVariable();
+ vars.setEnablePaimonRustReader(true);
+ vars.enableFileScannerV2 = true;
+
+ PaimonScanNode node = new PaimonScanNode(new PlanNodeId(0), desc, false, vars,
+ ScanContext.EMPTY);
+ PaimonSource source = Mockito.mock(PaimonSource.class);
+ Mockito.when(source.getTableLocation()).thenReturn("s3://bucket/wh/db.db/t");
+ FileStoreTable paimonTable = Mockito.mock(FileStoreTable.class);
+ PaimonExternalTable externalTable = Mockito.mock(PaimonExternalTable.class);
+ if (expectRust[i]) {
+ Mockito.when(source.getExternalTable()).thenReturn(externalTable);
+ Mockito.when(paimonTable.schema()).thenReturn(new TableSchema(
+ 1, Collections.singletonList(new DataField(0, "id", new IntType())),
+ 0, Collections.emptyList(), Collections.emptyList(),
+ Collections.emptyMap(), null));
+ Mockito.when(externalTable.getDbName()).thenReturn("db");
+ Mockito.when(externalTable.getName()).thenReturn("t");
+ }
+ node.setSource(source);
+ setField(PaimonScanNode.class, node, "processedTable", paimonTable);
+ setField(PaimonScanNode.class, node, "backendStorageProperties", ImmutableMap.of(
+ "AWS_CREDENTIALS_PROVIDER_TYPE", "DEFAULT",
+ "AWS_ACCESS_KEY", "ak",
+ "AWS_SECRET_KEY", "sk",
+ "AWS_ENDPOINT", "http://127.0.0.1:19001",
+ "AWS_REGION", "us-east-1"));
+
+ TFileRangeDesc rangeDesc = new TFileRangeDesc();
+ invokePrivateMethod(node, "setPaimonParams",
+ new Class<?>[] {TFileRangeDesc.class, PaimonSplit.class},
+ rangeDesc, new PaimonSplit(createDataSplit("variant_gate.parquet")));
+ Assert.assertEquals("projection " + Arrays.toString(shapes[i]),
+ expectRust[i] ? TPaimonReaderType.PAIMON_RUST : TPaimonReaderType.PAIMON_JNI,
+ rangeDesc.getTableFormatParams().getPaimonParams().getReaderType());
+ }
+ }
+
+ /**
+ * Every negative assertion starts from a complete Rust-eligible fixture, so missing
+ * schema or credentials cannot accidentally satisfy a fallback test.
+ */
+ private class RustRoutingFixture {
+ final SessionVariable vars = new SessionVariable();
+ final TupleDescriptor tuple = new TupleDescriptor(new TupleId(0));
+ final PaimonSource source = Mockito.mock(PaimonSource.class);
+ final FileStoreTable table = Mockito.mock(FileStoreTable.class);
+ final org.apache.paimon.schema.SchemaManager schemas =
+ Mockito.mock(org.apache.paimon.schema.SchemaManager.class);
+ final PaimonScanNode node;
+ final DataSplit split = createDataSplit("capabilities.parquet");
+
+ RustRoutingFixture() throws Exception {
+ vars.setEnablePaimonRustReader(true);
+ vars.enableFileScannerV2 = true;
+ node = new PaimonScanNode(new PlanNodeId(0), tuple, false, vars, ScanContext.EMPTY);
+ PaimonExternalTable external = Mockito.mock(PaimonExternalTable.class);
+ Mockito.when(source.getExternalTable()).thenReturn(external);
+ Mockito.when(external.getDbName()).thenReturn("db");
+ Mockito.when(external.getName()).thenReturn("t");
+ Mockito.when(source.getTableLocation()).thenReturn("s3://warehouse/wh/db.db/t");
+ Mockito.when(table.schemaManager()).thenReturn(schemas);
+ schema(1, new IntType(), Collections.emptyMap(), false);
+ node.setSource(source);
+ setField(PaimonScanNode.class, node, "processedTable", table);
+ credentials("DEFAULT");
+ Assert.assertEquals(TPaimonReaderType.PAIMON_RUST, reader(split));
+ }
+
+ void credentials(String provider) throws Exception {
+ setField(PaimonScanNode.class, node, "backendStorageProperties", ImmutableMap.of(
+ "AWS_CREDENTIALS_PROVIDER_TYPE", provider,
+ "AWS_ACCESS_KEY", "example-key", "AWS_SECRET_KEY", "example-secret"));
+ }
+
+ void schema(long id, org.apache.paimon.types.DataType valueType,
+ Map<String, String> options, boolean pk) throws Exception {
+ TableSchema schema = new TableSchema(id, Arrays.asList(
+ new DataField(0, "id", new IntType(false)), new DataField(1, "v", valueType)),
+ 10, Collections.emptyList(), pk ? Collections.singletonList("id") : Collections.emptyList(),
+ options, null);
+ setField(PaimonScanNode.class, node, "rustReaderCapabilities", null);
+ Mockito.when(table.schema()).thenReturn(schema);
+ Mockito.when(table.coreOptions()).thenReturn(new CoreOptions(options));
+ }
+
+ TPaimonReaderType reader(DataSplit input) throws Exception {
+ TFileRangeDesc range = new TFileRangeDesc();
+ invokePrivateMethod(node, "setPaimonParams",
+ new Class<?>[] {TFileRangeDesc.class, PaimonSplit.class}, range, new PaimonSplit(input));
+ return range.getTableFormatParams().getPaimonParams().getReaderType();
+ }
+
+ void project(Type full, Type pruned) throws Exception {
+ setField(PaimonScanNode.class, node, "rustReaderCapabilities", null);
+ SlotDescriptor slot = new SlotDescriptor(new SlotId(0), tuple);
+ slot.setColumn(new Column("v", full));
+ slot.setType(pruned);
+ tuple.addSlot(slot);
+ }
+ }
+
+ @Test
+ public void testRustPrunedNestedProjectionFallback() throws Exception {
+ StructType full = new StructType(new StructField("a", Type.INT), new StructField("b", Type.INT));
+ StructType pruned = new StructType(new StructField("b", Type.INT));
+ Type[][] shapes = {{full, pruned}, {new ArrayType(full), new ArrayType(pruned)},
+ {new MapType(Type.INT, full), new MapType(Type.INT, pruned)}};
+ for (Type[] shape : shapes) {
+ RustRoutingFixture f = new RustRoutingFixture();
+ f.project(shape[0], shape[0]);
+ Assert.assertEquals(TPaimonReaderType.PAIMON_RUST, f.reader(f.split));
+ f.tuple.getSlots().get(0).setType(shape[1]);
+ setField(PaimonScanNode.class, f.node, "rustReaderCapabilities", null);
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI, f.reader(f.split));
+ }
+ }
+
+ @Test
+ public void testRustHistoricalIntegerNarrowingFallback() throws Exception {
+ for (org.apache.paimon.types.DataType target : Arrays.asList(
+ new org.apache.paimon.types.TinyIntType(), new org.apache.paimon.types.SmallIntType(),
+ new IntType())) {
+ RustRoutingFixture f = new RustRoutingFixture();
+ TableSchema old = new TableSchema(1, Arrays.asList(new DataField(0, "id", new IntType(false)),
+ new DataField(1, "v", new org.apache.paimon.types.BigIntType())), 1,
+ Collections.emptyList(), Collections.emptyList(), Collections.emptyMap(), null);
+ Mockito.when(f.schemas.schema(1)).thenReturn(old);
+ f.schema(2, target, Collections.emptyMap(), false);
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI, f.reader(f.split));
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI, f.reader(f.split));
+ Mockito.verify(f.schemas, Mockito.times(1)).schema(1);
+ }
+ }
+
+ @Test
+ public void testRustStringHistoryToTemporalFallback() throws Exception {
+ for (org.apache.paimon.types.DataType sourceType : Arrays.asList(
+ DataTypes.STRING(), DataTypes.VARCHAR(20), DataTypes.CHAR(20))) {
+ for (org.apache.paimon.types.DataType target : Arrays.asList(DataTypes.DATE(),
+ DataTypes.TIMESTAMP(0), DataTypes.TIMESTAMP(3), DataTypes.TIMESTAMP(6),
+ DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(3))) {
+ assertHistoricalTemporalRouting(sourceType, target, TPaimonReaderType.PAIMON_JNI);
+ }
+ }
+ assertHistoricalTemporalRouting(DataTypes.DATE(), DataTypes.DATE(), TPaimonReaderType.PAIMON_RUST);
+ }
+
+ @Test
+ public void testRustIntegerHistoryToTimestampFallback() throws Exception {
+ for (int precision : new int[] {0, 3, 6}) {
+ for (org.apache.paimon.types.DataType target : Arrays.asList(DataTypes.TIMESTAMP(precision),
+ DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(precision))) {
+ for (org.apache.paimon.types.DataType sourceType : Arrays.asList(DataTypes.INT(), DataTypes.BIGINT())) {
+ assertHistoricalTemporalRouting(sourceType, target, TPaimonReaderType.PAIMON_JNI);
+ }
+ assertHistoricalTemporalRouting(target, target, TPaimonReaderType.PAIMON_RUST);
+ }
+ }
+ }
+
+ private void assertHistoricalTemporalRouting(org.apache.paimon.types.DataType sourceType,
+ org.apache.paimon.types.DataType target, TPaimonReaderType expected) throws Exception {
+ org.apache.paimon.types.DataType[][] shapes = {
+ {sourceType, target},
+ {DataTypes.ROW(new DataField(2, "old_name", sourceType)),
+ DataTypes.ROW(new DataField(2, "renamed", target))},
+ {DataTypes.ARRAY(sourceType), DataTypes.ARRAY(target)},
+ {DataTypes.MAP(sourceType, DataTypes.INT()), DataTypes.MAP(target, DataTypes.INT())},
+ {DataTypes.MAP(DataTypes.INT(), sourceType), DataTypes.MAP(DataTypes.INT(), target)}};
+ for (org.apache.paimon.types.DataType[] shape : shapes) {
+ RustRoutingFixture f = new RustRoutingFixture();
+ Mockito.when(f.schemas.schema(1)).thenReturn(new TableSchema(1, Arrays.asList(
+ new DataField(0, "id", new IntType(false)), new DataField(1, "v", shape[0])),
+ 10, Collections.emptyList(), Collections.emptyList(), Collections.emptyMap(), null));
+ f.schema(2, shape[1], Collections.emptyMap(), false);
+ Assert.assertEquals(Arrays.toString(shape), expected, f.reader(f.split));
+ }
+ }
+
+ @Test
+ public void testRustFloatingHistoryToDecimalFallback() throws Exception {
+ for (org.apache.paimon.types.DataType oldType : Arrays.asList(
+ DataTypes.FLOAT(), DataTypes.DOUBLE(), DataTypes.INT(), DataTypes.DECIMAL(3, 2))) {
+ org.apache.paimon.types.DataType decimal = DataTypes.DECIMAL(10, 2);
+ org.apache.paimon.types.DataType[][] shapes = {
+ {oldType, decimal},
+ {DataTypes.ROW(new DataField(2, "old_name", oldType)),
+ DataTypes.ROW(new DataField(2, "renamed", decimal))},
+ {DataTypes.ARRAY(oldType), DataTypes.ARRAY(decimal)},
+ {DataTypes.MAP(oldType, DataTypes.INT()), DataTypes.MAP(decimal, DataTypes.INT())},
+ {DataTypes.MAP(DataTypes.INT(), oldType), DataTypes.MAP(DataTypes.INT(), decimal)}};
+ for (org.apache.paimon.types.DataType[] shape : shapes) {
+ RustRoutingFixture f = new RustRoutingFixture();
+ Mockito.when(f.schemas.schema(1)).thenReturn(new TableSchema(1, Arrays.asList(
+ new DataField(0, "id", new IntType(false)), new DataField(1, "v", shape[0])),
+ 10, Collections.emptyList(), Collections.emptyList(), Collections.emptyMap(), null));
+ f.schema(2, shape[1], Collections.emptyMap(), false);
+ boolean floating = oldType.equals(DataTypes.FLOAT()) || oldType.equals(DataTypes.DOUBLE());
+ Assert.assertEquals(Arrays.toString(shape), floating
+ ? TPaimonReaderType.PAIMON_JNI : TPaimonReaderType.PAIMON_RUST, f.reader(f.split));
+ }
+ }
+ }
+
+ @Test
+ public void testRustUnusedAggregateNamesFallback() throws Exception {
+ for (String engine : Arrays.asList("aggregation", "partial-update")) {
+ List<String> optionKeys = new ArrayList<>(Arrays.asList(
+ "fields.default-aggregate-function", "fields.id.aggregate-function"));
+ if ("partial-update".equals(engine)) {
+ optionKeys.add("fields.seq.aggregate-function");
+ }
+ for (String key : optionKeys) {
+ for (String function : Arrays.asList("collect", "sum")) {
+ RustRoutingFixture f = new RustRoutingFixture();
+ Map<String, String> options = new HashMap<>();
+ options.put("merge-engine", engine);
+ options.put("fields.v.aggregate-function", "max");
+ options.put(key, function);
+ f.schema(1, new IntType(), options, true);
+ if ("partial-update".equals(engine)) {
+ options.put("fields.seq.sequence-group", "v");
+ Mockito.when(f.table.schema()).thenReturn(new TableSchema(1, Arrays.asList(
+ new DataField(0, "id", new IntType(false)), new DataField(1, "v", new IntType()),
+ new DataField(2, "seq", new IntType())), 2, Collections.emptyList(),
+ Collections.singletonList("id"), options, null));
+ }
+ DataFileMeta file = Mockito.spy(f.split.dataFiles().get(0));
+ Mockito.doReturn(Optional.of(0L)).when(file).deleteRowCount();
+ DataSplit split = DataSplit.builder().withPartition(BinaryRow.EMPTY_ROW)
+ .withBucket(0).withBucketPath("file:///warehouse/db/t/bucket-0")
+ .withDataFiles(Collections.singletonList(file)).build();
+ Assert.assertEquals(engine + " " + key + "=" + function,
+ TPaimonReaderType.PAIMON_JNI, f.reader(split));
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testRustNonIntegerHistoryToIntegerFallback() throws Exception {
+ for (org.apache.paimon.types.DataType sourceType : Arrays.asList(
+ DataTypes.FLOAT(), DataTypes.DOUBLE(), DataTypes.DECIMAL(10, 2), DataTypes.STRING())) {
+ for (org.apache.paimon.types.DataType targetType : Arrays.asList(
+ DataTypes.TINYINT(), DataTypes.SMALLINT(), DataTypes.INT(), DataTypes.BIGINT())) {
+ org.apache.paimon.types.DataType[][] shapes = {
+ {sourceType, targetType},
+ {DataTypes.ROW(new DataField(2, "old_name", sourceType)),
+ DataTypes.ROW(new DataField(2, "renamed", targetType))},
+ {DataTypes.ARRAY(sourceType), DataTypes.ARRAY(targetType)},
+ {DataTypes.MAP(sourceType, DataTypes.INT()), DataTypes.MAP(targetType, DataTypes.INT())},
+ {DataTypes.MAP(DataTypes.INT(), sourceType), DataTypes.MAP(DataTypes.INT(), targetType)}};
+ for (org.apache.paimon.types.DataType[] shape : shapes) {
+ RustRoutingFixture f = new RustRoutingFixture();
+ Mockito.when(f.schemas.schema(1)).thenReturn(new TableSchema(1, Arrays.asList(
+ new DataField(0, "id", new IntType(false)), new DataField(1, "v", shape[0])),
+ 10, Collections.emptyList(), Collections.emptyList(), Collections.emptyMap(), null));
+ f.schema(2, shape[1], Collections.emptyMap(), false);
+ Assert.assertEquals(Arrays.toString(shape), TPaimonReaderType.PAIMON_JNI, f.reader(f.split));
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testRustPartialUpdateDefaultAggregateAndSequenceFields() throws Exception {
+ for (org.apache.paimon.types.DataType valueType : Arrays.asList(
+ DataTypes.TINYINT(), DataTypes.DECIMAL(2, 0), DataTypes.DOUBLE())) {
+ for (boolean override : Arrays.asList(false, true)) {
+ RustRoutingFixture f = new RustRoutingFixture();
+ Map<String, String> options = new HashMap<>();
+ options.put("merge-engine", "partial-update");
+ options.put("fields.seq,seq2.sequence-group", "v");
+ options.put("fields.default-aggregate-function", "sum");
+ if (override) {
+ options.put("fields.v.aggregate-function", "max");
+ }
+ f.schema(1, valueType, options, true);
+ Mockito.when(f.table.schema()).thenReturn(new TableSchema(1, Arrays.asList(
+ new DataField(0, "id", new IntType(false)), new DataField(1, "v", valueType),
+ new DataField(2, "seq", new IntType()), new DataField(3, "seq2", new IntType())),
+ 3, Collections.emptyList(), Collections.singletonList("id"), options, null));
+ DataFileMeta file = Mockito.spy(f.split.dataFiles().get(0));
+ Mockito.doReturn(Optional.of(0L)).when(file).deleteRowCount();
+ // Legacy writers can retain an unbounded same-key run even in one file.
+ DataSplit split = DataSplit.builder().withPartition(BinaryRow.EMPTY_ROW)
+ .withBucket(0).withBucketPath("file:///warehouse/db/t/bucket-0")
+ .withDataFiles(Collections.singletonList(file)).build();
+ Assert.assertEquals(valueType + " override=" + override,
+ TPaimonReaderType.PAIMON_JNI,
+ f.reader(split));
+ }
+ }
+ }
+
+ @Test
+ public void testRustMultiFilePrimaryKeyMergeFallback() throws Exception {
+ for (String engine : Arrays.asList("deduplicate", "aggregation", "partial-update")) {
+ for (boolean primaryKey : Arrays.asList(false, true)) {
+ for (int fileCount : new int[] {1, 2, 8}) {
+ RustRoutingFixture f = new RustRoutingFixture();
+ f.schema(1, new IntType(), ImmutableMap.of("merge-engine", engine,
+ "read.batch-size", "1"), primaryKey);
+ List<DataFileMeta> files = new ArrayList<>();
+ for (int i = 0; i < fileCount; i++) {
+ DataFileMeta file = Mockito.spy(createDataSplit("run-" + i + ".parquet")
+ .dataFiles().get(0));
+ Mockito.doReturn(Optional.of(0L)).when(file).deleteRowCount();
+ files.add(file);
+ }
+ DataSplit split = DataSplit.builder().withPartition(BinaryRow.EMPTY_ROW)
+ .withBucket(0).withBucketPath("file:///warehouse/db/t/bucket-0")
+ .withDataFiles(files).build();
+ Assert.assertEquals(engine + " pk=" + primaryKey + " files=" + fileCount,
+ primaryKey && (fileCount > 1 || !"deduplicate".equals(engine)) ? TPaimonReaderType.PAIMON_JNI
+ : TPaimonReaderType.PAIMON_RUST, f.reader(split));
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testRustNestedEvolutionAndCompatibleHistory() throws Exception {
+ org.apache.paimon.types.DataType small = new org.apache.paimon.types.TinyIntType();
+ org.apache.paimon.types.DataType large = new org.apache.paimon.types.BigIntType();
+ org.apache.paimon.types.RowType oldRow = new org.apache.paimon.types.RowType(Arrays.asList(
+ new DataField(2, "a", large), new DataField(3, "b", new IntType())));
+ org.apache.paimon.types.RowType newRow = new org.apache.paimon.types.RowType(Arrays.asList(
+ new DataField(3, "b", new IntType()), new DataField(2, "renamed", small)));
+ org.apache.paimon.types.DataType[][] shapes = {
+ {large, small}, {oldRow, newRow},
+ {new org.apache.paimon.types.ArrayType(oldRow), new org.apache.paimon.types.ArrayType(newRow)},
+ {new org.apache.paimon.types.MapType(large, new IntType()),
+ new org.apache.paimon.types.MapType(small, new IntType())},
+ {new org.apache.paimon.types.MapType(new IntType(), oldRow),
+ new org.apache.paimon.types.MapType(new IntType(), newRow)}};
+ for (org.apache.paimon.types.DataType[] shape : shapes) {
+ for (boolean narrow : Arrays.asList(true, false)) {
+ RustRoutingFixture f = new RustRoutingFixture();
+ TableSchema old = new TableSchema(1, Arrays.asList(new DataField(0, "id", new IntType(false)),
+ new DataField(1, "v", shape[narrow ? 0 : 1])), 10,
+ Collections.emptyList(), Collections.emptyList(), Collections.emptyMap(), null);
+ Mockito.when(f.schemas.schema(1)).thenReturn(old);
+ f.schema(2, shape[narrow ? 1 : 0], Collections.emptyMap(), false);
+ Assert.assertEquals(narrow ? TPaimonReaderType.PAIMON_JNI : TPaimonReaderType.PAIMON_RUST,
+ f.reader(f.split));
+ }
+ }
+ RustRoutingFixture missing = new RustRoutingFixture();
+ missing.schema(2, new IntType(), Collections.emptyMap(), false);
+ Mockito.when(missing.schemas.schema(1)).thenThrow(new IllegalStateException("missing schema"));
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI, missing.reader(missing.split));
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI, missing.reader(missing.split));
+ Mockito.verify(missing.schemas, Mockito.times(1)).schema(1);
+ }
+
+ @Test
+ public void testRustDefaultAggregateAndFieldOverride() throws Exception {
+ for (Map<String, String> options : Arrays.asList(
+ ImmutableMap.of("merge-engine", "aggregation", "fields.default-aggregate-function", "sum"),
+ ImmutableMap.of("merge-engine", "aggregation", "fields.default-aggregate-function", "max"),
+ ImmutableMap.of("merge-engine", "aggregation", "fields.default-aggregate-function", "sum",
+ "fields.v.aggregate-function", "max"))) {
+ RustRoutingFixture f = new RustRoutingFixture();
+ f.schema(1, new IntType(), options, true);
+ DataFileMeta file = Mockito.spy(f.split.dataFiles().get(0));
+ Mockito.doReturn(Optional.of(0L)).when(file).deleteRowCount();
+ DataSplit split = DataSplit.builder().withPartition(BinaryRow.EMPTY_ROW)
+ .withBucket(0).withBucketPath("file:///warehouse/db/t/bucket-0")
+ .withDataFiles(Collections.singletonList(file)).build();
+ Assert.assertEquals(TPaimonReaderType.PAIMON_JNI,
+ f.reader(split));
+ }
+ }
+
+ @Test
+ public void testRustPrimaryKeyRetractsFallback() throws Exception {
+ for (String engine : Arrays.asList("deduplicate", "aggregation", "partial-update")) {
+ for (Long deletes : Arrays.asList(0L, 1L, null)) {
+ RustRoutingFixture f = new RustRoutingFixture();
+ f.schema(1, new IntType(), ImmutableMap.of("merge-engine", engine), true);
+ DataFileMeta file = Mockito.spy(f.split.dataFiles().get(0));
+ Mockito.doReturn(Optional.ofNullable(deletes)).when(file).deleteRowCount();
+ DataSplit split = DataSplit.builder().withPartition(BinaryRow.EMPTY_ROW)
+ .withBucket(0).withBucketPath("file:///warehouse/db/t/bucket-0")
+ .withDataFiles(Collections.singletonList(file)).build();
+ Assert.assertEquals(engine + " deletes=" + deletes,
+ "deduplicate".equals(engine) && deletes != null && deletes == 0L ? TPaimonReaderType.PAIMON_RUST
+ : TPaimonReaderType.PAIMON_JNI, f.reader(split));
+ }
+ }
+ }
+
+ @Test
+ public void testRustAggregateFunctionAndArithmeticFallback() throws Exception {
+ Object[][] cases = {
+ {"collect", new org.apache.paimon.types.ArrayType(new IntType()), false},
+ {"sum", new org.apache.paimon.types.TinyIntType(), false},
+ {"sum", new org.apache.paimon.types.SmallIntType(), false},
+ {"sum", new IntType(), false},
+ {"sum", new org.apache.paimon.types.BigIntType(), false},
+ {"product", new IntType(), false},
+ {"sum", new org.apache.paimon.types.DecimalType(2, 0), false},
+ {"sum", new org.apache.paimon.types.DecimalType(18, 2), false},
+ {"sum", new org.apache.paimon.types.DecimalType(19, 2), true},
+ {"product", new org.apache.paimon.types.DecimalType(20, 2), false},
+ {"sum", new org.apache.paimon.types.DoubleType(), true},
+ {"max", new IntType(), true},
+ {"max", new org.apache.paimon.types.ArrayType(new IntType()), false},
+ {"bool_and", new org.apache.paimon.types.BooleanType(), true},
+ {"last_non_null_value", new org.apache.paimon.types.ArrayType(new IntType()), true}
+ };
+ for (String engine : Arrays.asList("aggregation", "partial-update")) {
+ for (Object[] shape : cases) {
+ RustRoutingFixture f = new RustRoutingFixture();
+ f.schema(1, (org.apache.paimon.types.DataType) shape[1], ImmutableMap.of(
+ "merge-engine", engine, "fields.v.aggregate-function", (String) shape[0]), true);
+ DataFileMeta file = Mockito.spy(f.split.dataFiles().get(0));
+ Mockito.doReturn(Optional.of(0L)).when(file).deleteRowCount();
+ DataSplit split = DataSplit.builder().withPartition(BinaryRow.EMPTY_ROW)
+ .withBucket(0).withBucketPath("file:///warehouse/db/t/bucket-0")
+ .withDataFiles(Collections.singletonList(file)).build();
+ Assert.assertEquals(engine + " " + Arrays.toString(shape),
+ TPaimonReaderType.PAIMON_JNI, f.reader(split));
+ }
+ }
+ }
+
+
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/qe/runtime/PipelineExecutionTaskBuilderTest.java b/fe/fe-core/src/test/java/org/apache/doris/qe/runtime/PipelineExecutionTaskBuilderTest.java
new file mode 100644
index 00000000000..994853a04cd
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/qe/runtime/PipelineExecutionTaskBuilderTest.java
@@ -0,0 +1,96 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe.runtime;
+
+import org.apache.doris.common.profile.ExecutionProfile;
+import org.apache.doris.system.Backend;
+import org.apache.doris.thrift.TDetailedReportParams;
+import org.apache.doris.thrift.TPipelineFragmentParams;
+import org.apache.doris.thrift.TPipelineFragmentParamsList;
+import org.apache.doris.thrift.TPipelineInstanceParams;
+import org.apache.doris.thrift.TQueryProfile;
+import org.apache.doris.thrift.TRuntimeProfileNode;
+import org.apache.doris.thrift.TRuntimeProfileTree;
+import org.apache.doris.thrift.TUniqueId;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Map;
+
+public class PipelineExecutionTaskBuilderTest {
+ @Test
+ public void buildSingleFragmentPipelineTaskRegistersBackendWithExecutionProfile() {
+ TUniqueId queryId = new TUniqueId(1, 2);
+ ExecutionProfile executionProfile = new ExecutionProfile(queryId, Lists.newArrayList(0));
+ Backend backend = new Backend(1L, "127.0.0.1", 9050);
+
+ Map<Integer, SingleFragmentPipelineTask> tasks =
+ PipelineExecutionTaskBuilder.buildSingleFragmentPipelineTask(
+ executionProfile, backend, createFragmentParamsList(0));
+
+ Assert.assertEquals(1, tasks.size());
+ Assert.assertTrue(tasks.containsKey(0));
+
+ executionProfile.updateProfile(createQueryProfile(queryId, 0), backend.getHeartbeatAddress(), true);
+ Assert.assertTrue(executionProfile.isCompleted());
+ }
+
+ private static TPipelineFragmentParamsList createFragmentParamsList(int fragmentId) {
+ TPipelineInstanceParams instanceParams = new TPipelineInstanceParams();
+ instanceParams.setFragmentInstanceId(new TUniqueId(3, 4));
+
+ TPipelineFragmentParams fragmentParams = new TPipelineFragmentParams();
+ fragmentParams.setFragmentId(fragmentId);
+ fragmentParams.setLocalParams(Lists.newArrayList(instanceParams));
+
+ TPipelineFragmentParamsList paramsList = new TPipelineFragmentParamsList();
+ paramsList.setParamsList(Lists.newArrayList(fragmentParams));
+ return paramsList;
+ }
+
+ private static TQueryProfile createQueryProfile(TUniqueId queryId, int fragmentId) {
+ TQueryProfile queryProfile = new TQueryProfile();
+ queryProfile.setQueryId(queryId);
+ queryProfile.putToFragmentIdToProfile(fragmentId, Lists.newArrayList(createReportParam()));
+ return queryProfile;
+ }
+
+ private static TDetailedReportParams createReportParam() {
+ TRuntimeProfileNode node = new TRuntimeProfileNode();
+ node.setName("PipelineProfile");
+ node.setNumChildren(0);
+ node.setCounters(Lists.newArrayList());
+ node.setMetadata(0);
+ node.setIndent(false);
+ node.setInfoStrings(Maps.newHashMap());
+ node.setInfoStringsDisplayOrder(Lists.newArrayList());
+ node.setChildCountersMap(Maps.newHashMap());
+ node.setTimestamp(0);
+
+ TRuntimeProfileTree tree = new TRuntimeProfileTree();
+ tree.setNodes(Lists.newArrayList(node));
+
+ TDetailedReportParams params = new TDetailedReportParams();
+ params.setProfile(tree);
+ params.setIsFragmentLevel(false);
+ return params;
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/system/BackendPaimonRustCapabilityTest.java b/fe/fe-core/src/test/java/org/apache/doris/system/BackendPaimonRustCapabilityTest.java
new file mode 100644
index 00000000000..9b085b3c7e4
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/system/BackendPaimonRustCapabilityTest.java
@@ -0,0 +1,70 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.system;
+
+import org.apache.doris.persist.gson.GsonUtils;
+import org.apache.doris.resource.Tag;
+import org.apache.doris.thrift.TBackendInfo;
+
+import com.google.gson.JsonObject;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class BackendPaimonRustCapabilityTest {
+ @Test
+ public void testCapabilitySurvivesHeartbeatReplayAndBackendImage() {
+ Backend backend = new Backend(1, "127.0.0.1", 9050);
+ Assert.assertFalse(backend.isPaimonRustReaderSupported());
+ BackendHbResponse response = heartbeat();
+ response.setPaimonRustReaderSupported(true);
+ BackendHbResponse replay = GsonUtils.GSON.fromJson(GsonUtils.GSON.toJson(response), BackendHbResponse.class);
+ Assert.assertTrue(replay.isPaimonRustReaderSupported());
+ Assert.assertTrue(backend.handleHbResponse(replay, true));
+ Assert.assertTrue(backend.isPaimonRustReaderSupported());
+ Backend restored = GsonUtils.GSON.fromJson(GsonUtils.GSON.toJson(backend), Backend.class);
+ Assert.assertTrue(restored.isPaimonRustReaderSupported());
+ }
+
+ @Test
+ public void testLegacyHeartbeatClearsPreviouslyAdvertisedSupport() {
+ Backend backend = new Backend(1, "127.0.0.1", 9050);
+ BackendHbResponse response = heartbeat();
+ response.setPaimonRustReaderSupported(true);
+ backend.handleHbResponse(response, false);
+ Assert.assertTrue(backend.isPaimonRustReaderSupported());
+
+ TBackendInfo legacy = new TBackendInfo();
+ Assert.assertFalse(legacy.isSetSupportsPaimonRustReader());
+ Assert.assertFalse(legacy.isSupportsPaimonRustReader());
+ BackendHbResponse legacyResponse = heartbeat();
+ Assert.assertFalse(legacyResponse.isPaimonRustReaderSupported());
+ Assert.assertTrue(backend.handleHbResponse(legacyResponse, false));
+ Assert.assertFalse(backend.isPaimonRustReaderSupported());
+
+ JsonObject legacyJson = GsonUtils.GSON.toJsonTree(heartbeat()).getAsJsonObject();
+ legacyJson.remove("supportsPaimonRustReader");
+ BackendHbResponse replay = GsonUtils.GSON.fromJson(legacyJson, BackendHbResponse.class);
+ Assert.assertFalse(replay.isPaimonRustReaderSupported());
+ Assert.assertFalse(GsonUtils.GSON.fromJson("{}", Backend.class).isPaimonRustReaderSupported());
+ }
+
+ private BackendHbResponse heartbeat() {
+ return new BackendHbResponse(1, 9060, 8040, 8060, 1000, 1000, "test-version", Tag.VALUE_MIX,
+ 0, 0, false, 8070);
+ }
+}
diff --git a/gensrc/thrift/HeartbeatService.thrift b/gensrc/thrift/HeartbeatService.thrift
index 3fde7fdf2df..8608853b29f 100644
--- a/gensrc/thrift/HeartbeatService.thrift
+++ b/gensrc/thrift/HeartbeatService.thrift
@@ -63,6 +63,7 @@ struct TBackendInfo {
8: optional bool is_shutdown
9: optional Types.TPort arrow_flight_sql_port
10: optional i64 be_mem // The physical memory available for use by BE.
+ 11: optional bool supports_paimon_rust_reader
// For cloud
1000: optional i64 fragment_executing_count
1001: optional i64 fragment_last_active_time
diff --git a/gensrc/thrift/PaloInternalService.thrift b/gensrc/thrift/PaloInternalService.thrift
index 5aa105c3608..2044f058052 100644
--- a/gensrc/thrift/PaloInternalService.thrift
+++ b/gensrc/thrift/PaloInternalService.thrift
@@ -510,6 +510,9 @@ struct TQueryOptions {
// Fall back to RE2 when Hyperscan cannot compile a regular expression.
228: optional bool enable_hyperscan_fallback = true;
+ // Use paimon-rust to read Paimon splits on BE. Higher priority than
+ // enable_paimon_cpp_reader.
+ 229: optional bool enable_paimon_rust_reader = false;
// Master uses 226 for this option; branch-4.2 already uses 226-228, so the id was moved.
229: optional bool enable_prune_nested_column = false;
diff --git a/gensrc/thrift/PlanNodes.thrift b/gensrc/thrift/PlanNodes.thrift
index aaf057b6327..3e626e6f0ad 100644
--- a/gensrc/thrift/PlanNodes.thrift
+++ b/gensrc/thrift/PlanNodes.thrift
@@ -357,6 +357,7 @@ enum TPaimonReaderType {
PAIMON_JNI = 1,
// Deprecated wire value kept during rolling upgrades. New plans never emit it.
PAIMON_CPP = 2,
+ PAIMON_RUST = 3,
}
struct TPaimonFileDesc {
@@ -378,6 +379,12 @@ struct TPaimonFileDesc {
16: optional i64 schema_id; // for schema change.
// Reader implementation for logical paimon split. Native file split uses range format type.
17: optional TPaimonReaderType reader_type;
+ // paimon-rust: TableSchema serialized via JsonSerdeUtil.toJson(table.schema()).
+ // Consumed by paimon_table_from_schema_json on BE; skips catalog/warehouse resolution.
+ 18: optional string paimon_table_schema_json
+ // paimon-rust: non-default branch name. Unset means main branch (matches
+ // upstream paimon commit 742da63: null-if-DEFAULT_MAIN_BRANCH).
+ 19: optional string paimon_branch
}
struct TTrinoConnectorFileDesc {
diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_rust_reader_cast_predicates.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_rust_reader_cast_predicates.groovy
new file mode 100644
index 00000000000..5daac9772fa
--- /dev/null
+++ b/regression-test/suites/external_table_p0/paimon/test_paimon_rust_reader_cast_predicates.groovy
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// CASTed operands through the paimon rust reader. The rust predicate converter
+// must NOT strip a cast from a column before pushing: for a DECIMAL(10,2)
+// column holding 1.24, `CAST(amount AS DECIMAL(10,1)) = 1.2` must retain the
+// row, while the unwrapped `amount = 1.2` prunes it — and rows pruned by the
+// pushed filter cannot be recovered by the Doris residual. The FE converter
+// already rejects casted operands (PaimonPredicateConverterTest's
+// rejectDecimalScaleCast / rejectStringToIntegerCast), so JNI never pushes
+// them; the rust reader must match. Cast-induced NULL semantics
+// (`CAST(s AS INT) IS NULL`) are covered the same way.
+//
+// Both differential legs run with force_jni_scanner=true: these parquet
+// append tables convert to raw native splits, which getSplits() would
+// otherwise prefer — both legs would silently use the native reader and
+// never reach the JNI / rust converters. The actual reader path is verified
+// per leg through the query profile (the rust reader's PaimonRustReader
+// timer group).
+import org.apache.doris.regression.action.ProfileAction
+
+suite("test_paimon_rust_reader_cast_predicates", "p0,external,paimon") {
+ String enabled = context.config.otherConfigs.get("enablePaimonTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("disabled paimon test")
+ return
+ }
+
+ String catalogName = "test_paimon_rust_cast"
+ String dbName = "test_paimon_rust_cast_db"
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
+
+ // Table is created via Spark because Doris does not support Paimon DDL.
+ // t_decimal_cast mirrors the FE rejectDecimalScaleCast shape: a
+ // DECIMAL(10,2) column whose scale-reducing cast must not be pushed.
+ // t_cast_null covers cast-related NULL semantics on a string column.
+ spark_paimon_multi """
+ CREATE DATABASE IF NOT EXISTS paimon.${dbName};
+ DROP TABLE IF EXISTS paimon.${dbName}.t_decimal_cast;
+ CREATE TABLE paimon.${dbName}.t_decimal_cast (
+ id INT, amount DECIMAL(10, 2)
+ ) USING paimon;
+ INSERT INTO paimon.${dbName}.t_decimal_cast VALUES
+ (1, 1.24), (2, 1.20), (3, 1.30), (4, NULL);
+
+ DROP TABLE IF EXISTS paimon.${dbName}.t_cast_null;
+ CREATE TABLE paimon.${dbName}.t_cast_null (
+ id INT, s VARCHAR(10)
+ ) USING paimon;
+ INSERT INTO paimon.${dbName}.t_cast_null VALUES (1, '5'), (2, NULL);
+
+ DROP TABLE IF EXISTS paimon.${dbName}.t_partition_constants;
+ CREATE TABLE paimon.${dbName}.t_partition_constants (id INT, p VARCHAR(3))
+ USING paimon PARTITIONED BY (p) TBLPROPERTIES ('bucket'='-1');
+ INSERT INTO paimon.${dbName}.t_partition_constants VALUES (1,'abc'), (2,'abc'), (3,NULL);
+ """
+
+ // The s3.region property is required: paimon-rust's S3 client rejects a
+ // missing region, while the JNI reader falls back to the SDK default.
+ sql """drop catalog if exists ${catalogName}"""
+ sql """
+ CREATE CATALOG ${catalogName} PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'filesystem',
+ 'warehouse' = 's3://warehouse/wh',
+ 's3.endpoint' = 'http://${externalEnvIp}:${minioPort}',
+ 's3.access_key' = 'admin',
+ 's3.secret_key' = 'password',
+ 's3.region' = 'us-east-1',
+ 'use_path_style' = 'true'
+ );
+ """
+
+ // Capture the settings this suite overrides so finally can restore them.
+ def originalForceJni = sql("select @@force_jni_scanner")[0][0]
+ def originalEnableProfile = sql("select @@enable_profile")[0][0]
+ def originalTruncate = sql("select @@truncate_char_or_varchar_columns")[0][0]
+
+ try {
+ sql """switch ${catalogName}"""
+ sql """use ${dbName}"""
+ sql """set enable_file_scanner_v2=true"""
+ // These are parquet append tables whose DataSplits convert to raw
+ // native splits; force the logical (JNI / rust) reader path so the
+ // differential actually exercises both converters.
+ sql """set force_jni_scanner=true"""
+ sql """set enable_profile=true"""
+ sql "set truncate_char_or_varchar_columns=true"
+
+ // Use configured HTTP credentials and wait for the Rust group as well as profile
+ // completion: the scanner group can appear before its child reader timers arrive.
+ def profiles = new ProfileAction(context)
+ def profileTextOf = { String query, boolean rustExpected ->
+ sql(query)
+ def queryId = sql("select last_query_id()")[0][0].toString()
+ def required = rustExpected ? ["FileScannerV2", "PaimonRustReader"] : ["FileScannerV2"]
+ profiles.getProfile(queryId, required)
+ }
+
+ def testQueries = [
+ // Scale-reducing cast: 1.24 rounds to 1.2 under the cast, so
+ // the 1.24 row (id 1) must be retained — exactly the row a
+ // wrongly pushed `amount = 1.2` would drop forever.
+ """select id from t_decimal_cast where cast(amount as decimal(10,1)) = 1.2 order by id""",
+ """select id from t_decimal_cast where cast(amount as decimal(10,1)) = 1.3 order by id""",
+ // Plain (uncast) decimal equality still pushes and matches.
+ """select id from t_decimal_cast where amount = 1.24 order by id""",
+ // CAST IS NULL / IS NOT NULL on the casted column keep NULL
+ // semantics: only the NULL row (id 4) is null under the cast.
+ """select id from t_decimal_cast where cast(amount as decimal(10,1)) is null order by id""",
+ """select id from t_decimal_cast where cast(amount as decimal(10,1)) is not null order by id""",
+ // Cast-induced NULL: CAST('5' AS INT) is 5, not NULL.
+ """select id from t_cast_null where cast(s as int) is null order by id""",
+ """select id from t_cast_null where cast(s as int) = 5 order by id""",
+ // IN / NOT IN with a casted list value. FE folds the
+ // list-value cast at plan time (even with
+ // debug_skip_fold_constant) and aligns the column with a
+ // column-side cast instead, so both readers correctly keep
+ // everything in the residual here. The un-folded shape — the
+ // cast still inside the IN list — is covered by the
+ // PaimonRustPredicateConverterTest unit tests, where the
+ // converter rejects any casted IN-list child (FE's
+ // doInPredicate parity).
+ """select id from t_decimal_cast where amount in (cast(1.24 as decimal(10,1)), 1.30) order by id""",
+ """select id from t_decimal_cast where amount not in (cast(1.24 as decimal(10,1))) order by id"""
+ ]
+ def expectedResults = [
+ [[1], [2]],
+ [[3]],
+ [[1]],
+ [[4]],
+ [[1], [2], [3]],
+ [[2]],
+ [[1]],
+ [[2], [3]],
+ [[1], [3]]
+ ]
+ // Partition keys are supplied as nullable constants outside the Arrow batch.
+ testQueries.add("select id,p from t_partition_constants order by id")
+ expectedResults.add([[1,"abc"],[2,"abc"],[3,null]])
+ // Representative scale-cast query reused for the reader-path checks.
+ String scaleCastQuery = testQueries[0]
+
+ sql """set enable_paimon_rust_reader=false"""
+ def jniResults = testQueries.collect { query -> sql(query) }
+ // The JNI leg must ride the logical-split JNI reader: the profile of
+ // the representative query must not contain the rust reader's timer.
+ def jniProfile = profileTextOf(scaleCastQuery, false)
+ assertFalse(jniProfile.contains("PaimonRustReader"), "JNI leg must not use the rust reader")
+
+ sql """set enable_paimon_rust_reader=true"""
+ def rustResults = testQueries.collect { query -> sql(query) }
+ // The rust leg must actually run the rust reader: its profile carries
+ // the PaimonRustReader timer group, which only the rust reader creates.
+ def rustProfile = profileTextOf(scaleCastQuery, true)
+ assertTrue(rustProfile.contains("PaimonRustReader"),
+ "rust leg must use the rust reader (profile timer missing)")
+
+ assertTrue(profileTextOf(testQueries.last(), true).contains("PaimonRustReader"),
+ "partition constants must exercise the Rust materialization path")
+ for (int i = 0; i < testQueries.size(); i++) {
+ // The rust reader must agree with the JNI reader (which, like FE,
+ // never pushes a casted operand) on every form.
+ assertEquals(jniResults[i].toString(), rustResults[i].toString())
+ // And both must be right, not just mutually consistent: id 1 in
+ // the first query is exactly the row a wrongly pushed
+ // `amount = 1.2` would drop.
+ assertEquals(expectedResults[i].toString(), rustResults[i].toString())
+ }
+ } finally {
+ sql """set enable_paimon_rust_reader=false"""
+ sql """set force_jni_scanner=${originalForceJni}"""
+ sql """set enable_profile=${originalEnableProfile}"""
+ sql "set truncate_char_or_varchar_columns=${originalTruncate}"
+ sql """drop catalog if exists ${catalogName}"""
+ }
+}
diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_rust_reader_compatibility.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_rust_reader_compatibility.groovy
new file mode 100644
index 00000000000..cbfaa54525b
--- /dev/null
+++ b/regression-test/suites/external_table_p0/paimon/test_paimon_rust_reader_compatibility.groovy
@@ -0,0 +1,308 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.action.ProfileAction
+
+suite("test_paimon_rust_reader_compatibility", "p0,external,paimon") {
+ if (!"true".equalsIgnoreCase(context.config.otherConfigs.get("enablePaimonTest"))) {
+ return
+ }
+ def endpoint = context.config.otherConfigs.get("externalEnvIp")
+ def port = context.config.otherConfigs.get("iceberg_minio_port")
+ def catalog = "test_paimon_rust_compatibility"
+ def database = "test_paimon_rust_compatibility_db"
+ def settings = ["enable_paimon_rust_reader", "force_jni_scanner",
+ "enable_file_scanner_v2", "enable_profile", "enable_prune_nested_column",
+ "enable_push_down_no_group_agg", "time_zone"]
+ def saved = settings.collectEntries { [(it): sql("select @@${it}")[0][0]] }
+ sql "DROP CATALOG IF EXISTS ${catalog}"
+ sql """CREATE CATALOG ${catalog} PROPERTIES (
+ 'type'='paimon', 'paimon.catalog.type'='filesystem', 'fs.s3.support'='true',
+ 'warehouse'='s3://warehouse/wh', 's3.endpoint'='http://${endpoint}:${port}',
+ 's3.access_key'='admin', 's3.secret_key'='password',
+ 's3.region'='us-east-1', 'use_path_style'='true')"""
+ sql "SWITCH ${catalog}"
+ sql "DROP DATABASE IF EXISTS ${database} FORCE"
+ sql "CREATE DATABASE ${database}"
+ sql "USE ${database}"
+ try {
+ sql "set enable_profile=true"
+ sql "set time_zone='+00:00'"
+ sql "set enable_prune_nested_column=true"
+ sql "set enable_file_scanner_v2=true"
+ sql "set force_jni_scanner=true"
+ def profiles = new ProfileAction(context)
+ def check = { String query, List expected, boolean rustExpected ->
+ sql "set enable_paimon_rust_reader=false"
+ def baseline = sql(query)
+ assertEquals(expected.toString(), baseline.toString())
+ sql "set enable_paimon_rust_reader=true"
+ def actual = sql(query)
+ def queryId = sql("select last_query_id()")[0][0].toString()
+ // A scanner can be published before its children; absence is meaningful only at completion.
+ def required = ["FileScannerV2", "Profile Completion State: COMPLETE"]
+ if (rustExpected) {
+ required.add("PaimonRustReader")
+ }
+ def profile = profiles.getProfile(queryId, required)
+ assertEquals(expected.toString(), actual.toString())
+ assertEquals(rustExpected, profile.contains("PaimonRustReader"), query)
+ }
+ def createPk = { String name, String valueColumns, String extra ->
+ // Separate commits must remain separate files so the reader performs the merge.
+ sql """CREATE TABLE ${name} (id INT NOT NULL, ${valueColumns}) ENGINE=paimon
+ PROPERTIES ('primary-key'='id', 'bucket'='1', 'write-only'='true',
+ 'deletion-vectors.enabled'='false' ${extra})"""
+ }
+
+ createPk("credential_values", "v INT", "")
+ sql "INSERT INTO credential_values VALUES (1,11)"
+ check("select id,v from credential_values", [[1,11]], true)
+ ["'s3.credentials_provider_type'='ANONYMOUS'",
+ "'s3.role_arn'='arn:aws:iam::123456789012:role/example'",
+ "'s3.session_token'='expired-example-token'"].eachWithIndex { extra, index ->
+ def crossedCatalog = "${catalog}_credentials_${index}"
+ sql "DROP CATALOG IF EXISTS ${crossedCatalog}"
+ try {
+ // Select S3 explicitly: endpoint guessing would choose MinIO and ignore S3 auth modes.
+ // Hadoop chooses static keys first; conflicting settings must retain that identity.
+ sql """CREATE CATALOG ${crossedCatalog} PROPERTIES (
+ 'type'='paimon', 'paimon.catalog.type'='filesystem', 'fs.s3.support'='true',
+ 'warehouse'='s3://warehouse/wh', 's3.endpoint'='http://${endpoint}:${port}',
+ 's3.access_key'='admin', 's3.secret_key'='password',
+ 's3.region'='us-east-1', 'use_path_style'='true', ${extra})"""
+ check("select id,v from ${crossedCatalog}.${database}.credential_values", [[1,11]], false)
+ } finally {
+ sql "DROP CATALOG IF EXISTS ${crossedCatalog}"
+ }
+ }
+
+ createPk("branch_scan_mode", "v INT", ", 'scan.mode'='latest-full'")
+ sql "INSERT INTO branch_scan_mode VALUES (1,11)"
+ spark_paimon """CALL paimon.sys.create_tag(table => '${database}.branch_scan_mode', tag => 'audit_tag')"""
+ spark_paimon """CALL paimon.sys.create_branch(table => '${database}.branch_scan_mode',
+ branch => 'audit', tag => 'audit_tag')"""
+ // A branch preserves the persisted scan mode instead of deriving one from a snapshot selector.
+ check("select id,v from branch_scan_mode@branch(audit)", [[1,11]], false)
+
+ sql """CREATE TABLE footer_aggregates (v INT NULL) ENGINE=paimon
+ PROPERTIES ('bucket'='-1', 'file.format'='parquet')"""
+ sql "INSERT INTO footer_aggregates VALUES (NULL), (-3), (7), (NULL), (2), (5)"
+ def aggregateQueries = [
+ ["select count(v) from footer_aggregates", [[4]], "COUNT"],
+ ["select min(v), max(v) from footer_aggregates", [[-3,7]], "MINMAX"]]
+ def rawRows = { String profile ->
+ def values = (profile =~ /RawRowsRead: ([0-9]+)/).collect { it[1] as long }
+ assertFalse(values.isEmpty(), "missing Parquet RawRowsRead counter")
+ values
+ }
+ // Rust only replaces logical JNI splits. Native raw-file splits must retain footer
+ // aggregation with Rust enabled; result equality alone would hide a full-scan regression.
+ sql "set force_jni_scanner=false"
+ aggregateQueries.each { entry ->
+ [false, true].each { rustEnabled ->
+ sql "set enable_paimon_rust_reader=${rustEnabled}"
+ [false, true].each { pushdown ->
+ sql "set enable_push_down_no_group_agg=${pushdown}"
+ explain {
+ sql(entry[0])
+ contains "pushdown agg=${pushdown ? entry[2] : 'NONE'}"
+ contains "paimonNativeReadSplits="
+ }
+ assertEquals(entry[1].toString(), sql(entry[0]).toString())
+ def queryId = sql("select last_query_id()")[0][0].toString()
+ def profile = profiles.getProfile(queryId, ["FileScannerV2", "ParquetReader", "Profile Completion State: COMPLETE"])
+ assertFalse(profile.contains("PaimonRustReader"))
+ def rows = rawRows(profile)
+ if (pushdown) {
+ assertTrue(rows.every { it == 0 }, "footer aggregate must not read data rows")
+ } else {
+ assertTrue(rows.any { it > 0 }, "full-scan control must read data rows")
+ }
+ }
+ }
+ }
+ sql "set enable_push_down_no_group_agg=true"
+ sql "set force_jni_scanner=true"
+ // Forced logical splits still return real rows to the upper aggregate in both readers.
+ aggregateQueries.each { entry -> check(entry[0], entry[1], true) }
+
+ ["DATE", "DATETIME(3)"].eachWithIndex { target, index ->
+ def name = "string_to_temporal_${index}"
+ sql """CREATE TABLE ${name} (id INT, v STRING) ENGINE=paimon
+ PROPERTIES ('bucket'='-1', 'file.format'='parquet')"""
+ sql "INSERT INTO ${name} VALUES (1,'42'), (2,NULL)"
+ check("select id,v from ${name} order by id", [[1,"42"],[2,null]], true)
+ sql "ALTER TABLE ${name} MODIFY COLUMN v ${target} NULL"
+ // The historical string is an epoch count in Java, not an Arrow calendar literal.
+ check("select id,cast(v as string) from ${name} order by id",
+ [[1,index == 0 ? "1970-02-12" : "1970-01-01 00:00:00.042"],[2,null]], false)
+ }
+ ["INT", "BIGINT"].each { sourceType ->
+ [0, 3, 6].each { precision ->
+ def name = "integer_to_timestamp_${sourceType.toLowerCase()}_${precision}"
+ sql """CREATE TABLE ${name} (id INT, v ${sourceType}) ENGINE=paimon
+ PROPERTIES ('bucket'='-1', 'file.format'='parquet')"""
+ sql "INSERT INTO ${name} VALUES (1,1700000000), (2,NULL)"
+ sql "ALTER TABLE ${name} MODIFY COLUMN v DATETIME(${precision}) NULL"
+ // All target precisions must interpret the source number as epoch seconds.
+ check("select id,date_format(v,'%Y-%m-%d %H:%i:%s') from ${name} order by id",
+ [[1,"2023-11-14 22:13:20"],[2,null]], false)
+ }
+ }
+
+ createPk("nested_values", "v STRUCT<a:INT,b:INT>, arr ARRAY<STRUCT<a:INT,b:INT>>, "
+ + "m MAP<INT,STRUCT<a:INT,b:INT>>", "")
+ // Bracket array literals accept constants only; nested expressions need the array constructor.
+ sql """INSERT INTO nested_values VALUES (1, named_struct('a',11,'b',22),
+ array(named_struct('a',33,'b',44)), map(1,named_struct('a',55,'b',66)))"""
+ check("select id from nested_values", [[1]], true)
+ // Distinct siblings expose ordinal decoding of a pruned second child.
+ check("select struct_element(v,'b') from nested_values", [[22]], false)
+ check("select struct_element(arr[1],'b') from nested_values", [[44]], false)
+ check("select struct_element(m[1],'b') from nested_values", [[66]], false)
+
+ createPk("narrow_values", "v BIGINT NULL", "")
+ sql "INSERT INTO narrow_values VALUES (1,383), (2,-129), (3,NULL)"
+ check("select id,v from narrow_values order by id", [[1,383],[2,-129],[3,null]], true)
+ sql "ALTER TABLE narrow_values MODIFY COLUMN v TINYINT NULL"
+ check("select id,v from narrow_values order by id", [[1,127],[2,127],[3,null]], false)
+ // A narrowing predicate must also be evaluated after the Java-compatible cast.
+ check("select id,v from narrow_values where v=127 order by id", [[1,127],[2,127]], false)
+
+ def integerCases = [
+ ["TINYINT", "127", "1", "-128", "64", "2", "-128"],
+ ["SMALLINT", "32767", "1", "-32768", "16384", "2", "-32768"],
+ ["INT", "2147483647", "1", "-2147483648", "1073741824", "2", "-2147483648"],
+ ["BIGINT", "9223372036854775807", "1", "-9223372036854775808",
+ "4611686018427387904", "2", "-9223372036854775808"]]
+ integerCases.eachWithIndex { c, index ->
+ ["sum", "product"].each { function ->
+ def name = "integer_${function}_${index}"
+ int offset = function == "sum" ? 1 : 4
+ createPk(name, "v ${c[0]}", ", 'merge-engine'='aggregation', "
+ + "'fields.v.aggregate-function'='${function}'")
+ sql "INSERT INTO ${name} VALUES (1,${c[offset]})"
+ sql "INSERT INTO ${name} VALUES (1,${c[offset+1]})"
+ check("select cast(v as string) from ${name}", [[c[offset+2]]], false)
+ }
+ }
+ createPk("decimal_sum", "v DECIMAL(2,0)", ", 'merge-engine'='aggregation', "
+ + "'fields.v.aggregate-function'='sum'")
+ [99, 1, -1].each { sql "INSERT INTO decimal_sum VALUES (1,${it})" }
+ check("select cast(v as string) from decimal_sum", [["99"]], false)
+
+ createPk("collected_values", "v ARRAY<INT>", ", 'merge-engine'='aggregation', "
+ + "'fields.v.aggregate-function'='collect'")
+ sql "INSERT INTO collected_values VALUES (1,[11])"
+ sql "INSERT INTO collected_values VALUES (1,[22])"
+ check("select element_at(array_sort(v),1), element_at(array_sort(v),2) from collected_values",
+ [[11,22]], false)
+
+ // Safe single-file merge engines remain positive controls for the Rust route.
+ ["deduplicate", "first-row"].eachWithIndex { engine, index ->
+ def name = "single_file_merge_${index}"
+ createPk(name, "v INT", ", 'merge-engine'='${engine}'")
+ sql "INSERT INTO ${name} VALUES (1,11)"
+ if (engine == "first-row") {
+ // Paimon hides first-row level-0 files until compaction has deduplicated them.
+ spark_paimon """CALL paimon.sys.compact(
+ table => '${database}.${name}', compact_strategy => 'full')"""
+ }
+ check("select id,v from ${name} order by id", [[1,11]], true)
+ }
+
+ createPk("supported_sum", "v DOUBLE", ", 'merge-engine'='aggregation', "
+ + "'fields.v.aggregate-function'='sum'")
+ sql "INSERT INTO supported_sum VALUES (1,10.0)"
+ // A single physical file can contain multiple versions from older Rust writers.
+ check("select v from supported_sum", [[10.0]], false)
+ sql "INSERT INTO supported_sum VALUES (1,3.0)"
+ check("select v from supported_sum", [[13.0]], false)
+
+ [false, true].each { decimal ->
+ [false, true].each { override ->
+ def name = "partial_default_${decimal}_${override}"
+ def type = decimal ? "DECIMAL(2,0)" : "TINYINT"
+ def first = decimal ? 99 : 127
+ // Keep the trailing operator so Groovy continues the assignment across lines.
+ def extra = ", 'merge-engine'='partial-update', 'fields.seq.sequence-group'='v', " +
+ "'fields.default-aggregate-function'='sum'"
+ if (override) {
+ extra += ", 'fields.v.aggregate-function'='max'"
+ }
+ createPk(name, "v ${type}, seq INT", extra)
+ sql "INSERT INTO ${name} VALUES (1,${first},1)"
+ // The reader must reject these engines even before a second file is written.
+ check("select cast(v as string) from ${name}", [[first.toString()]], false)
+ sql "INSERT INTO ${name} VALUES (1,1,2)"
+ if (decimal) {
+ sql "INSERT INTO ${name} VALUES (1,-1,3)"
+ }
+ def expected = decimal ? "99" : override ? "127" : "-128"
+ check("select cast(v as string) from ${name}", [[expected]], false)
+ }
+ }
+
+ ["aggregation", "partial-update"].eachWithIndex { engine, index ->
+ def name = "unused_default_${index}"
+ def extra = ", 'merge-engine'='${engine}', 'fields.default-aggregate-function'='collect', " +
+ "'fields.v.aggregate-function'='max'"
+ boolean partial = engine == "partial-update"
+ if (partial) {
+ extra += ", 'fields.seq.sequence-group'='v'"
+ }
+ createPk(name, partial ? "v INT, seq INT" : "v INT", extra)
+ sql "INSERT INTO ${name} VALUES (1,7${partial ? ',1' : ''})"
+ // An unused Java SPI default still fails Rust's global function-name validation.
+ check("select v from ${name}", [[7]], false)
+ }
+
+ createPk("insert_only_runs", "v STRING", ", 'read.batch-size'='1'")
+ 3.times { run ->
+ sql """INSERT INTO insert_only_runs
+ SELECT number, concat('${run}', repeat('x',4096)) FROM numbers('number'='1025')"""
+ }
+ // Overlapping INSERT-only files have no tombstones, but Rust retains losing batches.
+ check("select count(*), min(length(v)), max(length(v)) from insert_only_runs "
+ + "where substring(v,1,1)='2'", [[1025,4097,4097]], false)
+
+ createPk("retract_sum", "v DOUBLE, kind STRING", ", 'merge-engine'='aggregation', "
+ + "'fields.v.aggregate-function'='sum', 'rowkind.field'='kind'")
+ sql "INSERT INTO retract_sum VALUES (1,10.0,'+I')"
+ sql "INSERT INTO retract_sum VALUES (1,3.0,'-D')"
+ check("select id,v from retract_sum", [[1,7.0]], false)
+
+ createPk("deleted_prefix", "v STRING, kind STRING", ", 'rowkind.field'='kind'")
+ sql """INSERT INTO deleted_prefix SELECT number, repeat('x',128), '+I'
+ FROM numbers('number'='4096')"""
+ sql """INSERT INTO deleted_prefix SELECT number, repeat('x',128), '-D'
+ FROM numbers('number'='4096')"""
+ // A scan-producing predicate avoids metadata COUNT shortcuts for an empty result.
+ check("select id from deleted_prefix where v is not null", [], false)
+ createPk("sparse_prefix", "v STRING, kind STRING", ", 'rowkind.field'='kind'")
+ sql """INSERT INTO sparse_prefix SELECT number, repeat('x',128), '+I'
+ FROM numbers('number'='4097')"""
+ sql """INSERT INTO sparse_prefix SELECT number, repeat('x',128), '-D'
+ FROM numbers('number'='4096')"""
+ check("select id from sparse_prefix order by id", [[4096]], false)
+ } finally {
+ saved.each { name, value -> sql "set ${name}='${value}'" }
+ sql "DROP DATABASE IF EXISTS ${database} FORCE"
+ sql "DROP CATALOG IF EXISTS ${catalog}"
+ }
+}
diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_rust_reader_eq_for_null.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_rust_reader_eq_for_null.groovy
new file mode 100644
index 00000000000..a5d4b3412ef
--- /dev/null
+++ b/regression-test/suites/external_table_p0/paimon/test_paimon_rust_reader_eq_for_null.groovy
@@ -0,0 +1,555 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// NULL-safe equality (`<=>`, EQ_FOR_NULL) through the paimon rust reader.
+// The rust predicate converter must NOT push `a <=> b` down as `a IS NULL`:
+// with rows (NULL, NULL), (1, 1), (1, 2) that would wrongly drop (1, 1), and
+// rows dropped by the pushed filter cannot be recovered by the residual
+// conjunct. FE rewrites the literal forms (`a <=> 1` -> `a = 1`,
+// `a <=> NULL` -> `a IS NULL`) before they reach the BE, so only the
+// column-to-column form exercises EQ_FOR_NULL here; the literal forms still
+// guard the rewrite + pushdown chain end to end.
+//
+// Both differential legs run with force_jni_scanner=true: these parquet
+// append tables convert to raw native splits, which getSplits() would
+// otherwise prefer — both legs would silently use the native reader and
+// never reach the JNI / rust converters. The actual reader path is verified
+// per leg through the query profile (the rust reader's PaimonRustReader
+// timer group), and the join leg forces an IN runtime filter
+// (runtime_filter_type=1 + runtime_filter_wait_infinitely) that must be
+// planned onto the probe scan and arrive before the split opens.
+import org.apache.doris.regression.action.ProfileAction
+
+suite("test_paimon_rust_reader_eq_for_null", "p0,external,paimon") {
+ String enabled = context.config.otherConfigs.get("enablePaimonTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("disabled paimon test")
+ return
+ }
+
+ String catalogName = "test_paimon_rust_eq_null"
+ String dbName = "test_paimon_rust_eq_null_db"
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
+
+ // Use Spark to prepare the timestamp and nested-schema fixtures.
+ // Both columns stay nullable: `a <=> b` survives FE's NullSafeEqualToEqual
+ // rewrite (which only fires when one side is non-nullable / a NULL literal)
+ // precisely when both sides are nullable.
+ //
+ // t_frac_ts uses Spark TIMESTAMP_NTZ, which maps to Paimon TIMESTAMP
+ // (wall-clock, microsecond precision). Note Spark's plain TIMESTAMP maps
+ // to Paimon TIMESTAMP_LTZ, so the NTZ semantics must be spelled out in the
+ // DDL. spark.sql.timestampType=TIMESTAMP_NTZ makes the TIMESTAMP '...'
+ // literals parse as NTZ civil times too.
+ //
+ // The predicate literals are millisecond-aligned on purpose: FE truncates
+ // plan-time pushed-down timestamp predicates to 3 fractional digits, so a
+ // 6-digit literal would reach the readers truncated to milliseconds and
+ // the exact residual conjunct would then drop every row it kept — for the
+ // JNI, Rust and native readers alike. Literal conversion in the BE retains
+ // microseconds, but timestamp predicates remain
+ // residual because a DATETIMEV2 slot cannot reveal source nanosecond precision.
+ // ---- NaN differential on DOUBLE (total-ordering semantics) ----
+ // t_nan is created below: Doris defines NaN as equal to itself and
+ // greater than every finite value, but the pinned rust evaluator compares
+ // doubles with f64::partial_cmp (IEEE: NaN unordered, NaN != NaN), so the
+ // rust converter must NOT push DOUBLE predicates — a pushed `d > 1.0`
+ // would drop the stored NaN row Doris retains, and rows pruned by the
+ // rust filter cannot be recovered by the residual. The JNI path is
+ // unaffected: paimon-java's CompareUtils compares through Double.compareTo,
+ // which matches Doris's total ordering.
+ spark_paimon_multi """
+ SET spark.sql.timestampType=TIMESTAMP_NTZ;
+ CREATE DATABASE IF NOT EXISTS paimon.${dbName};
+ DROP TABLE IF EXISTS paimon.${dbName}.t_eq_null;
+ CREATE TABLE paimon.${dbName}.t_eq_null (
+ a INT, b INT
+ ) USING paimon;
+ INSERT INTO paimon.${dbName}.t_eq_null VALUES (NULL, NULL), (1, 1), (1, 2);
+
+ DROP TABLE IF EXISTS paimon.${dbName}.t_frac_ts;
+ CREATE TABLE paimon.${dbName}.t_frac_ts (
+ id INT, ts TIMESTAMP_NTZ
+ ) USING paimon;
+ INSERT INTO paimon.${dbName}.t_frac_ts VALUES
+ (1, TIMESTAMP '2024-01-01 00:00:00.123456'),
+ (2, TIMESTAMP '2024-01-01 00:00:00.123000'),
+ (3, TIMESTAMP '2024-01-02 00:00:00.999999');
+
+ DROP TABLE IF EXISTS paimon.${dbName}.t_frac_ts_dim;
+ CREATE TABLE paimon.${dbName}.t_frac_ts_dim (
+ id INT, ts TIMESTAMP_NTZ
+ ) USING paimon;
+ INSERT INTO paimon.${dbName}.t_frac_ts_dim VALUES
+ (1, TIMESTAMP '2024-01-01 00:00:00.123456'),
+ (2, TIMESTAMP '2024-01-02 00:00:00.999999');
+
+ DROP TABLE IF EXISTS paimon.${dbName}.t_narrowed;
+ CREATE TABLE paimon.${dbName}.t_narrowed (
+ id INT, v VARCHAR(10), amount DECIMAL(10,3)
+ ) USING paimon TBLPROPERTIES ('file.format' = 'parquet');
+ INSERT INTO paimon.${dbName}.t_narrowed VALUES (1, 'abcdef', 1.234), (2, 'xyzuvw', 2.345);
+ DROP TABLE IF EXISTS paimon.${dbName}.t_narrowed_dim;
+ CREATE TABLE paimon.${dbName}.t_narrowed_dim (
+ v VARCHAR(3), amount DECIMAL(10,2)
+ ) USING paimon;
+ INSERT INTO paimon.${dbName}.t_narrowed_dim VALUES ('abc', 1.23);
+
+ DROP TABLE IF EXISTS paimon.${dbName}.t_nan;
+ CREATE TABLE paimon.${dbName}.t_nan (
+ id INT, d DOUBLE
+ ) USING paimon;
+ INSERT INTO paimon.${dbName}.t_nan VALUES
+ (1, 1.5), (2, CAST('NaN' AS DOUBLE)), (3, NULL);
+
+ DROP TABLE IF EXISTS paimon.${dbName}.t_dedup_ignore_del;
+ CREATE TABLE paimon.${dbName}.t_dedup_ignore_del (
+ id INT, v INT
+ ) USING paimon TBLPROPERTIES (
+ 'primary-key' = 'id',
+ 'merge-engine' = 'deduplicate',
+ 'deduplicate.ignore-delete' = 'true',
+ 'file.format' = 'parquet'
+ );
+ INSERT INTO paimon.${dbName}.t_dedup_ignore_del VALUES (1, 11), (2, 22);
+ DELETE FROM paimon.${dbName}.t_dedup_ignore_del WHERE id = 1;
+
+ DROP TABLE IF EXISTS paimon.${dbName}.t_pu_remove_record_del;
+ CREATE TABLE paimon.${dbName}.t_pu_remove_record_del (
+ id INT, v INT
+ ) USING paimon TBLPROPERTIES (
+ 'primary-key' = 'id',
+ 'merge-engine' = 'partial-update',
+ 'partial-update.remove-record-on-delete' = 'true',
+ 'file.format' = 'parquet'
+ );
+ INSERT INTO paimon.${dbName}.t_pu_remove_record_del VALUES (1, 11), (2, 22);
+ DELETE FROM paimon.${dbName}.t_pu_remove_record_del WHERE id = 1;
+
+ DROP TABLE IF EXISTS paimon.${dbName}.t_nested_evo;
+ CREATE TABLE paimon.${dbName}.t_nested_evo (
+ id INT, s STRUCT<a: INT, b: STRING>
+ ) USING paimon TBLPROPERTIES (
+ 'bucket' = '-1',
+ 'file.format' = 'parquet'
+ );
+ INSERT INTO paimon.${dbName}.t_nested_evo VALUES (1, struct(10, 'x')), (2, struct(20, 'y'));
+ ALTER TABLE paimon.${dbName}.t_nested_evo ADD COLUMN s.c INT;
+ INSERT INTO paimon.${dbName}.t_nested_evo VALUES (3, struct(30, 'z', 33));
+ """
+
+ def originalSparkTimeZone = spark_paimon("SET spark.sql.session.timeZone")[0][1]
+ try {
+ spark_paimon "SET spark.sql.session.timeZone=Asia/Shanghai"
+ ["parquet", "orc"].each { format ->
+ // Legacy ORC LTZ encodes the writer JVM zone and requires the reader JVM
+ // to match it. Modern encoding keeps this routing test independent of host zones.
+ spark_paimon_multi """
+ DROP TABLE IF EXISTS paimon.${dbName}.t_ltz_${format};
+ CREATE TABLE paimon.${dbName}.t_ltz_${format} (id INT, ts_tz TIMESTAMP_LTZ)
+ USING paimon TBLPROPERTIES (
+ 'bucket'='-1', 'file.format'='${format}',
+ 'orc.timestamp-ltz.legacy.type'='false');
+ INSERT INTO paimon.${dbName}.t_ltz_${format} VALUES
+ (1, TIMESTAMP_LTZ '2025-01-01 00:00:00'),
+ (2, TIMESTAMP_LTZ '2025-06-01 12:34:56.789'),
+ (3, TIMESTAMP_LTZ '2025-12-31 23:59:59.999999'), (4, NULL);
+ """
+ }
+ } finally {
+ spark_paimon "SET spark.sql.session.timeZone=${originalSparkTimeZone}"
+ }
+
+ // The s3.region property is required: paimon-rust's S3 client rejects a
+ // missing region, while the JNI reader falls back to the SDK default.
+ sql """drop catalog if exists ${catalogName}"""
+ sql """
+ CREATE CATALOG ${catalogName} PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'filesystem',
+ 'warehouse' = 's3://warehouse/wh',
+ 's3.endpoint' = 'http://${externalEnvIp}:${minioPort}',
+ 's3.access_key' = 'admin',
+ 's3.secret_key' = 'password',
+ 's3.region' = 'us-east-1',
+ 'use_path_style' = 'true'
+ );
+ """
+
+ // Capture the settings this suite overrides so finally can restore them.
+ def originalForceJni = sql("select @@force_jni_scanner")[0][0]
+ def originalEnableProfile = sql("select @@enable_profile")[0][0]
+ def originalRfWait = sql("select @@runtime_filter_wait_infinitely")[0][0]
+ def originalRfType = sql("select @@runtime_filter_type")[0][0]
+ def originalRfPrune = sql("select @@enable_runtime_filter_prune")[0][0]
+ def originalRfMaxIn = sql("select @@runtime_filter_max_in_num")[0][0]
+
+ try {
+ sql """switch ${catalogName}"""
+ sql """use ${dbName}"""
+ // Spark rejects narrowing in its analyzer. Doris forwards these explicit
+ // casts to Paimon and publishes a new schema without rewriting old files.
+ sql "ALTER TABLE t_narrowed MODIFY COLUMN v VARCHAR(3) NULL"
+ sql "ALTER TABLE t_narrowed MODIFY COLUMN amount DECIMAL(10,2) NULL"
+ sql """set enable_file_scanner_v2=true"""
+ // These tables are parquet append tables, whose DataSplits convert to
+ // raw native splits; without forcing, getSplits() would hand both legs
+ // to the native reader and bypass the JNI / rust converters entirely.
+ sql """set force_jni_scanner=true"""
+ sql "set enable_paimon_rust_reader=false"
+ // Verify the DDL really narrows historical values before testing filter pushdown.
+ assertEquals([[1, "abc", "1.23"], [2, "xyz", "2.35"]].toString(),
+ sql("select id,v,cast(amount as string) from t_narrowed order by id").toString())
+ // Profile capture for the reader-path verification below.
+ sql """set enable_profile=true"""
+ // The join leg must receive its IN runtime filter before the split
+ // opens, so the rust converter sees it in the conjuncts.
+ sql """set runtime_filter_wait_infinitely=true"""
+ // TRuntimeFilterType.IN == 1: force the IN runtime-filter shape.
+ sql """set runtime_filter_type=1"""
+
+ // Reuse the framework's profile readiness polling and configured HTTP credentials.
+ def profileAction = new ProfileAction(context)
+ def profileTextOf = { String query, boolean rustExpected = true ->
+ sql(query)
+ def queryId = sql("select last_query_id()")[0][0]
+ // Negative and zero-counter checks must never inspect a partially published profile.
... 1169 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org