You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "shyjsarah (via GitHub)" <gi...@apache.org> on 2026/08/26 07:01:41 UTC
[PR] feat: add native object table read support [paimon-rust]
shyjsarah opened a new pull request, #749:
URL: https://github.com/apache/paimon-rust/pull/749
## Summary
Add native, read-only Object Table support to Paimon Rust and the DataFusion integration. The implementation follows Paimon Java's fixed Object Table schema and REST path semantics, so Object Tables no longer require an externally registered table engine for reads.
## Changes
- Add `ObjectTable` and `ObjectEntry` with the Java-compatible fixed schema: `path`, `name`, `length`, `mtime`, `atime`, and `owner`.
- Recursively list object metadata with relative paths and deterministic ordering.
- Route Object Tables natively from filesystem and REST catalogs.
- Build filesystem I/O from the Object Table's own path, including cross-scheme locations.
- Use `GetTableResponse.path` for REST Object Tables, matching Paimon Java and preserving REST data-token handling.
- Add a DataFusion table provider with projection and limit support.
- Keep Object Tables read-only and reject branches, time travel, and writes.
- Keep non-native table types on the existing external table-engine path.
## Testing
- [x] `cargo clippy --locked -p paimon -p paimon-datafusion --all-targets -- -D warnings`
- [x] `cargo test -p paimon-datafusion --test object_table`
- [x] `cargo test -p paimon-datafusion --test table_type_routing`
- [x] `cargo test -p paimon --lib catalog`
- [x] `cargo test -p paimon --test rest_catalog_test test_load_table_constructs_native_object_table -- --exact`
- [x] `cargo test -p paimon --test rest_catalog_test test_catalog_get_table -- --exact`
- [x] `cargo check -p paimon-rest-server --all-targets`
- [x] `cargo fmt --all -- --check`
- [x] `git diff --check`
## Notes
- OpenDAL does not expose access time or owner portably, so `atime` is currently `0` and `owner` is `NULL`; modification time and length are populated when available.
- Public API impact: adds `LoadedTable::Object`, `ObjectTable`, and `ObjectEntry`. Existing Paimon and external-engine variants keep their behavior.
- This is complementary to #747, which keeps metadata queries available when unsupported external engines are not registered; neither PR depends on the other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add native object table read support [paimon-rust]
Posted by "shyjsarah (via GitHub)" <gi...@apache.org>.
shyjsarah commented on code in PR #749:
URL: https://github.com/apache/paimon-rust/pull/749#discussion_r3870480386
##########
crates/integrations/datafusion/src/table/object.rs:
##########
@@ -0,0 +1,153 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use datafusion::arrow::array::{
+ new_null_array, ArrayRef, Int64Array, RecordBatch, StringViewArray,
+};
+use datafusion::arrow::datatypes::SchemaRef;
+use datafusion::catalog::Session;
+use datafusion::datasource::memory::MemorySourceConfig;
+use datafusion::datasource::{TableProvider, TableType};
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::logical_expr::dml::InsertOp;
+use datafusion::logical_expr::Expr;
+use datafusion::physical_plan::ExecutionPlan;
+use paimon::table::ObjectTable;
+
+use crate::error::to_datafusion_error;
+
+use super::datafusion_arrow_schema;
+
+/// DataFusion provider for a native read-only Paimon object table.
+#[derive(Debug, Clone)]
+pub(crate) struct ObjectTableProvider {
+ table: ObjectTable,
+ schema: SchemaRef,
+}
+
+impl ObjectTableProvider {
+ pub(crate) fn try_new(table: ObjectTable, schema_force_view_types: bool) -> DFResult<Self> {
+ let schema = datafusion_arrow_schema(&ObjectTable::fields(), schema_force_view_types)?;
+ Ok(Self { table, schema })
+ }
+}
+
+#[async_trait]
+impl TableProvider for ObjectTableProvider {
+ fn schema(&self) -> SchemaRef {
+ Arc::clone(&self.schema)
+ }
+
+ fn table_type(&self) -> TableType {
+ TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &dyn Session,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ limit: Option<usize>,
+ ) -> DFResult<Arc<dyn ExecutionPlan>> {
+ let mut entries = self
+ .table
+ .list_objects()
Review Comment:
Fixed in bb4c346. DataFusion now passes the limit into `ObjectTable`; recursive listing uses OpenDAL `lister_with` incrementally and keeps only the path-smallest K statuses in a bounded max-heap. Limited reads use O(K) memory and O(N log K) selection work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add native object table read support [paimon-rust]
Posted by "shyjsarah (via GitHub)" <gi...@apache.org>.
shyjsarah commented on code in PR #749:
URL: https://github.com/apache/paimon-rust/pull/749#discussion_r3877657824
##########
crates/integrations/datafusion/src/table/object.rs:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use datafusion::arrow::array::{
+ new_null_array, ArrayRef, Int64Array, RecordBatch, StringViewArray,
+};
+use datafusion::arrow::datatypes::SchemaRef;
+use datafusion::catalog::Session;
+use datafusion::datasource::memory::MemorySourceConfig;
+use datafusion::datasource::{TableProvider, TableType};
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::logical_expr::dml::InsertOp;
+use datafusion::logical_expr::Expr;
+use datafusion::physical_plan::ExecutionPlan;
+use paimon::table::ObjectTable;
+
+use crate::error::to_datafusion_error;
+
+use super::datafusion_arrow_schema;
+
+/// DataFusion provider for a native read-only Paimon object table.
+#[derive(Debug, Clone)]
+pub(crate) struct ObjectTableProvider {
+ table: ObjectTable,
+ schema: SchemaRef,
+}
+
+impl ObjectTableProvider {
+ pub(crate) fn try_new(table: ObjectTable, schema_force_view_types: bool) -> DFResult<Self> {
+ let schema = datafusion_arrow_schema(&ObjectTable::fields(), schema_force_view_types)?;
+ Ok(Self { table, schema })
+ }
+}
+
+#[async_trait]
+impl TableProvider for ObjectTableProvider {
+ fn schema(&self) -> SchemaRef {
+ Arc::clone(&self.schema)
+ }
+
+ fn table_type(&self) -> TableType {
+ TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &dyn Session,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ limit: Option<usize>,
+ ) -> DFResult<Arc<dyn ExecutionPlan>> {
+ let entries = self
Review Comment:
Fixed in e50ea4f. Object Table scans now return a projection-aware `StreamingTableExec` instead of listing and materializing during planning. The partition stream lazily consumes OpenDAL, emits batches at the session batch size, builds only projected Arrow columns (including zero-column projections), and does not globally sort scan output. Pushed limits still stop the underlying lister after K files. Added regressions proving the first status is yielded before the next lister poll, projected scans produce bounded batches, and `COUNT(*)` works through an empty projection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add native object table read support [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #749:
URL: https://github.com/apache/paimon-rust/pull/749#issuecomment-5435989576
- When `path` is not configured, Java defaults to using the table directory; Rust succeeds in creation but throws a `requires a non-empty 'path' option` error upon loading.
- Java ignores user-defined fields and persists a fixed six-column schema; Rust persists the user-defined schema as-is, yet the reader forces the exposure of a fixed schema, resulting in an inconsistency between the metadata and the query schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add native object table read support [paimon-rust]
Posted by "jerry-024 (via GitHub)" <gi...@apache.org>.
jerry-024 merged PR #749:
URL: https://github.com/apache/paimon-rust/pull/749
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add native object table read support [paimon-rust]
Posted by "jerry-024 (via GitHub)" <gi...@apache.org>.
jerry-024 commented on code in PR #749:
URL: https://github.com/apache/paimon-rust/pull/749#discussion_r3870089351
##########
crates/integrations/datafusion/src/table/object.rs:
##########
@@ -0,0 +1,153 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use datafusion::arrow::array::{
+ new_null_array, ArrayRef, Int64Array, RecordBatch, StringViewArray,
+};
+use datafusion::arrow::datatypes::SchemaRef;
+use datafusion::catalog::Session;
+use datafusion::datasource::memory::MemorySourceConfig;
+use datafusion::datasource::{TableProvider, TableType};
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::logical_expr::dml::InsertOp;
+use datafusion::logical_expr::Expr;
+use datafusion::physical_plan::ExecutionPlan;
+use paimon::table::ObjectTable;
+
+use crate::error::to_datafusion_error;
+
+use super::datafusion_arrow_schema;
+
+/// DataFusion provider for a native read-only Paimon object table.
+#[derive(Debug, Clone)]
+pub(crate) struct ObjectTableProvider {
+ table: ObjectTable,
+ schema: SchemaRef,
+}
+
+impl ObjectTableProvider {
+ pub(crate) fn try_new(table: ObjectTable, schema_force_view_types: bool) -> DFResult<Self> {
+ let schema = datafusion_arrow_schema(&ObjectTable::fields(), schema_force_view_types)?;
+ Ok(Self { table, schema })
+ }
+}
+
+#[async_trait]
+impl TableProvider for ObjectTableProvider {
+ fn schema(&self) -> SchemaRef {
+ Arc::clone(&self.schema)
+ }
+
+ fn table_type(&self) -> TableType {
+ TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &dyn Session,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ limit: Option<usize>,
+ ) -> DFResult<Arc<dyn ExecutionPlan>> {
+ let mut entries = self
+ .table
+ .list_objects()
Review Comment:
<!-- dlf-review -->
**[MAJOR]** `LIMIT` is applied only after `list_objects()` has recursively enumerated, materialized, and path-sorted the entire object namespace. Consequently, even `LIMIT 1` retains O(N) listing/memory work and O(N log N) sorting, which can cause high latency or exhaust memory on large object tables.
Please pass the limit into the listing layer and consume the recursive lister incrementally, retaining at most the path-smallest K entries. Keep the full listing only when no limit is supplied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add native object table read support [paimon-rust]
Posted by "shyjsarah (via GitHub)" <gi...@apache.org>.
shyjsarah commented on PR #749:
URL: https://github.com/apache/paimon-rust/pull/749#issuecomment-5437098514
Addressed in bb4c346. Filesystem object-table creation now follows Java semantics: it persists the fixed six-column schema, defaults a missing/blank `path` to the table directory, preserves an explicit path, and loads legacy metadata without `path` from the table directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add native object table read support [paimon-rust]
Posted by "shyjsarah (via GitHub)" <gi...@apache.org>.
shyjsarah commented on code in PR #749:
URL: https://github.com/apache/paimon-rust/pull/749#discussion_r3870480978
##########
crates/paimon/src/catalog/mod.rs:
##########
@@ -263,14 +263,16 @@ use async_trait::async_trait;
use crate::api::PagedList;
use crate::spec::{Partition, Schema, SchemaChange, TableType};
-use crate::table::Table;
+use crate::table::{ObjectTable, Table};
/// Outcome of [`Catalog::load_table`].
#[derive(Debug)]
pub enum LoadedTable {
/// A constructed Paimon table (boxed: far larger than the other variant).
Paimon(Box<Table>),
- /// A table this reader cannot construct.
+ /// A native read-only object table.
+ Object(ObjectTable),
+ /// A table that needs a registered external engine.
Review Comment:
Fixed in bb4c346. `LoadedTable` is now `#[non_exhaustive]`, downstream matches have explicit fallback handling, and the PR description documents the API impact. `LoadedTable` was introduced after v0.3.0, so this is done before its first v0.4 publication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add native object table read support [paimon-rust]
Posted by "jerry-024 (via GitHub)" <gi...@apache.org>.
jerry-024 commented on code in PR #749:
URL: https://github.com/apache/paimon-rust/pull/749#discussion_r3870780685
##########
crates/integrations/datafusion/src/table/object.rs:
##########
@@ -0,0 +1,153 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use datafusion::arrow::array::{
+ new_null_array, ArrayRef, Int64Array, RecordBatch, StringViewArray,
+};
+use datafusion::arrow::datatypes::SchemaRef;
+use datafusion::catalog::Session;
+use datafusion::datasource::memory::MemorySourceConfig;
+use datafusion::datasource::{TableProvider, TableType};
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::logical_expr::dml::InsertOp;
+use datafusion::logical_expr::Expr;
+use datafusion::physical_plan::ExecutionPlan;
+use paimon::table::ObjectTable;
+
+use crate::error::to_datafusion_error;
+
+use super::datafusion_arrow_schema;
+
+/// DataFusion provider for a native read-only Paimon object table.
+#[derive(Debug, Clone)]
+pub(crate) struct ObjectTableProvider {
+ table: ObjectTable,
+ schema: SchemaRef,
+}
+
+impl ObjectTableProvider {
+ pub(crate) fn try_new(table: ObjectTable, schema_force_view_types: bool) -> DFResult<Self> {
+ let schema = datafusion_arrow_schema(&ObjectTable::fields(), schema_force_view_types)?;
+ Ok(Self { table, schema })
+ }
+}
+
+#[async_trait]
+impl TableProvider for ObjectTableProvider {
+ fn schema(&self) -> SchemaRef {
+ Arc::clone(&self.schema)
+ }
+
+ fn table_type(&self) -> TableType {
+ TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &dyn Session,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ limit: Option<usize>,
+ ) -> DFResult<Arc<dyn ExecutionPlan>> {
+ let mut entries = self
+ .table
+ .list_objects()
Review Comment:
The update bounds retained memory to O(K) and reduces sorting to O(N log K), but the recursive lister is still drained to EOF. As a result, `LIMIT 1` still performs O(N) remote listing work and waits for the entire namespace.
Because SQL `LIMIT` without `ORDER BY` does not require the globally smallest paths, this path can stop after K files and optionally sort only those K results. A global traversal should be reserved for an explicit ordering requirement.
##########
crates/paimon/src/catalog/mod.rs:
##########
@@ -263,14 +263,16 @@ use async_trait::async_trait;
use crate::api::PagedList;
use crate::spec::{Partition, Schema, SchemaChange, TableType};
-use crate::table::Table;
+use crate::table::{ObjectTable, Table};
/// Outcome of [`Catalog::load_table`].
#[derive(Debug)]
pub enum LoadedTable {
/// A constructed Paimon table (boxed: far larger than the other variant).
Paimon(Box<Table>),
- /// A table this reader cannot construct.
+ /// A native read-only object table.
+ Object(ObjectTable),
+ /// A table that needs a registered external engine.
Review Comment:
Retraction after checking the published API history: `LoadedTable` is not present in the released v0.3.0 API and currently exists only on the unreleased 0.4.0 development line. Adding `Object` before the enum’s first release does not break released consumers, and `#[non_exhaustive]` prevents the same issue for future variants.
This is not a compatibility blocker; please disregard my original finding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add native object table read support [paimon-rust]
Posted by "jerry-024 (via GitHub)" <gi...@apache.org>.
jerry-024 commented on PR #749:
URL: https://github.com/apache/paimon-rust/pull/749#issuecomment-5436535366
<!-- dlf-review head=ab3a563eebfe4a8f731f72e9a6190fc97bf801c1 -->
New: 2; still open: 0; fixed candidates: 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add native object table read support [paimon-rust]
Posted by "jerry-024 (via GitHub)" <gi...@apache.org>.
jerry-024 commented on code in PR #749:
URL: https://github.com/apache/paimon-rust/pull/749#discussion_r3870779198
##########
crates/paimon/src/io/file_io.rs:
##########
@@ -223,23 +225,38 @@ impl FileIO {
/// List all files recursively under the given directory path.
pub async fn list_status_recursive(&self, path: &str) -> Result<Vec<FileStatus>> {
+ self.list_status_recursive_with_limit(path, None).await
+ }
+
+ pub(crate) async fn list_status_recursive_with_limit(
+ &self,
+ path: &str,
+ limit: Option<usize>,
+ ) -> Result<Vec<FileStatus>> {
+ if limit == Some(0) {
+ return Ok(Vec::new());
+ }
+
let (op, relative_path) = self.create(path).await?;
// See `list_status`: `relative_path` is a byte-suffix of `path` except
// for Windows local paths, where it only swaps separators (same length).
let base_path = &path[..path.len() - relative_path.len()];
let list_path = normalize_root(relative_path.as_ref());
- let entries =
- op.list_with(&list_path)
+ let mut entries =
+ op.lister_with(&list_path)
.recursive(true)
.await
.context(IoUnexpectedSnafu {
message: format!("Failed to list files recursively in '{path}'"),
})?;
let mut statuses = Vec::new();
+ let mut smallest = limit.map(|limit| BinaryHeap::with_capacity(limit.saturating_add(1)));
Review Comment:
<!-- dlf-review -->
**[MAJOR]** `BinaryHeap::with_capacity(limit.saturating_add(1))` allocates from the SQL `LIMIT` before the lister yields any object. A query with a very large limit can therefore request gigabytes of memory—or panic on capacity overflow—even when the directory is empty.
Please use `BinaryHeap::new()` and let the heap grow only as matching files are observed. The existing `heap.len() > limit` check already keeps retained entries bounded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add native object table read support [paimon-rust]
Posted by "jerry-024 (via GitHub)" <gi...@apache.org>.
jerry-024 commented on code in PR #749:
URL: https://github.com/apache/paimon-rust/pull/749#discussion_r3877568962
##########
crates/integrations/datafusion/src/table/object.rs:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use datafusion::arrow::array::{
+ new_null_array, ArrayRef, Int64Array, RecordBatch, StringViewArray,
+};
+use datafusion::arrow::datatypes::SchemaRef;
+use datafusion::catalog::Session;
+use datafusion::datasource::memory::MemorySourceConfig;
+use datafusion::datasource::{TableProvider, TableType};
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::logical_expr::dml::InsertOp;
+use datafusion::logical_expr::Expr;
+use datafusion::physical_plan::ExecutionPlan;
+use paimon::table::ObjectTable;
+
+use crate::error::to_datafusion_error;
+
+use super::datafusion_arrow_schema;
+
+/// DataFusion provider for a native read-only Paimon object table.
+#[derive(Debug, Clone)]
+pub(crate) struct ObjectTableProvider {
+ table: ObjectTable,
+ schema: SchemaRef,
+}
+
+impl ObjectTableProvider {
+ pub(crate) fn try_new(table: ObjectTable, schema_force_view_types: bool) -> DFResult<Self> {
+ let schema = datafusion_arrow_schema(&ObjectTable::fields(), schema_force_view_types)?;
+ Ok(Self { table, schema })
+ }
+}
+
+#[async_trait]
+impl TableProvider for ObjectTableProvider {
+ fn schema(&self) -> SchemaRef {
+ Arc::clone(&self.schema)
+ }
+
+ fn table_type(&self) -> TableType {
+ TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &dyn Session,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ limit: Option<usize>,
+ ) -> DFResult<Arc<dyn ExecutionPlan>> {
+ let entries = self
Review Comment:
<!-- dlf-review -->
**[Major] Avoid fully materializing unbounded object scans**
When `limit` is `None`, this path recursively lists and sorts the entire namespace, then duplicates its metadata into six vectors and one `RecordBatch` before the execution plan can emit any rows. This adds `O(N log N)` setup time and several `O(N)` allocations, can exhaust memory for large object stores, and still constructs columns excluded by the projection. Please stream entries into bounded `RecordBatch`es, construct only projected columns, and avoid a global sort unless ordering is explicitly required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add native object table read support [paimon-rust]
Posted by "shyjsarah (via GitHub)" <gi...@apache.org>.
shyjsarah commented on code in PR #749:
URL: https://github.com/apache/paimon-rust/pull/749#discussion_r3870969674
##########
crates/paimon/src/io/file_io.rs:
##########
@@ -223,23 +225,38 @@ impl FileIO {
/// List all files recursively under the given directory path.
pub async fn list_status_recursive(&self, path: &str) -> Result<Vec<FileStatus>> {
+ self.list_status_recursive_with_limit(path, None).await
+ }
+
+ pub(crate) async fn list_status_recursive_with_limit(
+ &self,
+ path: &str,
+ limit: Option<usize>,
+ ) -> Result<Vec<FileStatus>> {
+ if limit == Some(0) {
+ return Ok(Vec::new());
+ }
+
let (op, relative_path) = self.create(path).await?;
// See `list_status`: `relative_path` is a byte-suffix of `path` except
// for Windows local paths, where it only swaps separators (same length).
let base_path = &path[..path.len() - relative_path.len()];
let list_path = normalize_root(relative_path.as_ref());
- let entries =
- op.list_with(&list_path)
+ let mut entries =
+ op.lister_with(&list_path)
.recursive(true)
.await
.context(IoUnexpectedSnafu {
message: format!("Failed to list files recursively in '{path}'"),
})?;
let mut statuses = Vec::new();
+ let mut smallest = limit.map(|limit| BinaryHeap::with_capacity(limit.saturating_add(1)));
Review Comment:
Fixed in fc31528. Limited recursive listing no longer allocates from the SQL LIMIT at all: the bounded heap was removed, entries are appended as the lister yields them, and polling stops once K files have been collected. A regression with `usize::MAX` on an empty location verifies that no capacity allocation or panic occurs.
##########
crates/integrations/datafusion/src/table/object.rs:
##########
@@ -0,0 +1,153 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use datafusion::arrow::array::{
+ new_null_array, ArrayRef, Int64Array, RecordBatch, StringViewArray,
+};
+use datafusion::arrow::datatypes::SchemaRef;
+use datafusion::catalog::Session;
+use datafusion::datasource::memory::MemorySourceConfig;
+use datafusion::datasource::{TableProvider, TableType};
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::logical_expr::dml::InsertOp;
+use datafusion::logical_expr::Expr;
+use datafusion::physical_plan::ExecutionPlan;
+use paimon::table::ObjectTable;
+
+use crate::error::to_datafusion_error;
+
+use super::datafusion_arrow_schema;
+
+/// DataFusion provider for a native read-only Paimon object table.
+#[derive(Debug, Clone)]
+pub(crate) struct ObjectTableProvider {
+ table: ObjectTable,
+ schema: SchemaRef,
+}
+
+impl ObjectTableProvider {
+ pub(crate) fn try_new(table: ObjectTable, schema_force_view_types: bool) -> DFResult<Self> {
+ let schema = datafusion_arrow_schema(&ObjectTable::fields(), schema_force_view_types)?;
+ Ok(Self { table, schema })
+ }
+}
+
+#[async_trait]
+impl TableProvider for ObjectTableProvider {
+ fn schema(&self) -> SchemaRef {
+ Arc::clone(&self.schema)
+ }
+
+ fn table_type(&self) -> TableType {
+ TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &dyn Session,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ limit: Option<usize>,
+ ) -> DFResult<Arc<dyn ExecutionPlan>> {
+ let mut entries = self
+ .table
+ .list_objects()
Review Comment:
Addressed in fc31528. For a pushed `LIMIT K` without `ORDER BY`, the recursive OpenDAL lister now stops after K files instead of draining to EOF; only those K entries are sorted before materialization. Unlimited scans still traverse the full namespace. I also added an instrumented lister regression proving it is not polled past K, plus an `ORDER BY path LIMIT 1` regression to preserve ordered-query correctness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add native object table read support [paimon-rust]
Posted by "jerry-024 (via GitHub)" <gi...@apache.org>.
jerry-024 commented on code in PR #749:
URL: https://github.com/apache/paimon-rust/pull/749#discussion_r3870092021
##########
crates/paimon/src/catalog/mod.rs:
##########
@@ -263,14 +263,16 @@ use async_trait::async_trait;
use crate::api::PagedList;
use crate::spec::{Partition, Schema, SchemaChange, TableType};
-use crate::table::Table;
+use crate::table::{ObjectTable, Table};
/// Outcome of [`Catalog::load_table`].
#[derive(Debug)]
pub enum LoadedTable {
/// A constructed Paimon table (boxed: far larger than the other variant).
Paimon(Box<Table>),
- /// A table this reader cannot construct.
+ /// A native read-only object table.
+ Object(ObjectTable),
+ /// A table that needs a registered external engine.
Review Comment:
<!-- dlf-review -->
**[MAJOR]** Adding `Object` to the public, exhaustive `LoadedTable` enum is a breaking API change: downstream crates with exhaustive matches will fail to compile after upgrading, even if they never use object tables.
Please treat and document this as a semver-breaking release. If more table kinds are expected, consider marking the enum `#[non_exhaustive]` in the same breaking release so subsequent variants do not repeatedly break consumers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org