You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "luoyuxia (via GitHub)" <gi...@apache.org> on 2026/03/22 03:59:51 UTC
[PR] feat: support datafusion integration [paimon-rust]
luoyuxia opened a new pull request, #150:
URL: https://github.com/apache/paimon-rust/pull/150
### Purpose
Linked issue: N/A
Add an initial read-only DataFusion integration for Paimon tables so users can register a Paimon table as a DataFusion `TableProvider` and query it with SQL/DataFrame APIs.
### Brief change log
- add a new `paimon-datafusion` integration crate
- implement a read-only `PaimonTableProvider` and physical scan plan for DataFusion
- convert Paimon schema types to Arrow/DataFusion schema types, including time, timestamp, local-zoned timestamp, and decimal precision mapping
- add DataFusion integration tests for reading a log table and a DV-backed primary key table
- wire the DataFusion integration test into CI's integration job
### Tests
- `cargo test -p paimon-datafusion schema:: --lib`
- `cargo test -p paimon-datafusion --test read_tables --no-run`
### API and Format
- adds a new integration crate: `crates/integrations/datafusion`
- no storage format changes
- no changes to existing public APIs in `paimon`
### Documentation
- no additional documentation changes in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support datafusion integration [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #150:
URL: https://github.com/apache/paimon-rust/pull/150
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support datafusion integration [paimon-rust]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on code in PR #150:
URL: https://github.com/apache/paimon-rust/pull/150#discussion_r2975872379
##########
crates/integrations/datafusion/src/physical_plan/scan.rs:
##########
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use datafusion::arrow::datatypes::SchemaRef as ArrowSchemaRef;
+use datafusion::error::Result as DFResult;
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::physical_expr::EquivalenceProperties;
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
+use datafusion::physical_plan::{DisplayAs, ExecutionPlan, Partitioning, PlanProperties};
+use futures::{StreamExt, TryStreamExt};
+use paimon::table::Table;
+
+use crate::error::to_datafusion_error;
+
+/// Execution plan that scans a Paimon table (read-only, no projection, no predicate, no limit).
+#[derive(Debug)]
+pub struct PaimonTableScan {
+ table: Table,
+ plan_properties: PlanProperties,
+}
+
+impl PaimonTableScan {
+ pub(crate) fn new(schema: ArrowSchemaRef, table: Table) -> Self {
+ let plan_properties = PlanProperties::new(
+ EquivalenceProperties::new(schema.clone()),
+ Partitioning::UnknownPartitioning(1),
Review Comment:
Is the single-partition execution intentional for the initial version?
`PaimonTableScan` hardcodes `UnknownPartitioning(1)`, and `execute()` plans/reads all Paimon splits inside one execution partition. Since the underlying Paimon scan already produces bin-packed splits, this means we lose DataFusion parallelism completely.
Not necessarily a blocker for the first PR, but I think this limitation should be called out explicitly (e.g. a `// TODO` comment), or followed up by exposing one execution partition per Paimon split.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org