You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "jackylee-ch (via GitHub)" <gi...@apache.org> on 2026/10/05 12:48:48 UTC
[PR] feat(datafusion): support the paimon_incremental_query table function [paimon-rust]
jackylee-ch opened a new pull request, #1042:
URL: https://github.com/apache/paimon-rust/pull/1042
Closes #506.
Adds the `paimon_incremental_query` table function so DataFusion SQL can read a
snapshot-id range `(start_exclusive, end_inclusive]` as a batch:
```sql
SELECT * FROM paimon_incremental_query('db.t', 0, 5);
SELECT * FROM paimon_incremental_query('db.t', 0, 5, 'delta');
SELECT * FROM paimon_incremental_query('db.t$audit_log', 0, 5, 'changelog');
```
The optional mode arg is `auto`/`delta`/`changelog`/`diff` (default `auto`); a
`$audit_log` suffix prepends the `rowkind` column. Snapshot planning stays in
paimon core — this only parses args and streams the core plan's Arrow output,
mirroring the existing vector-search table function.
Tests cover the range bounds, explicit `delta`, projection, `$audit_log`
rowkind, and the unknown-mode / bad-arity / end-before-start errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): support the paimon_incremental_query table function [paimon-rust]
Posted by "jackylee-ch (via GitHub) via issues" <is...@paimon.apache.org>.
jackylee-ch closed pull request #1042: feat(datafusion): support the paimon_incremental_query table function
URL: https://github.com/apache/paimon-rust/pull/1042
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): support the paimon_incremental_query table function [paimon-rust]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #1042:
URL: https://github.com/apache/paimon-rust/pull/1042#discussion_r4358211699
##########
crates/paimon/src/table/table_read.rs:
##########
@@ -776,10 +776,21 @@ impl<'a> PaimonTableRead<'a> {
}
}
}
- // Delta / Changelog rows are read as-is from planned files (no full-table
- // merge against historical base versions).
+ // Delta / Changelog rows are read as-is from planned files, without a merge
+ // against historical base versions. Files are still decoded as an ordinary
+ // read decodes them: primary-key BLOB descriptors resolve to payloads, and
+ // data-evolution files in a split that hold column slices of the same rows
+ // merge by row id.
let data_splits = self.matching_splits(&data_splits)?;
- self.new_data_file_reader()?.read(&data_splits)
+ let core_options = self.table.schema().core_options();
+ if !self.table.schema().primary_keys().is_empty() {
+ return self
+ .resolving_primary_key_blobs(&core_options, |read| read.read_raw(&data_splits));
+ }
+ if core_options.data_evolution_enabled() {
+ return self.read_with_evolution(&data_splits, &core_options);
Review Comment:
Fixed in 8f26c02: change events merge a row-id group only when its files are column slices of one row range; others, like this VECTOR-only update, are decoded file by file. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): support the paimon_incremental_query table function [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1042:
URL: https://github.com/apache/paimon-rust/pull/1042#discussion_r4202988106
##########
crates/integrations/datafusion/src/incremental_query.rs:
##########
@@ -0,0 +1,365 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The `paimon_incremental_query` table function reads a snapshot-id range
+//! `(start_exclusive, end_inclusive]` as a batch. Snapshot planning stays in
+//! paimon core; this only parses arguments and streams the core plan's output.
+//!
+//! ```sql
+//! SELECT * FROM paimon_incremental_query('db.t', 0, 5);
+//! SELECT * FROM paimon_incremental_query('db.t', 0, 5, 'delta');
+//! SELECT * FROM paimon_incremental_query('db.t$audit_log', 0, 5, 'changelog');
+//! ```
+//!
+//! A `$audit_log` table-name suffix prepends the `rowkind` audit column (and the
+//! optional `_SEQUENCE_NUMBER`), mirroring the core audit-log read.
+
+use std::fmt;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use datafusion::arrow::datatypes::SchemaRef as ArrowSchemaRef;
+use datafusion::arrow::record_batch::RecordBatch;
+use datafusion::catalog::{Session, TableFunctionImpl};
+use datafusion::common::stats::Precision;
+use datafusion::common::{internal_err, project_schema, Statistics};
+use datafusion::datasource::{TableProvider, TableType};
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::logical_expr::Expr;
+use datafusion::physical_expr::EquivalenceProperties;
+use datafusion::physical_plan::empty::EmptyExec;
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
+use datafusion::physical_plan::{
+ DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use datafusion::prelude::SessionContext;
+use futures::{stream, TryStreamExt};
+use paimon::catalog::Catalog;
+use paimon::table::{AuditLogTable, IncrementalScanMode, Table};
+
+use crate::error::to_datafusion_error;
+use crate::runtime::{await_with_runtime, block_on_with_runtime};
+use crate::table::datafusion_arrow_schema;
+use crate::table_function_args::{
+ extract_int_literal, extract_string_literal, parse_table_identifier,
+};
+use crate::table_loader::load_data_table_for_read;
+
+const FUNCTION_NAME: &str = "paimon_incremental_query";
+const AUDIT_LOG_SUFFIX: &str = "$audit_log";
+
+/// Registers `paimon_incremental_query` against `catalog`.
+pub(crate) fn register_incremental_query(
+ ctx: &SessionContext,
+ catalog: Arc<dyn Catalog>,
+ default_database: &str,
+) {
+ ctx.register_udtf(
+ FUNCTION_NAME,
+ Arc::new(IncrementalQueryFunction {
+ catalog,
+ default_database: default_database.to_string(),
+ }),
+ );
+}
+
+pub(crate) struct IncrementalQueryFunction {
+ catalog: Arc<dyn Catalog>,
+ default_database: String,
+}
+
+impl fmt::Debug for IncrementalQueryFunction {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.debug_struct("IncrementalQueryFunction")
+ .field("default_database", &self.default_database)
+ .finish()
+ }
+}
+
+/// Parses the optional mode argument. `Auto` resolves from `changelog-producer`
+/// during planning; `Diff` is handled by core's diff planning.
+fn parse_mode(raw: &str) -> DFResult<IncrementalScanMode> {
+ match raw.to_ascii_lowercase().as_str() {
+ "auto" => Ok(IncrementalScanMode::Auto),
+ "delta" => Ok(IncrementalScanMode::Delta),
+ "changelog" => Ok(IncrementalScanMode::Changelog),
+ "diff" => Ok(IncrementalScanMode::Diff),
+ other => Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME}: unknown scan mode '{other}', expected auto, delta, changelog, or diff"
+ ))),
+ }
+}
+
+impl TableFunctionImpl for IncrementalQueryFunction {
+ fn call(&self, args: &[Expr]) -> DFResult<Arc<dyn TableProvider>> {
+ if args.len() < 3 || args.len() > 4 {
+ return Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME} requires 3 or 4 arguments: (table_name, start_snapshot_exclusive, end_snapshot_inclusive [, mode])"
+ )));
+ }
+ let table_name = extract_string_literal(FUNCTION_NAME, &args[0], "table_name")?;
+ let start_exclusive =
+ extract_int_literal(FUNCTION_NAME, &args[1], "start_snapshot_exclusive")?;
+ let end_inclusive = extract_int_literal(FUNCTION_NAME, &args[2], "end_snapshot_inclusive")?;
+ let mode = match args.get(3) {
+ Some(expr) => parse_mode(&extract_string_literal(FUNCTION_NAME, expr, "mode")?)?,
+ None => IncrementalScanMode::Auto,
+ };
+ if end_inclusive < start_exclusive {
+ return Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME}: end_snapshot_inclusive ({end_inclusive}) must be >= start_snapshot_exclusive ({start_exclusive})"
+ )));
+ }
+
+ // `db.t$audit_log` selects the audit-log projection of `db.t`.
+ let (base_name, audit_log) = match table_name.strip_suffix(AUDIT_LOG_SUFFIX) {
+ Some(base) => (base.to_string(), true),
+ None => (table_name.clone(), false),
+ };
+ let identifier = parse_table_identifier(FUNCTION_NAME, &base_name, &self.default_database)?;
+
+ let catalog = Arc::clone(&self.catalog);
+ let table = block_on_with_runtime(
+ async move { load_data_table_for_read(&catalog, &identifier, FUNCTION_NAME).await },
+ "paimon_incremental_query: catalog access thread panicked",
+ )?;
+
+ Ok(Arc::new(IncrementalQueryTableProvider::try_new(
+ table,
+ audit_log,
+ mode,
+ start_exclusive,
+ end_inclusive,
+ )?))
+ }
+}
+#[derive(Debug, Clone)]
+struct IncrementalQueryTableProvider {
+ table: Table,
+ audit_log: bool,
+ mode: IncrementalScanMode,
+ start_exclusive: i64,
+ end_inclusive: i64,
+ schema: ArrowSchemaRef,
+}
+
+impl IncrementalQueryTableProvider {
+ fn try_new(
+ table: Table,
+ audit_log: bool,
+ mode: IncrementalScanMode,
+ start_exclusive: i64,
+ end_inclusive: i64,
+ ) -> DFResult<Self> {
+ // Audit-log output leads with `rowkind` (+ optional `_SEQUENCE_NUMBER`);
+ // the plain projection is the table's own fields.
+ let fields = if audit_log {
+ AuditLogTable::new(table.clone())
+ .fields()
+ .map_err(to_datafusion_error)?
+ } else {
+ table.schema().fields().to_vec()
+ };
+ // Core streams raw Arrow (`Utf8`, not `Utf8View`), so declare the schema
+ // the reader actually yields rather than the view-forced scan schema.
+ let schema = datafusion_arrow_schema(&fields, false)?;
+ Ok(Self {
+ table,
+ audit_log,
+ mode,
+ start_exclusive,
+ end_inclusive,
+ schema,
+ })
+ }
+}
+
+#[async_trait]
+impl TableProvider for IncrementalQueryTableProvider {
+ fn schema(&self) -> ArrowSchemaRef {
+ Arc::clone(&self.schema)
+ }
+
+ fn table_type(&self) -> TableType {
+ TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &dyn Session,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ limit: Option<usize>,
+ ) -> DFResult<Arc<dyn ExecutionPlan>> {
+ let projected_schema = project_schema(&self.schema, projection)?;
+ // An outer `LIMIT 0` needs no rows.
+ if limit == Some(0) {
+ return Ok(Arc::new(EmptyExec::new(projected_schema)));
+ }
+ Ok(Arc::new(IncrementalQueryExec::new(
+ self.table.clone(),
+ self.audit_log,
+ self.mode,
+ self.start_exclusive,
+ self.end_inclusive,
+ projection.cloned(),
+ projected_schema,
+ )))
+ }
+}
+/// Execution-time plan: builds the incremental plan in paimon core and streams
+/// its Arrow output when polled, so planning and `EXPLAIN` stay cheap.
+#[derive(Debug, Clone)]
+struct IncrementalQueryExec {
+ table: Table,
+ audit_log: bool,
+ mode: IncrementalScanMode,
+ start_exclusive: i64,
+ end_inclusive: i64,
+ projection: Option<Vec<usize>>,
+ output_schema: ArrowSchemaRef,
+ plan_properties: Arc<PlanProperties>,
+}
+
+impl IncrementalQueryExec {
+ fn new(
+ table: Table,
+ audit_log: bool,
+ mode: IncrementalScanMode,
+ start_exclusive: i64,
+ end_inclusive: i64,
+ projection: Option<Vec<usize>>,
+ output_schema: ArrowSchemaRef,
+ ) -> Self {
+ let plan_properties = Arc::new(PlanProperties::new(
+ EquivalenceProperties::new(output_schema.clone()),
+ Partitioning::UnknownPartitioning(1),
+ EmissionType::Incremental,
+ Boundedness::Bounded,
+ ));
+ Self {
+ table,
+ audit_log,
+ mode,
+ start_exclusive,
+ end_inclusive,
+ projection,
+ output_schema,
+ plan_properties,
+ }
+ }
+
+ async fn compute_batches(&self) -> DFResult<Vec<RecordBatch>> {
+ // Core owns snapshot planning; its stream is `'static`, so it outlives
+ // the transient read builder it is built from.
+ let batches = await_with_runtime(async {
+ let read_builder = self.table.new_read_builder();
+ let scan = read_builder.new_incremental_scan(
+ self.mode,
+ self.start_exclusive,
+ self.end_inclusive,
+ );
+ let plan = scan.plan().await.map_err(to_datafusion_error)?;
+ let read = read_builder.new_read().map_err(to_datafusion_error)?;
+ let mut stream = if self.audit_log {
+ read.to_audit_log_arrow(&plan)
+ } else {
+ read.to_incremental_arrow(&plan)
+ }
+ .map_err(to_datafusion_error)?;
+ let mut batches = Vec::new();
+ while let Some(batch) = stream.try_next().await.map_err(to_datafusion_error)? {
+ batches.push(batch);
Review Comment:
[P1] Keep incremental reads streaming and honor execution resources
This drains the core stream into a Vec before the first output batch; projection happens only afterwards and `execute` ignores its TaskContext. Delta/Changelog core already yields bounded batches, but the SQL adapter retains all decoded columns for the entire snapshot range, even for `SELECT id ... LIMIT 1`. I reproduced roughly 27.4 MB peak incremental allocations from a 24 MiB payload range that the query never selects. There is no cap on the range, so normal CDC/backfill ranges spanning multiple 256 MiB target files can exhaust the process before producing any row. The configured DataFusion budget does not protect it: with `GreedyMemoryPool(0)`, ordinary `SELECT id FROM table` returns ResourcesExhausted, while this function reads all three rows successfully. Please forward the core stream lazily, project batches as they are emitted (and push a supported read projection where possible), and attach the executing TaskContext's reader resources as the ordinary Paimon scan does. Ret
ained output, if any is necessary, must be accounted by its consumer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): support the paimon_incremental_query table function [paimon-rust]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #1042:
URL: https://github.com/apache/paimon-rust/pull/1042#discussion_r4343700844
##########
crates/integrations/datafusion/src/incremental_query.rs:
##########
@@ -0,0 +1,435 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The `paimon_incremental_query` table function reads a snapshot-id range
+//! `(start_exclusive, end_inclusive]` as a batch. Snapshot planning stays in
+//! paimon core; this parses arguments, picks the core planner that matches
+//! Java's `incremental-between` for the table, and streams its output.
+//!
+//! ```sql
+//! SELECT * FROM paimon_incremental_query('db.t', 0, 5);
+//! SELECT * FROM paimon_incremental_query('db.t', 0, 5, 'delta');
+//! SELECT * FROM paimon_incremental_query('db.t$audit_log', 0, 5, 'changelog');
+//! ```
+//!
+//! A `$audit_log` table-name suffix prepends the `rowkind` audit column (and the
+//! optional `_SEQUENCE_NUMBER`), mirroring the core audit-log read.
+
+use std::fmt;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use datafusion::arrow::datatypes::SchemaRef as ArrowSchemaRef;
+use datafusion::catalog::{Session, TableFunctionImpl};
+use datafusion::common::stats::Precision;
+use datafusion::common::{internal_err, project_schema, Statistics};
+use datafusion::datasource::{TableProvider, TableType};
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::logical_expr::Expr;
+use datafusion::physical_expr::EquivalenceProperties;
+use datafusion::physical_plan::empty::EmptyExec;
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
+use datafusion::physical_plan::{
+ DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use datafusion::prelude::SessionContext;
+use futures::{stream, StreamExt, TryStreamExt};
+use paimon::catalog::Catalog;
+use paimon::resource::ResourceContext;
+use paimon::spec::{CoreOptions, DataField};
+use paimon::table::{
+ ArrowRecordBatchStream, AuditLogTable, IncrementalPlan, IncrementalScanMode, IncrementalSplit,
+ Table,
+};
+
+use crate::error::to_datafusion_error;
+use crate::memory::reader_resources;
+use crate::physical_plan::scan::to_datafusion_batch;
+use crate::runtime::{await_with_runtime, block_on_with_runtime};
+use crate::table::datafusion_arrow_schema;
+use crate::table_function_args::{
+ extract_int_literal, extract_string_literal, parse_table_identifier,
+};
+use crate::table_loader::load_data_table_for_read;
+
+const FUNCTION_NAME: &str = "paimon_incremental_query";
+const AUDIT_LOG_SUFFIX: &str = "$audit_log";
+
+/// Registers `paimon_incremental_query` against `catalog`.
+pub(crate) fn register_incremental_query(
+ ctx: &SessionContext,
+ catalog: Arc<dyn Catalog>,
+ default_database: &str,
+) {
+ ctx.register_udtf(
+ FUNCTION_NAME,
+ Arc::new(IncrementalQueryFunction {
+ catalog,
+ default_database: default_database.to_string(),
+ }),
+ );
+}
+
+pub(crate) struct IncrementalQueryFunction {
+ catalog: Arc<dyn Catalog>,
+ default_database: String,
+}
+
+impl fmt::Debug for IncrementalQueryFunction {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.debug_struct("IncrementalQueryFunction")
+ .field("default_database", &self.default_database)
+ .finish()
+ }
+}
+
+/// Parses the optional mode argument. `Auto` resolves from `changelog-producer`
+/// during planning; `Diff` is handled by core's diff planning.
+fn parse_mode(raw: &str) -> DFResult<IncrementalScanMode> {
+ match raw.to_ascii_lowercase().as_str() {
+ "auto" => Ok(IncrementalScanMode::Auto),
+ "delta" => Ok(IncrementalScanMode::Delta),
+ "changelog" => Ok(IncrementalScanMode::Changelog),
+ "diff" => Ok(IncrementalScanMode::Diff),
+ other => Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME}: unknown scan mode '{other}', expected auto, delta, changelog, or diff"
+ ))),
+ }
+}
+
+impl TableFunctionImpl for IncrementalQueryFunction {
+ fn call(&self, args: &[Expr]) -> DFResult<Arc<dyn TableProvider>> {
+ if args.len() < 3 || args.len() > 4 {
+ return Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME} requires 3 or 4 arguments: (table_name, start_snapshot_exclusive, end_snapshot_inclusive [, mode])"
+ )));
+ }
+ let table_name = extract_string_literal(FUNCTION_NAME, &args[0], "table_name")?;
+ let start_exclusive =
+ extract_int_literal(FUNCTION_NAME, &args[1], "start_snapshot_exclusive")?;
+ let end_inclusive = extract_int_literal(FUNCTION_NAME, &args[2], "end_snapshot_inclusive")?;
+ let mode = match args.get(3) {
+ Some(expr) => parse_mode(&extract_string_literal(FUNCTION_NAME, expr, "mode")?)?,
+ None => IncrementalScanMode::Auto,
+ };
+ if end_inclusive < start_exclusive {
+ return Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME}: end_snapshot_inclusive ({end_inclusive}) must be >= start_snapshot_exclusive ({start_exclusive})"
+ )));
+ }
+
+ // `db.t$audit_log` selects the audit-log projection of `db.t`.
+ let (base_name, audit_log) = match table_name.strip_suffix(AUDIT_LOG_SUFFIX) {
+ Some(base) => (base.to_string(), true),
+ None => (table_name.clone(), false),
+ };
+ let identifier = parse_table_identifier(FUNCTION_NAME, &base_name, &self.default_database)?;
+
+ let catalog = Arc::clone(&self.catalog);
+ let table = block_on_with_runtime(
+ async move { load_data_table_for_read(&catalog, &identifier, FUNCTION_NAME).await },
+ "paimon_incremental_query: catalog access thread panicked",
+ )?;
+
+ Ok(Arc::new(IncrementalQueryTableProvider::try_new(
+ table,
+ audit_log,
+ mode,
+ start_exclusive,
+ end_inclusive,
+ )?))
+ }
+}
+#[derive(Debug, Clone)]
+struct IncrementalQueryTableProvider {
+ table: Table,
+ audit_log: bool,
+ mode: IncrementalScanMode,
+ start_exclusive: i64,
+ end_inclusive: i64,
+ fields: Vec<DataField>,
+ schema: ArrowSchemaRef,
+}
+
+impl IncrementalQueryTableProvider {
+ fn try_new(
+ table: Table,
+ audit_log: bool,
+ mode: IncrementalScanMode,
+ start_exclusive: i64,
+ end_inclusive: i64,
+ ) -> DFResult<Self> {
+ // Audit-log output leads with `rowkind` (+ optional `_SEQUENCE_NUMBER`);
+ // the plain projection is the table's own fields.
+ let fields = if audit_log {
+ AuditLogTable::new(table.clone())
+ .fields()
+ .map_err(to_datafusion_error)?
+ } else {
+ table.schema().fields().to_vec()
+ };
+ // Core streams raw Arrow (`Utf8`, not `Utf8View`), so declare the schema
+ // the reader actually yields rather than the view-forced scan schema.
+ let schema = datafusion_arrow_schema(&fields, false)?;
+ Ok(Self {
+ table,
+ audit_log,
+ mode,
+ start_exclusive,
+ end_inclusive,
+ fields,
+ schema,
+ })
+ }
+
+ /// Splits a projection into the table fields core reads and the column of
+ /// core's batches each output column comes from. Core's audit-log batches
+ /// lead with the same audit columns as `fields`, then the read fields.
+ fn read_projection(&self, indices: &[usize]) -> ReadProjection {
+ let audit_columns = self.fields.len() - self.table.schema().fields().len();
+ let mut read_type = Vec::new();
+ let columns = indices
+ .iter()
+ .map(|&index| {
+ if index < audit_columns {
+ return index;
+ }
+ read_type.push(self.fields[index].clone());
+ audit_columns + read_type.len() - 1
+ })
+ .collect();
+ ReadProjection { read_type, columns }
+ }
+}
+
+#[async_trait]
+impl TableProvider for IncrementalQueryTableProvider {
+ fn schema(&self) -> ArrowSchemaRef {
+ Arc::clone(&self.schema)
+ }
+
+ fn table_type(&self) -> TableType {
+ TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &dyn Session,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ limit: Option<usize>,
+ ) -> DFResult<Arc<dyn ExecutionPlan>> {
+ let projected_schema = project_schema(&self.schema, projection)?;
+ // An outer `LIMIT 0` needs no rows.
+ if limit == Some(0) {
+ return Ok(Arc::new(EmptyExec::new(projected_schema)));
+ }
+ Ok(Arc::new(IncrementalQueryExec::new(
+ self.table.clone(),
+ self.audit_log,
+ self.mode,
+ self.start_exclusive,
+ self.end_inclusive,
+ projection.map(|indices| self.read_projection(indices)),
+ projected_schema,
+ )))
+ }
+}
+
+/// The table fields to read, and per output column the column of core's
+/// batches it comes from.
+#[derive(Debug, Clone)]
+struct ReadProjection {
+ read_type: Vec<DataField>,
+ columns: Vec<usize>,
+}
+
+/// Execution-time plan: builds the incremental plan in paimon core and streams
+/// its Arrow output when polled, so planning and `EXPLAIN` stay cheap.
+#[derive(Debug, Clone)]
+struct IncrementalQueryExec {
+ table: Table,
+ audit_log: bool,
+ mode: IncrementalScanMode,
+ start_exclusive: i64,
+ end_inclusive: i64,
+ projection: Option<ReadProjection>,
+ output_schema: ArrowSchemaRef,
+ plan_properties: Arc<PlanProperties>,
+}
+
+impl IncrementalQueryExec {
+ fn new(
+ table: Table,
+ audit_log: bool,
+ mode: IncrementalScanMode,
+ start_exclusive: i64,
+ end_inclusive: i64,
+ projection: Option<ReadProjection>,
+ output_schema: ArrowSchemaRef,
+ ) -> Self {
+ let plan_properties = Arc::new(PlanProperties::new(
+ EquivalenceProperties::new(output_schema.clone()),
+ Partitioning::UnknownPartitioning(1),
+ EmissionType::Incremental,
+ Boundedness::Bounded,
+ ));
+ Self {
+ table,
+ audit_log,
+ mode,
+ start_exclusive,
+ end_inclusive,
+ projection,
+ output_schema,
+ plan_properties,
+ }
+ }
+
+ /// Plans the range in paimon core and opens its batch stream. Reader memory
+ /// comes from the executing task's pool, as in the ordinary scan. The
+ /// projection is pushed into the read: Delta and Changelog decode the
+ /// projected fields, while Diff decodes full rows to compare them and
+ /// projects its output. The stream is `'static`, so it outlives the
+ /// transient read builder it is built from.
+ async fn open(&self, resources: ResourceContext) -> DFResult<ArrowRecordBatchStream> {
+ let mut read_builder = self.table.new_read_builder();
Review Comment:
Fixed in f8016e1: session options go through the `copy_with_time_travel` an ordinary read uses, so `blob-as-descriptor` and `read.batch-size` apply and descriptors resolve (regression added). Options choosing another range (time-travel selectors, `incremental-between*`, creation-time filters, non-default `scan.mode`) are rejected, whether the session or the table read sets them. Immutable overrides apply as in ordinary SELECT: after `SET 'paimon.data-evolution.enabled'='false'` both return NULL-id fragments. Java's `checkImmutability` rejects them. Reject in core, or keep the stored value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): support the paimon_incremental_query table function [paimon-rust]
Posted by "jackylee-ch (via GitHub) via issues" <is...@paimon.apache.org>.
jackylee-ch commented on code in PR #1042:
URL: https://github.com/apache/paimon-rust/pull/1042#discussion_r4444418450
##########
crates/paimon/src/table/data_evolution_reader.rs:
##########
@@ -267,6 +267,160 @@ impl DataEvolutionReader {
}
}
+ /// Whether a row-id group of change events merges into rows: its files
+ /// are column slices of one row range, and the source plan reads each
+ /// field from a file that holds it, in the one layout the group stores it
+ /// in. Other groups are read file by file.
+ async fn merges_change_events(&self, files: &[DataFileMeta]) -> crate::Result<bool> {
+ if !is_data_evolution_column_slice_group(files) {
+ return Ok(false);
+ }
+ // The source plan reads a BLOB field from dedicated files only, or,
+ // for descriptor and view fields, from normal files only, whichever
+ // layout the field had when the files were written.
+ let dedicated_blobs: HashSet<i32> = self
+ .table_fields
+ .iter()
+ .filter(|field| {
+ field.data_type().is_blob_file_field()
+ && !self.blob_descriptor_fields.contains(field.name())
+ && !self.blob_view_fields.contains(field.name())
+ })
+ .map(DataField::id)
+ .collect();
+ let (infos, _) = load_file_infos(
+ &self.schema_manager,
+ self.table_schema_id,
+ &self.table_fields,
+ files,
+ )
+ .await?;
+ let mut layouts = Vec::with_capacity(files.len());
+ // Normal files with the fields their write columns name.
+ let mut normal_files = Vec::new();
+ for (file, info) in files.iter().zip(infos) {
+ if is_vector_store_file_name(&file.file_name) {
+ let columns = info.normalized_write_cols.unwrap_or_default();
+ layouts.push((
+ FieldLayout::Vector(vector_bunch_key(file, &columns)),
+ info.field_ids,
+ ));
+ } else if is_blob_or_video_file_name(&file.file_name) {
+ layouts.push((FieldLayout::Blob, info.field_ids));
+ } else {
+ normal_files.push((file, info.field_ids));
+ }
+ }
+ // Only the layouts of the table's BLOB and VECTOR fields can decide:
+ // the merge reads any other field from normal files alone, a field
+ // retyped from VECTOR to text too, and does not read a field dropped
+ // since. So normal layouts list only those fields, while dedicated
+ // layouts keep every field, as the source plan rejects a field of two
+ // vector bunches, read or not. A normal file without write columns
+ // leaves out the fields its own schema stores in dedicated files; that
+ // schema is read only where the answer can decide: for a BLOB or
+ // VECTOR field that a dedicated file of the group also holds or that
+ // the table now stores in BLOB files.
+ let maybe_dedicated: HashSet<i32> = self
+ .table_fields
+ .iter()
+ .filter(|field| {
+ field.data_type().is_blob_file_field()
+ || matches!(field.data_type(), DataType::Vector(_))
+ })
+ .map(DataField::id)
+ .collect();
+ let in_dedicated_files: HashSet<i32> = layouts
+ .iter()
+ .flat_map(|(_, field_ids)| field_ids.iter().copied())
+ .collect();
+ for (file, written) in &normal_files {
+ let decides = file.write_cols.is_none()
+ && written.iter().any(|id| {
+ maybe_dedicated.contains(id)
+ && (in_dedicated_files.contains(id) || dedicated_blobs.contains(id))
+ });
+ let held = if decides {
+ self.fields_without_write_cols(file).await?
Review Comment:
Fixed in 5147e44: a file without write columns now lacks only the columns its own commit wrote to dedicated files; when that can't decide the merge, the group is read file by file. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): support the paimon_incremental_query table function [paimon-rust]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #1042:
URL: https://github.com/apache/paimon-rust/pull/1042#discussion_r4282909585
##########
crates/integrations/datafusion/src/incremental_query.rs:
##########
@@ -0,0 +1,365 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The `paimon_incremental_query` table function reads a snapshot-id range
+//! `(start_exclusive, end_inclusive]` as a batch. Snapshot planning stays in
+//! paimon core; this only parses arguments and streams the core plan's output.
+//!
+//! ```sql
+//! SELECT * FROM paimon_incremental_query('db.t', 0, 5);
+//! SELECT * FROM paimon_incremental_query('db.t', 0, 5, 'delta');
+//! SELECT * FROM paimon_incremental_query('db.t$audit_log', 0, 5, 'changelog');
+//! ```
+//!
+//! A `$audit_log` table-name suffix prepends the `rowkind` audit column (and the
+//! optional `_SEQUENCE_NUMBER`), mirroring the core audit-log read.
+
+use std::fmt;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use datafusion::arrow::datatypes::SchemaRef as ArrowSchemaRef;
+use datafusion::arrow::record_batch::RecordBatch;
+use datafusion::catalog::{Session, TableFunctionImpl};
+use datafusion::common::stats::Precision;
+use datafusion::common::{internal_err, project_schema, Statistics};
+use datafusion::datasource::{TableProvider, TableType};
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::logical_expr::Expr;
+use datafusion::physical_expr::EquivalenceProperties;
+use datafusion::physical_plan::empty::EmptyExec;
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
+use datafusion::physical_plan::{
+ DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use datafusion::prelude::SessionContext;
+use futures::{stream, TryStreamExt};
+use paimon::catalog::Catalog;
+use paimon::table::{AuditLogTable, IncrementalScanMode, Table};
+
+use crate::error::to_datafusion_error;
+use crate::runtime::{await_with_runtime, block_on_with_runtime};
+use crate::table::datafusion_arrow_schema;
+use crate::table_function_args::{
+ extract_int_literal, extract_string_literal, parse_table_identifier,
+};
+use crate::table_loader::load_data_table_for_read;
+
+const FUNCTION_NAME: &str = "paimon_incremental_query";
+const AUDIT_LOG_SUFFIX: &str = "$audit_log";
+
+/// Registers `paimon_incremental_query` against `catalog`.
+pub(crate) fn register_incremental_query(
+ ctx: &SessionContext,
+ catalog: Arc<dyn Catalog>,
+ default_database: &str,
+) {
+ ctx.register_udtf(
+ FUNCTION_NAME,
+ Arc::new(IncrementalQueryFunction {
+ catalog,
+ default_database: default_database.to_string(),
+ }),
+ );
+}
+
+pub(crate) struct IncrementalQueryFunction {
+ catalog: Arc<dyn Catalog>,
+ default_database: String,
+}
+
+impl fmt::Debug for IncrementalQueryFunction {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.debug_struct("IncrementalQueryFunction")
+ .field("default_database", &self.default_database)
+ .finish()
+ }
+}
+
+/// Parses the optional mode argument. `Auto` resolves from `changelog-producer`
+/// during planning; `Diff` is handled by core's diff planning.
+fn parse_mode(raw: &str) -> DFResult<IncrementalScanMode> {
+ match raw.to_ascii_lowercase().as_str() {
+ "auto" => Ok(IncrementalScanMode::Auto),
+ "delta" => Ok(IncrementalScanMode::Delta),
+ "changelog" => Ok(IncrementalScanMode::Changelog),
+ "diff" => Ok(IncrementalScanMode::Diff),
+ other => Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME}: unknown scan mode '{other}', expected auto, delta, changelog, or diff"
+ ))),
+ }
+}
+
+impl TableFunctionImpl for IncrementalQueryFunction {
+ fn call(&self, args: &[Expr]) -> DFResult<Arc<dyn TableProvider>> {
+ if args.len() < 3 || args.len() > 4 {
+ return Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME} requires 3 or 4 arguments: (table_name, start_snapshot_exclusive, end_snapshot_inclusive [, mode])"
+ )));
+ }
+ let table_name = extract_string_literal(FUNCTION_NAME, &args[0], "table_name")?;
+ let start_exclusive =
+ extract_int_literal(FUNCTION_NAME, &args[1], "start_snapshot_exclusive")?;
+ let end_inclusive = extract_int_literal(FUNCTION_NAME, &args[2], "end_snapshot_inclusive")?;
+ let mode = match args.get(3) {
+ Some(expr) => parse_mode(&extract_string_literal(FUNCTION_NAME, expr, "mode")?)?,
+ None => IncrementalScanMode::Auto,
+ };
+ if end_inclusive < start_exclusive {
+ return Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME}: end_snapshot_inclusive ({end_inclusive}) must be >= start_snapshot_exclusive ({start_exclusive})"
+ )));
+ }
+
+ // `db.t$audit_log` selects the audit-log projection of `db.t`.
+ let (base_name, audit_log) = match table_name.strip_suffix(AUDIT_LOG_SUFFIX) {
+ Some(base) => (base.to_string(), true),
+ None => (table_name.clone(), false),
+ };
+ let identifier = parse_table_identifier(FUNCTION_NAME, &base_name, &self.default_database)?;
+
+ let catalog = Arc::clone(&self.catalog);
+ let table = block_on_with_runtime(
+ async move { load_data_table_for_read(&catalog, &identifier, FUNCTION_NAME).await },
+ "paimon_incremental_query: catalog access thread panicked",
+ )?;
+
+ Ok(Arc::new(IncrementalQueryTableProvider::try_new(
+ table,
+ audit_log,
+ mode,
+ start_exclusive,
+ end_inclusive,
+ )?))
+ }
+}
+#[derive(Debug, Clone)]
+struct IncrementalQueryTableProvider {
+ table: Table,
+ audit_log: bool,
+ mode: IncrementalScanMode,
+ start_exclusive: i64,
+ end_inclusive: i64,
+ schema: ArrowSchemaRef,
+}
+
+impl IncrementalQueryTableProvider {
+ fn try_new(
+ table: Table,
+ audit_log: bool,
+ mode: IncrementalScanMode,
+ start_exclusive: i64,
+ end_inclusive: i64,
+ ) -> DFResult<Self> {
+ // Audit-log output leads with `rowkind` (+ optional `_SEQUENCE_NUMBER`);
+ // the plain projection is the table's own fields.
+ let fields = if audit_log {
+ AuditLogTable::new(table.clone())
+ .fields()
+ .map_err(to_datafusion_error)?
+ } else {
+ table.schema().fields().to_vec()
+ };
+ // Core streams raw Arrow (`Utf8`, not `Utf8View`), so declare the schema
+ // the reader actually yields rather than the view-forced scan schema.
+ let schema = datafusion_arrow_schema(&fields, false)?;
+ Ok(Self {
+ table,
+ audit_log,
+ mode,
+ start_exclusive,
+ end_inclusive,
+ schema,
+ })
+ }
+}
+
+#[async_trait]
+impl TableProvider for IncrementalQueryTableProvider {
+ fn schema(&self) -> ArrowSchemaRef {
+ Arc::clone(&self.schema)
+ }
+
+ fn table_type(&self) -> TableType {
+ TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &dyn Session,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ limit: Option<usize>,
+ ) -> DFResult<Arc<dyn ExecutionPlan>> {
+ let projected_schema = project_schema(&self.schema, projection)?;
+ // An outer `LIMIT 0` needs no rows.
+ if limit == Some(0) {
+ return Ok(Arc::new(EmptyExec::new(projected_schema)));
+ }
+ Ok(Arc::new(IncrementalQueryExec::new(
+ self.table.clone(),
+ self.audit_log,
+ self.mode,
+ self.start_exclusive,
+ self.end_inclusive,
+ projection.cloned(),
+ projected_schema,
+ )))
+ }
+}
+/// Execution-time plan: builds the incremental plan in paimon core and streams
+/// its Arrow output when polled, so planning and `EXPLAIN` stay cheap.
+#[derive(Debug, Clone)]
+struct IncrementalQueryExec {
+ table: Table,
+ audit_log: bool,
+ mode: IncrementalScanMode,
+ start_exclusive: i64,
+ end_inclusive: i64,
+ projection: Option<Vec<usize>>,
+ output_schema: ArrowSchemaRef,
+ plan_properties: Arc<PlanProperties>,
+}
+
+impl IncrementalQueryExec {
+ fn new(
+ table: Table,
+ audit_log: bool,
+ mode: IncrementalScanMode,
+ start_exclusive: i64,
+ end_inclusive: i64,
+ projection: Option<Vec<usize>>,
+ output_schema: ArrowSchemaRef,
+ ) -> Self {
+ let plan_properties = Arc::new(PlanProperties::new(
+ EquivalenceProperties::new(output_schema.clone()),
+ Partitioning::UnknownPartitioning(1),
+ EmissionType::Incremental,
+ Boundedness::Bounded,
+ ));
+ Self {
+ table,
+ audit_log,
+ mode,
+ start_exclusive,
+ end_inclusive,
+ projection,
+ output_schema,
+ plan_properties,
+ }
+ }
+
+ async fn compute_batches(&self) -> DFResult<Vec<RecordBatch>> {
+ // Core owns snapshot planning; its stream is `'static`, so it outlives
+ // the transient read builder it is built from.
+ let batches = await_with_runtime(async {
+ let read_builder = self.table.new_read_builder();
+ let scan = read_builder.new_incremental_scan(
+ self.mode,
+ self.start_exclusive,
+ self.end_inclusive,
+ );
+ let plan = scan.plan().await.map_err(to_datafusion_error)?;
+ let read = read_builder.new_read().map_err(to_datafusion_error)?;
+ let mut stream = if self.audit_log {
+ read.to_audit_log_arrow(&plan)
+ } else {
+ read.to_incremental_arrow(&plan)
+ }
+ .map_err(to_datafusion_error)?;
+ let mut batches = Vec::new();
+ while let Some(batch) = stream.try_next().await.map_err(to_datafusion_error)? {
+ batches.push(batch);
Review Comment:
Fixed in 9568181: `IncrementalQueryExec` now forwards core's stream batch by batch, attaches `reader_resources(&context, partition)` like the ordinary scan, and pushes the projection via `with_read_type` (Diff still decodes full rows). Core's `to_incremental_arrow` read files raw, splitting data-evolution BLOB/VECTOR columns into separate rows and returning primary-key BLOB descriptors; it now decodes them as `to_arrow` does. A data-evolution range is planned as one batch, as in Java's `incremental-between`. Tests cover the zero-byte pool, `SELECT id` over 1 MiB payloads and these reads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): support the paimon_incremental_query table function [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1042:
URL: https://github.com/apache/paimon-rust/pull/1042#discussion_r4290811593
##########
crates/integrations/datafusion/src/incremental_query.rs:
##########
@@ -0,0 +1,435 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The `paimon_incremental_query` table function reads a snapshot-id range
+//! `(start_exclusive, end_inclusive]` as a batch. Snapshot planning stays in
+//! paimon core; this parses arguments, picks the core planner that matches
+//! Java's `incremental-between` for the table, and streams its output.
+//!
+//! ```sql
+//! SELECT * FROM paimon_incremental_query('db.t', 0, 5);
+//! SELECT * FROM paimon_incremental_query('db.t', 0, 5, 'delta');
+//! SELECT * FROM paimon_incremental_query('db.t$audit_log', 0, 5, 'changelog');
+//! ```
+//!
+//! A `$audit_log` table-name suffix prepends the `rowkind` audit column (and the
+//! optional `_SEQUENCE_NUMBER`), mirroring the core audit-log read.
+
+use std::fmt;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use datafusion::arrow::datatypes::SchemaRef as ArrowSchemaRef;
+use datafusion::catalog::{Session, TableFunctionImpl};
+use datafusion::common::stats::Precision;
+use datafusion::common::{internal_err, project_schema, Statistics};
+use datafusion::datasource::{TableProvider, TableType};
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::logical_expr::Expr;
+use datafusion::physical_expr::EquivalenceProperties;
+use datafusion::physical_plan::empty::EmptyExec;
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
+use datafusion::physical_plan::{
+ DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use datafusion::prelude::SessionContext;
+use futures::{stream, StreamExt, TryStreamExt};
+use paimon::catalog::Catalog;
+use paimon::resource::ResourceContext;
+use paimon::spec::{CoreOptions, DataField};
+use paimon::table::{
+ ArrowRecordBatchStream, AuditLogTable, IncrementalPlan, IncrementalScanMode, IncrementalSplit,
+ Table,
+};
+
+use crate::error::to_datafusion_error;
+use crate::memory::reader_resources;
+use crate::physical_plan::scan::to_datafusion_batch;
+use crate::runtime::{await_with_runtime, block_on_with_runtime};
+use crate::table::datafusion_arrow_schema;
+use crate::table_function_args::{
+ extract_int_literal, extract_string_literal, parse_table_identifier,
+};
+use crate::table_loader::load_data_table_for_read;
+
+const FUNCTION_NAME: &str = "paimon_incremental_query";
+const AUDIT_LOG_SUFFIX: &str = "$audit_log";
+
+/// Registers `paimon_incremental_query` against `catalog`.
+pub(crate) fn register_incremental_query(
+ ctx: &SessionContext,
+ catalog: Arc<dyn Catalog>,
+ default_database: &str,
+) {
+ ctx.register_udtf(
+ FUNCTION_NAME,
+ Arc::new(IncrementalQueryFunction {
+ catalog,
+ default_database: default_database.to_string(),
+ }),
+ );
+}
+
+pub(crate) struct IncrementalQueryFunction {
+ catalog: Arc<dyn Catalog>,
+ default_database: String,
+}
+
+impl fmt::Debug for IncrementalQueryFunction {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.debug_struct("IncrementalQueryFunction")
+ .field("default_database", &self.default_database)
+ .finish()
+ }
+}
+
+/// Parses the optional mode argument. `Auto` resolves from `changelog-producer`
+/// during planning; `Diff` is handled by core's diff planning.
+fn parse_mode(raw: &str) -> DFResult<IncrementalScanMode> {
+ match raw.to_ascii_lowercase().as_str() {
+ "auto" => Ok(IncrementalScanMode::Auto),
+ "delta" => Ok(IncrementalScanMode::Delta),
+ "changelog" => Ok(IncrementalScanMode::Changelog),
+ "diff" => Ok(IncrementalScanMode::Diff),
+ other => Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME}: unknown scan mode '{other}', expected auto, delta, changelog, or diff"
+ ))),
+ }
+}
+
+impl TableFunctionImpl for IncrementalQueryFunction {
+ fn call(&self, args: &[Expr]) -> DFResult<Arc<dyn TableProvider>> {
+ if args.len() < 3 || args.len() > 4 {
+ return Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME} requires 3 or 4 arguments: (table_name, start_snapshot_exclusive, end_snapshot_inclusive [, mode])"
+ )));
+ }
+ let table_name = extract_string_literal(FUNCTION_NAME, &args[0], "table_name")?;
+ let start_exclusive =
+ extract_int_literal(FUNCTION_NAME, &args[1], "start_snapshot_exclusive")?;
+ let end_inclusive = extract_int_literal(FUNCTION_NAME, &args[2], "end_snapshot_inclusive")?;
+ let mode = match args.get(3) {
+ Some(expr) => parse_mode(&extract_string_literal(FUNCTION_NAME, expr, "mode")?)?,
+ None => IncrementalScanMode::Auto,
+ };
+ if end_inclusive < start_exclusive {
+ return Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME}: end_snapshot_inclusive ({end_inclusive}) must be >= start_snapshot_exclusive ({start_exclusive})"
+ )));
+ }
+
+ // `db.t$audit_log` selects the audit-log projection of `db.t`.
+ let (base_name, audit_log) = match table_name.strip_suffix(AUDIT_LOG_SUFFIX) {
+ Some(base) => (base.to_string(), true),
+ None => (table_name.clone(), false),
+ };
+ let identifier = parse_table_identifier(FUNCTION_NAME, &base_name, &self.default_database)?;
+
+ let catalog = Arc::clone(&self.catalog);
Review Comment:
[P1] Honor the explicit catalog instead of silently reading another warehouse
The UDTF captures one catalog, and each `register_catalog` replaces the same globally registered function. Meanwhile `parse_table_identifier` discards the first component of a three-part name. I reproduced with two warehouses registered as `cat_a` and `cat_b`, both containing `default.t`: ordinary fully qualified SELECT returns [1] from cat_a and [2] from cat_b, but `paimon_incremental_query('cat_a.default.t', 0, 1)` and the cat_b call both return [2]. A supported, explicitly qualified table name thus silently reads the wrong data source. Please resolve the catalog component against the session's catalog registry, or at least reject a component that does not match the bound catalog rather than discarding it. Add the two-warehouse regression; successful single-catalog reads do not cover this.
##########
crates/paimon/src/table/table_read.rs:
##########
@@ -466,10 +466,21 @@ impl<'a> PaimonTableRead<'a> {
}
}
}
- // Delta / Changelog rows are read as-is from planned files (no full-table
- // merge against historical base versions).
+ // Delta / Changelog rows are read as-is from planned files, without a merge
+ // against historical base versions. Files are still decoded as an ordinary
+ // read decodes them: primary-key BLOB descriptors resolve to payloads, and
+ // data-evolution files in a split that hold column slices of the same rows
+ // merge by row id.
let data_splits = self.matching_splits(&data_splits)?;
- self.new_data_file_reader()?.read(&data_splits)
+ let core_options = self.table.schema().core_options();
+ if !self.table.schema().primary_keys().is_empty() {
+ return self
+ .resolving_primary_key_blobs(&core_options, |read| read.read_raw(&data_splits));
+ }
+ if core_options.data_evolution_enabled() {
+ return self.read_with_evolution(&data_splits, &core_options);
Review Comment:
[P2] Allow absent base columns when merging an update-only incremental range
The new ordinary evolution-reader path also applies its full-table missing-provider check to partial incremental data. With a valid DE table `(id INT NOT NULL, name STRING, value INT)`, snapshot 1 inserts two rows, snapshot 2 MERGE-updates name, and snapshot 3 MERGE-updates value. I reproduced `paimon_incremental_query('default.de', 1, 3)` failing with `Cannot read non-nullable field 'id' without a provider`. The range legitimately contains the two partial column files but no original id file. The single-update range (1,2] succeeds and NULL-fills id, as this PR's documentation promises for columns outside the range; adding the second partial file switches into the merge path and fails. Please allow the incremental evolution merge to NULL-fill fields absent from the selected range, including logically NOT NULL fields, while retaining the ordinary full-table integrity check. Cover the update-only range in auto/delta and plain/audit reads; changing the DataFusion Arrow schema alone w
ill not fix the reader's validation.
##########
crates/integrations/datafusion/src/incremental_query.rs:
##########
@@ -0,0 +1,435 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The `paimon_incremental_query` table function reads a snapshot-id range
+//! `(start_exclusive, end_inclusive]` as a batch. Snapshot planning stays in
+//! paimon core; this parses arguments, picks the core planner that matches
+//! Java's `incremental-between` for the table, and streams its output.
+//!
+//! ```sql
+//! SELECT * FROM paimon_incremental_query('db.t', 0, 5);
+//! SELECT * FROM paimon_incremental_query('db.t', 0, 5, 'delta');
+//! SELECT * FROM paimon_incremental_query('db.t$audit_log', 0, 5, 'changelog');
+//! ```
+//!
+//! A `$audit_log` table-name suffix prepends the `rowkind` audit column (and the
+//! optional `_SEQUENCE_NUMBER`), mirroring the core audit-log read.
+
+use std::fmt;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use datafusion::arrow::datatypes::SchemaRef as ArrowSchemaRef;
+use datafusion::catalog::{Session, TableFunctionImpl};
+use datafusion::common::stats::Precision;
+use datafusion::common::{internal_err, project_schema, Statistics};
+use datafusion::datasource::{TableProvider, TableType};
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::logical_expr::Expr;
+use datafusion::physical_expr::EquivalenceProperties;
+use datafusion::physical_plan::empty::EmptyExec;
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
+use datafusion::physical_plan::{
+ DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use datafusion::prelude::SessionContext;
+use futures::{stream, StreamExt, TryStreamExt};
+use paimon::catalog::Catalog;
+use paimon::resource::ResourceContext;
+use paimon::spec::{CoreOptions, DataField};
+use paimon::table::{
+ ArrowRecordBatchStream, AuditLogTable, IncrementalPlan, IncrementalScanMode, IncrementalSplit,
+ Table,
+};
+
+use crate::error::to_datafusion_error;
+use crate::memory::reader_resources;
+use crate::physical_plan::scan::to_datafusion_batch;
+use crate::runtime::{await_with_runtime, block_on_with_runtime};
+use crate::table::datafusion_arrow_schema;
+use crate::table_function_args::{
+ extract_int_literal, extract_string_literal, parse_table_identifier,
+};
+use crate::table_loader::load_data_table_for_read;
+
+const FUNCTION_NAME: &str = "paimon_incremental_query";
+const AUDIT_LOG_SUFFIX: &str = "$audit_log";
+
+/// Registers `paimon_incremental_query` against `catalog`.
+pub(crate) fn register_incremental_query(
+ ctx: &SessionContext,
+ catalog: Arc<dyn Catalog>,
+ default_database: &str,
+) {
+ ctx.register_udtf(
+ FUNCTION_NAME,
+ Arc::new(IncrementalQueryFunction {
+ catalog,
+ default_database: default_database.to_string(),
+ }),
+ );
+}
+
+pub(crate) struct IncrementalQueryFunction {
+ catalog: Arc<dyn Catalog>,
+ default_database: String,
+}
+
+impl fmt::Debug for IncrementalQueryFunction {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.debug_struct("IncrementalQueryFunction")
+ .field("default_database", &self.default_database)
+ .finish()
+ }
+}
+
+/// Parses the optional mode argument. `Auto` resolves from `changelog-producer`
+/// during planning; `Diff` is handled by core's diff planning.
+fn parse_mode(raw: &str) -> DFResult<IncrementalScanMode> {
+ match raw.to_ascii_lowercase().as_str() {
+ "auto" => Ok(IncrementalScanMode::Auto),
+ "delta" => Ok(IncrementalScanMode::Delta),
+ "changelog" => Ok(IncrementalScanMode::Changelog),
+ "diff" => Ok(IncrementalScanMode::Diff),
+ other => Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME}: unknown scan mode '{other}', expected auto, delta, changelog, or diff"
+ ))),
+ }
+}
+
+impl TableFunctionImpl for IncrementalQueryFunction {
+ fn call(&self, args: &[Expr]) -> DFResult<Arc<dyn TableProvider>> {
+ if args.len() < 3 || args.len() > 4 {
+ return Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME} requires 3 or 4 arguments: (table_name, start_snapshot_exclusive, end_snapshot_inclusive [, mode])"
+ )));
+ }
+ let table_name = extract_string_literal(FUNCTION_NAME, &args[0], "table_name")?;
+ let start_exclusive =
+ extract_int_literal(FUNCTION_NAME, &args[1], "start_snapshot_exclusive")?;
+ let end_inclusive = extract_int_literal(FUNCTION_NAME, &args[2], "end_snapshot_inclusive")?;
+ let mode = match args.get(3) {
+ Some(expr) => parse_mode(&extract_string_literal(FUNCTION_NAME, expr, "mode")?)?,
+ None => IncrementalScanMode::Auto,
+ };
+ if end_inclusive < start_exclusive {
+ return Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME}: end_snapshot_inclusive ({end_inclusive}) must be >= start_snapshot_exclusive ({start_exclusive})"
+ )));
+ }
+
+ // `db.t$audit_log` selects the audit-log projection of `db.t`.
+ let (base_name, audit_log) = match table_name.strip_suffix(AUDIT_LOG_SUFFIX) {
+ Some(base) => (base.to_string(), true),
+ None => (table_name.clone(), false),
+ };
+ let identifier = parse_table_identifier(FUNCTION_NAME, &base_name, &self.default_database)?;
+
+ let catalog = Arc::clone(&self.catalog);
+ let table = block_on_with_runtime(
+ async move { load_data_table_for_read(&catalog, &identifier, FUNCTION_NAME).await },
+ "paimon_incremental_query: catalog access thread panicked",
+ )?;
+
+ Ok(Arc::new(IncrementalQueryTableProvider::try_new(
+ table,
+ audit_log,
+ mode,
+ start_exclusive,
+ end_inclusive,
+ )?))
+ }
+}
+#[derive(Debug, Clone)]
+struct IncrementalQueryTableProvider {
+ table: Table,
+ audit_log: bool,
+ mode: IncrementalScanMode,
+ start_exclusive: i64,
+ end_inclusive: i64,
+ fields: Vec<DataField>,
+ schema: ArrowSchemaRef,
+}
+
+impl IncrementalQueryTableProvider {
+ fn try_new(
+ table: Table,
+ audit_log: bool,
+ mode: IncrementalScanMode,
+ start_exclusive: i64,
+ end_inclusive: i64,
+ ) -> DFResult<Self> {
+ // Audit-log output leads with `rowkind` (+ optional `_SEQUENCE_NUMBER`);
+ // the plain projection is the table's own fields.
+ let fields = if audit_log {
+ AuditLogTable::new(table.clone())
+ .fields()
+ .map_err(to_datafusion_error)?
+ } else {
+ table.schema().fields().to_vec()
+ };
+ // Core streams raw Arrow (`Utf8`, not `Utf8View`), so declare the schema
+ // the reader actually yields rather than the view-forced scan schema.
+ let schema = datafusion_arrow_schema(&fields, false)?;
+ Ok(Self {
+ table,
+ audit_log,
+ mode,
+ start_exclusive,
+ end_inclusive,
+ fields,
+ schema,
+ })
+ }
+
+ /// Splits a projection into the table fields core reads and the column of
+ /// core's batches each output column comes from. Core's audit-log batches
+ /// lead with the same audit columns as `fields`, then the read fields.
+ fn read_projection(&self, indices: &[usize]) -> ReadProjection {
+ let audit_columns = self.fields.len() - self.table.schema().fields().len();
+ let mut read_type = Vec::new();
+ let columns = indices
+ .iter()
+ .map(|&index| {
+ if index < audit_columns {
+ return index;
+ }
+ read_type.push(self.fields[index].clone());
+ audit_columns + read_type.len() - 1
+ })
+ .collect();
+ ReadProjection { read_type, columns }
+ }
+}
+
+#[async_trait]
+impl TableProvider for IncrementalQueryTableProvider {
+ fn schema(&self) -> ArrowSchemaRef {
+ Arc::clone(&self.schema)
+ }
+
+ fn table_type(&self) -> TableType {
+ TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &dyn Session,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ limit: Option<usize>,
+ ) -> DFResult<Arc<dyn ExecutionPlan>> {
+ let projected_schema = project_schema(&self.schema, projection)?;
+ // An outer `LIMIT 0` needs no rows.
+ if limit == Some(0) {
+ return Ok(Arc::new(EmptyExec::new(projected_schema)));
+ }
+ Ok(Arc::new(IncrementalQueryExec::new(
+ self.table.clone(),
+ self.audit_log,
+ self.mode,
+ self.start_exclusive,
+ self.end_inclusive,
+ projection.map(|indices| self.read_projection(indices)),
+ projected_schema,
+ )))
+ }
+}
+
+/// The table fields to read, and per output column the column of core's
+/// batches it comes from.
+#[derive(Debug, Clone)]
+struct ReadProjection {
+ read_type: Vec<DataField>,
+ columns: Vec<usize>,
+}
+
+/// Execution-time plan: builds the incremental plan in paimon core and streams
+/// its Arrow output when polled, so planning and `EXPLAIN` stay cheap.
+#[derive(Debug, Clone)]
+struct IncrementalQueryExec {
+ table: Table,
+ audit_log: bool,
+ mode: IncrementalScanMode,
+ start_exclusive: i64,
+ end_inclusive: i64,
+ projection: Option<ReadProjection>,
+ output_schema: ArrowSchemaRef,
+ plan_properties: Arc<PlanProperties>,
+}
+
+impl IncrementalQueryExec {
+ fn new(
+ table: Table,
+ audit_log: bool,
+ mode: IncrementalScanMode,
+ start_exclusive: i64,
+ end_inclusive: i64,
+ projection: Option<ReadProjection>,
+ output_schema: ArrowSchemaRef,
+ ) -> Self {
+ let plan_properties = Arc::new(PlanProperties::new(
+ EquivalenceProperties::new(output_schema.clone()),
+ Partitioning::UnknownPartitioning(1),
+ EmissionType::Incremental,
+ Boundedness::Bounded,
+ ));
+ Self {
+ table,
+ audit_log,
+ mode,
+ start_exclusive,
+ end_inclusive,
+ projection,
+ output_schema,
+ plan_properties,
+ }
+ }
+
+ /// Plans the range in paimon core and opens its batch stream. Reader memory
+ /// comes from the executing task's pool, as in the ordinary scan. The
+ /// projection is pushed into the read: Delta and Changelog decode the
+ /// projected fields, while Diff decodes full rows to compare them and
+ /// projects its output. The stream is `'static`, so it outlives the
+ /// transient read builder it is built from.
+ async fn open(&self, resources: ResourceContext) -> DFResult<ArrowRecordBatchStream> {
+ let mut read_builder = self.table.new_read_builder();
Review Comment:
[P2] Apply session BLOB read options before opening the range
The new function retains only the catalog-loaded Table and never receives the SQLContext's dynamic options. This affects actual output semantics, not just snapshot-selection settings: after `SET 'paimon.blob-as-descriptor' = 'true'`, I wrote a primary-key BLOB table and compared the same snapshot through ordinary SELECT and this UDTF. The ordinary scan returns a 177-byte descriptor, while the incremental call returns the three payload bytes `AAA`. A caller explicitly requesting descriptors still fetches/receives the full payload through the new entry point. Please propagate relevant session read options into the table/read builder, with an explicit policy for selectors that conflict with the function's range, and add the ordinary-vs-incremental descriptor regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): support the paimon_incremental_query table function [paimon-rust]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #1042:
URL: https://github.com/apache/paimon-rust/pull/1042#discussion_r4343698905
##########
crates/paimon/src/table/table_read.rs:
##########
@@ -466,10 +466,21 @@ impl<'a> PaimonTableRead<'a> {
}
}
}
- // Delta / Changelog rows are read as-is from planned files (no full-table
- // merge against historical base versions).
+ // Delta / Changelog rows are read as-is from planned files, without a merge
+ // against historical base versions. Files are still decoded as an ordinary
+ // read decodes them: primary-key BLOB descriptors resolve to payloads, and
+ // data-evolution files in a split that hold column slices of the same rows
+ // merge by row id.
let data_splits = self.matching_splits(&data_splits)?;
- self.new_data_file_reader()?.read(&data_splits)
+ let core_options = self.table.schema().core_options();
+ if !self.table.schema().primary_keys().is_empty() {
+ return self
+ .resolving_primary_key_blobs(&core_options, |read| read.read_raw(&data_splits));
+ }
+ if core_options.data_evolution_enabled() {
+ return self.read_with_evolution(&data_splits, &core_options);
Review Comment:
Fixed in f8016e1: for a split of change events (`DataSplit::is_streaming`), the merge reader reads a field no file of the group wrote as NULL, NOT NULL and nested ones included; snapshot reads keep the check. This covers auto/delta and plain/audit here and core's other change-event reads (`plan_combined_delta` + `to_arrow`, `incremental-between-timestamp`, stream follow-ups). Java rejects this case; the SQL guide says so. A merge group of only `.blob`/`.vector.` files (e.g. a range that only rewrites a VECTOR column) still fails with "requires at least one normal data file", as on main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): support the paimon_incremental_query table function [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1042:
URL: https://github.com/apache/paimon-rust/pull/1042#discussion_r4345235098
##########
crates/paimon/src/table/table_read.rs:
##########
@@ -776,10 +776,21 @@ impl<'a> PaimonTableRead<'a> {
}
}
}
- // Delta / Changelog rows are read as-is from planned files (no full-table
- // merge against historical base versions).
+ // Delta / Changelog rows are read as-is from planned files, without a merge
+ // against historical base versions. Files are still decoded as an ordinary
+ // read decodes them: primary-key BLOB descriptors resolve to payloads, and
+ // data-evolution files in a split that hold column slices of the same rows
+ // merge by row id.
let data_splits = self.matching_splits(&data_splits)?;
- self.new_data_file_reader()?.read(&data_splits)
+ let core_options = self.table.schema().core_options();
+ if !self.table.schema().primary_keys().is_empty() {
+ return self
+ .resolving_primary_key_blobs(&core_options, |read| read.read_raw(&data_splits));
+ }
+ if core_options.data_evolution_enabled() {
+ return self.read_with_evolution(&data_splits, &core_options);
Review Comment:
[P2] Keep dedicated-only VECTOR updates readable incrementally
This unconditional DE routing regresses a valid existing to_incremental_arrow call. With row tracking, data evolution and vector.file.format=parquet enabled, the public data-evolution writer updating only embedding intentionally commits a single *.vector.parquet file. Delta (1,2] therefore contains only that streaming file. On this head the new evolution-reader path errors with "Field merge split with .blob/.vector. files requires at least one normal data file"; the identical public-API probe passes on base 8d08e48a and returns the rewritten vectors with the unwritten id column NULL. The ordinary evolution reader's pre-existing dedicated-only limitation does not apply to the baseline raw incremental reader. Preserve dedicated-only change-event decoding, or support its row-range anchor in the merge reader, without borrowing historical files outside the requested range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): support the paimon_incremental_query table function [paimon-rust]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #1042:
URL: https://github.com/apache/paimon-rust/pull/1042#discussion_r4343697620
##########
crates/integrations/datafusion/src/incremental_query.rs:
##########
@@ -0,0 +1,435 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The `paimon_incremental_query` table function reads a snapshot-id range
+//! `(start_exclusive, end_inclusive]` as a batch. Snapshot planning stays in
+//! paimon core; this parses arguments, picks the core planner that matches
+//! Java's `incremental-between` for the table, and streams its output.
+//!
+//! ```sql
+//! SELECT * FROM paimon_incremental_query('db.t', 0, 5);
+//! SELECT * FROM paimon_incremental_query('db.t', 0, 5, 'delta');
+//! SELECT * FROM paimon_incremental_query('db.t$audit_log', 0, 5, 'changelog');
+//! ```
+//!
+//! A `$audit_log` table-name suffix prepends the `rowkind` audit column (and the
+//! optional `_SEQUENCE_NUMBER`), mirroring the core audit-log read.
+
+use std::fmt;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use datafusion::arrow::datatypes::SchemaRef as ArrowSchemaRef;
+use datafusion::catalog::{Session, TableFunctionImpl};
+use datafusion::common::stats::Precision;
+use datafusion::common::{internal_err, project_schema, Statistics};
+use datafusion::datasource::{TableProvider, TableType};
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::logical_expr::Expr;
+use datafusion::physical_expr::EquivalenceProperties;
+use datafusion::physical_plan::empty::EmptyExec;
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
+use datafusion::physical_plan::{
+ DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use datafusion::prelude::SessionContext;
+use futures::{stream, StreamExt, TryStreamExt};
+use paimon::catalog::Catalog;
+use paimon::resource::ResourceContext;
+use paimon::spec::{CoreOptions, DataField};
+use paimon::table::{
+ ArrowRecordBatchStream, AuditLogTable, IncrementalPlan, IncrementalScanMode, IncrementalSplit,
+ Table,
+};
+
+use crate::error::to_datafusion_error;
+use crate::memory::reader_resources;
+use crate::physical_plan::scan::to_datafusion_batch;
+use crate::runtime::{await_with_runtime, block_on_with_runtime};
+use crate::table::datafusion_arrow_schema;
+use crate::table_function_args::{
+ extract_int_literal, extract_string_literal, parse_table_identifier,
+};
+use crate::table_loader::load_data_table_for_read;
+
+const FUNCTION_NAME: &str = "paimon_incremental_query";
+const AUDIT_LOG_SUFFIX: &str = "$audit_log";
+
+/// Registers `paimon_incremental_query` against `catalog`.
+pub(crate) fn register_incremental_query(
+ ctx: &SessionContext,
+ catalog: Arc<dyn Catalog>,
+ default_database: &str,
+) {
+ ctx.register_udtf(
+ FUNCTION_NAME,
+ Arc::new(IncrementalQueryFunction {
+ catalog,
+ default_database: default_database.to_string(),
+ }),
+ );
+}
+
+pub(crate) struct IncrementalQueryFunction {
+ catalog: Arc<dyn Catalog>,
+ default_database: String,
+}
+
+impl fmt::Debug for IncrementalQueryFunction {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.debug_struct("IncrementalQueryFunction")
+ .field("default_database", &self.default_database)
+ .finish()
+ }
+}
+
+/// Parses the optional mode argument. `Auto` resolves from `changelog-producer`
+/// during planning; `Diff` is handled by core's diff planning.
+fn parse_mode(raw: &str) -> DFResult<IncrementalScanMode> {
+ match raw.to_ascii_lowercase().as_str() {
+ "auto" => Ok(IncrementalScanMode::Auto),
+ "delta" => Ok(IncrementalScanMode::Delta),
+ "changelog" => Ok(IncrementalScanMode::Changelog),
+ "diff" => Ok(IncrementalScanMode::Diff),
+ other => Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME}: unknown scan mode '{other}', expected auto, delta, changelog, or diff"
+ ))),
+ }
+}
+
+impl TableFunctionImpl for IncrementalQueryFunction {
+ fn call(&self, args: &[Expr]) -> DFResult<Arc<dyn TableProvider>> {
+ if args.len() < 3 || args.len() > 4 {
+ return Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME} requires 3 or 4 arguments: (table_name, start_snapshot_exclusive, end_snapshot_inclusive [, mode])"
+ )));
+ }
+ let table_name = extract_string_literal(FUNCTION_NAME, &args[0], "table_name")?;
+ let start_exclusive =
+ extract_int_literal(FUNCTION_NAME, &args[1], "start_snapshot_exclusive")?;
+ let end_inclusive = extract_int_literal(FUNCTION_NAME, &args[2], "end_snapshot_inclusive")?;
+ let mode = match args.get(3) {
+ Some(expr) => parse_mode(&extract_string_literal(FUNCTION_NAME, expr, "mode")?)?,
+ None => IncrementalScanMode::Auto,
+ };
+ if end_inclusive < start_exclusive {
+ return Err(DataFusionError::Plan(format!(
+ "{FUNCTION_NAME}: end_snapshot_inclusive ({end_inclusive}) must be >= start_snapshot_exclusive ({start_exclusive})"
+ )));
+ }
+
+ // `db.t$audit_log` selects the audit-log projection of `db.t`.
+ let (base_name, audit_log) = match table_name.strip_suffix(AUDIT_LOG_SUFFIX) {
+ Some(base) => (base.to_string(), true),
+ None => (table_name.clone(), false),
+ };
+ let identifier = parse_table_identifier(FUNCTION_NAME, &base_name, &self.default_database)?;
+
+ let catalog = Arc::clone(&self.catalog);
Review Comment:
Fixed in f8016e1: the function now resolves the name in the calling session as a `FROM` clause does (`t` and `db.t` take the current catalog and database, `cat.db.t` names the catalog) and looks the catalog up in the session's catalog list; an unknown or non-Paimon catalog is an error. Added the two-warehouse regression, checked against ordinary SELECT for 1-, 2- and 3-part names, after `USE`, and with a database named like the other catalog. `vector_search`, `full_text_search`, `hybrid_search` and `blob_view` still bind the catalog registered last; this PR leaves them unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): support the paimon_incremental_query table function [paimon-rust]
Posted by "JingsongLi (via GitHub) via issues" <is...@paimon.apache.org>.
JingsongLi commented on PR #1042:
URL: https://github.com/apache/paimon-rust/pull/1042#issuecomment-6908179557
The PR is still too large; please break it down to make it easier to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): support the paimon_incremental_query table function [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1042:
URL: https://github.com/apache/paimon-rust/pull/1042#discussion_r4386203017
##########
crates/paimon/src/table/data_evolution_reader.rs:
##########
@@ -2459,6 +2520,42 @@ fn resolve_field_ids(file: &DataFileMeta, fields: &[DataField]) -> crate::Result
}
}
+/// The key of the vector bunch a vector file joins: its schema, format and
+/// normalized write columns.
+fn vector_bunch_key(
+ file: &DataFileMeta,
+ normalized_write_cols: &[String],
+) -> (i64, String, Vec<String>) {
+ (
+ file.schema_id,
+ vector_format_suffix(&file.file_name),
+ normalized_write_cols.to_vec(),
+ )
+}
+
+/// Whether some VECTOR field is written by files of two vector bunch keys,
+/// such as a rewrite after `ALTER TABLE`; the source plan cannot combine
+/// their bunches.
+fn vector_field_has_two_layouts(files: &[DataFileMeta], infos: &[ResolvedFileInfo]) -> bool {
+ let mut layouts = HashMap::new();
+ for (file, info) in files.iter().zip(infos) {
+ let Some(columns) = info
+ .normalized_write_cols
+ .as_deref()
+ .filter(|_| is_vector_store_file_name(&file.file_name))
Review Comment:
[P1] Include inline VECTOR providers in the layout fallback
This filter ignores normal Parquet files that actually carry an inline VECTOR column. Removing vector.file.format is accepted: after an initial dedicated-vector write, a subsequent embedding rewrite produces a newer normal partial-column file covering the same row IDs. With just the old .vector.parquet counted here, the combined group is approved for merging, but build_source_plan prefers the dedicated provider and silently drops the new inline embedding. I reproduced this through public write, persisted RemoveOption, rewrite, combined Delta scan and projected read APIs: the new [9,9] update disappears on this head, while the unchanged test retains it on exact base ecbf5806. The ordinary snapshot-reader limitation pre-exists; this PR newly exposes it to to_incremental_arrow by routing that API through the evolution reader. Include actual inline VECTOR providers when deciding whether layouts can merge, and use the existing file-by-file fallback for mixed inline/dedicated groups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): support the paimon_incremental_query table function [paimon-rust]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #1042:
URL: https://github.com/apache/paimon-rust/pull/1042#discussion_r4420096355
##########
crates/paimon/src/table/data_evolution_reader.rs:
##########
@@ -2459,6 +2520,42 @@ fn resolve_field_ids(file: &DataFileMeta, fields: &[DataField]) -> crate::Result
}
}
+/// The key of the vector bunch a vector file joins: its schema, format and
+/// normalized write columns.
+fn vector_bunch_key(
+ file: &DataFileMeta,
+ normalized_write_cols: &[String],
+) -> (i64, String, Vec<String>) {
+ (
+ file.schema_id,
+ vector_format_suffix(&file.file_name),
+ normalized_write_cols.to_vec(),
+ )
+}
+
+/// Whether some VECTOR field is written by files of two vector bunch keys,
+/// such as a rewrite after `ALTER TABLE`; the source plan cannot combine
+/// their bunches.
+fn vector_field_has_two_layouts(files: &[DataFileMeta], infos: &[ResolvedFileInfo]) -> bool {
+ let mut layouts = HashMap::new();
+ for (file, info) in files.iter().zip(infos) {
+ let Some(columns) = info
+ .normalized_write_cols
+ .as_deref()
+ .filter(|_| is_vector_store_file_name(&file.file_name))
Review Comment:
Fixed in 60fcd30: change events merge a row-id group only if the merge reads each BLOB or VECTOR column where it is stored; otherwise they decode the group file by file. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): support the paimon_incremental_query table function [paimon-rust]
Posted by "JingsongLi (via GitHub) via issues" <is...@paimon.apache.org>.
JingsongLi commented on code in PR #1042:
URL: https://github.com/apache/paimon-rust/pull/1042#discussion_r4435303331
##########
crates/paimon/src/table/data_evolution_reader.rs:
##########
@@ -267,6 +267,160 @@ impl DataEvolutionReader {
}
}
+ /// Whether a row-id group of change events merges into rows: its files
+ /// are column slices of one row range, and the source plan reads each
+ /// field from a file that holds it, in the one layout the group stores it
+ /// in. Other groups are read file by file.
+ async fn merges_change_events(&self, files: &[DataFileMeta]) -> crate::Result<bool> {
+ if !is_data_evolution_column_slice_group(files) {
+ return Ok(false);
+ }
+ // The source plan reads a BLOB field from dedicated files only, or,
+ // for descriptor and view fields, from normal files only, whichever
+ // layout the field had when the files were written.
+ let dedicated_blobs: HashSet<i32> = self
+ .table_fields
+ .iter()
+ .filter(|field| {
+ field.data_type().is_blob_file_field()
+ && !self.blob_descriptor_fields.contains(field.name())
+ && !self.blob_view_fields.contains(field.name())
+ })
+ .map(DataField::id)
+ .collect();
+ let (infos, _) = load_file_infos(
+ &self.schema_manager,
+ self.table_schema_id,
+ &self.table_fields,
+ files,
+ )
+ .await?;
+ let mut layouts = Vec::with_capacity(files.len());
+ // Normal files with the fields their write columns name.
+ let mut normal_files = Vec::new();
+ for (file, info) in files.iter().zip(infos) {
+ if is_vector_store_file_name(&file.file_name) {
+ let columns = info.normalized_write_cols.unwrap_or_default();
+ layouts.push((
+ FieldLayout::Vector(vector_bunch_key(file, &columns)),
+ info.field_ids,
+ ));
+ } else if is_blob_or_video_file_name(&file.file_name) {
+ layouts.push((FieldLayout::Blob, info.field_ids));
+ } else {
+ normal_files.push((file, info.field_ids));
+ }
+ }
+ // Only the layouts of the table's BLOB and VECTOR fields can decide:
+ // the merge reads any other field from normal files alone, a field
+ // retyped from VECTOR to text too, and does not read a field dropped
+ // since. So normal layouts list only those fields, while dedicated
+ // layouts keep every field, as the source plan rejects a field of two
+ // vector bunches, read or not. A normal file without write columns
+ // leaves out the fields its own schema stores in dedicated files; that
+ // schema is read only where the answer can decide: for a BLOB or
+ // VECTOR field that a dedicated file of the group also holds or that
+ // the table now stores in BLOB files.
+ let maybe_dedicated: HashSet<i32> = self
+ .table_fields
+ .iter()
+ .filter(|field| {
+ field.data_type().is_blob_file_field()
+ || matches!(field.data_type(), DataType::Vector(_))
+ })
+ .map(DataField::id)
+ .collect();
+ let in_dedicated_files: HashSet<i32> = layouts
+ .iter()
+ .flat_map(|(_, field_ids)| field_ids.iter().copied())
+ .collect();
+ for (file, written) in &normal_files {
+ let decides = file.write_cols.is_none()
+ && written.iter().any(|id| {
+ maybe_dedicated.contains(id)
+ && (in_dedicated_files.contains(id) || dedicated_blobs.contains(id))
+ });
+ let held = if decides {
+ self.fields_without_write_cols(file).await?
Review Comment:
[P1] Do not infer physical VECTOR coverage from stored schema options
A file's stored schema does not establish the dynamic options that wrote it. `Table::copy_with_options` can change `vector.file.format` without changing the schema ID, and the public data-evolution writer uses those options. With write-column optimization enabled, a full normal-column rewrite can legitimately have `write_cols=None` even though its physical VECTOR layout differs from that stored schema. This call then omits the inline VECTOR from the normal-file layout, approves merging, and the source plan selects the older dedicated vector instead.
I reproduced this entirely through public APIs: persist id+VECTOR with `vector.file.format=parquet` and `data-evolution.write-cols-optimization.enabled=true`; insert two rows; copy the table with `vector.file.format=""`; update row 0's id and embedding to [9,9] using `new_data_evolution_writer([id,embedding])`. The new normal Parquet file has the original schema ID and `write_cols=None`. Combined Delta (0,2] through `to_incremental_arrow` on this head returns only [1,0] and [0,1], silently dropping the committed [9,9] event. The identical probe passes on exact base 421e6125 and retains [9,9]. No file metadata was manually modified.
The ordinary snapshot-reader limitation pre-exists; this PR newly exposes it to the existing incremental API. Establish actual physical coverage from durable write metadata/physical fields, or use file-by-file fallback when absent write columns cannot prove it. Please cover dynamic writer format overrides as well as persisted ALTER transitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): support the paimon_incremental_query table function [paimon-rust]
Posted by "jackylee-ch (via GitHub) via issues" <is...@paimon.apache.org>.
jackylee-ch commented on PR #1042:
URL: https://github.com/apache/paimon-rust/pull/1042#issuecomment-6788286188
Merged main in dc58c6d, with c62cb6d on top: `plan_merge` now refuses an incremental window on the table the native planner reads, naming RESET when the session set it, so the bare SessionContext is covered too; `IncrementalQueryExec` moves to the DataFusion 55 plan API and reports scan metrics. The UPDATE/DELETE/MERGE window tests pass and fail without the `plan_merge` check; core 4,858/0, DataFusion 1,210/0, clippy clean. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org