You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2026/10/16 06:35:25 UTC
Re: [PR] docs(rfc): add RFC-112 for Spark declarative pipelines [hudi]
nsivabalan commented on code in PR #19997:
URL: https://github.com/apache/hudi/pull/19997#discussion_r4291597595
##########
rfc/rfc-112/rfc-112.md:
##########
@@ -0,0 +1,429 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-112: Support Spark Declarative Pipelines
+
+## Proposers
+
+- @cshuo
+- @danny0405
+
+## Approvers
+
+- TBD
+
+## Status
+
+Issue: TBD
+
+## Abstract
+
+[Spark Declarative Pipelines (SDP)](https://spark.apache.org/docs/4.1.1/declarative-pipelines-programming-guide.html) lets users define tables and transformations while Spark builds the dependency graph and runs the batch and streaming flows. Hudi provides atomic commits, keyed updates, event ordering, deletes, incremental queries, and table services.
+
+This RFC makes Hudi a supported SDP source and target. It covers catalog-based table creation, materialized views, streaming tables, basic Hudi-to-Hudi incremental reads, and keyed upserts through Hudi's existing writer.
+
+SDP and Hudi retain their existing responsibilities:
+
+- SDP owns dataset declarations, dependency analysis, orchestration, flow checkpoints, retries, and execution order.
+- Hudi owns table storage, commit atomicity, record reconciliation, incremental read semantics, and table services.
+
+Users can build the same pipelines today with Spark jobs, Hudi writers or Hudi Streamer, and an external orchestrator. This integration removes much of that glue code while keeping the outputs as ordinary Hudi tables.
+
+## Background
+
+### Spark Declarative Pipelines
+
+Apache Spark 4.1 introduced SDP for declarative batch and streaming ETL. SDP persists two main dataset types:
+
+- A **materialized view**, which has exactly one batch flow that computes a table.
+- A **streaming table**, which has one or more streaming flows that append their output to a table.
+
+Users define pipelines in Python or SQL. SDP validates dataset references, builds the dataflow graph, runs independent flows in parallel, and stores Structured Streaming checkpoints in the pipeline storage location.
+
+In the Spark 4.1 implementation, a streaming table flow uses a Structured Streaming append-mode write to `DataStreamWriter.toTable`, and a batch table flow uses append-mode `DataFrameWriter.saveAsTable`. For an existing materialized view, SDP issues `TRUNCATE TABLE`, calls `catalog.alterTable` to reconcile schema and properties, then runs the batch flow. See Spark's [`FlowExecution`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala) and [`DatasetManager`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala).
+
+These table operations are the storage-format integration boundary; this RFC does not add a Hudi-specific execution engine.
+
+### Relevant Hudi capabilities
+
+Hudi already provides the required data-plane primitives:
+
+- `HoodieCatalog` can create and load Hudi tables through Spark's catalog APIs.
+- The Hudi batch writer supports bulk insert, insert, upsert, insert overwrite, and table insert overwrite operations. An upsert can carry delete-marker records.
+- `HoodieStreamingSink` writes Structured Streaming micro-batches and records checkpoint metadata used to avoid recommitting an already committed batch after retry.
+- Hudi's streaming source and incremental query modes can expose records changed since a prior instant.
+
+The integration exposes these capabilities through SDP's existing table and flow abstractions. Hudi Streamer, the DataFrame writer, Spark SQL DML, and direct Structured Streaming APIs remain unchanged.
+
+### Current gaps
+
+`format="hudi"` is not yet an end-to-end supported SDP contract. Some operations may already work, but table creation, refresh, restart, upsert configuration, and Hudi-to-Hudi chaining are neither documented nor covered by an integration suite.
+
+Materialized-view refresh needs end-to-end validation. SDP 4.1 truncates the target before reconciling its schema and writing the recomputed result. Hudi's current full-table `TRUNCATE TABLE` deletes the table path, including the timeline, and reinitializes its properties. This RFC reuses that behavior and documents its limitations; a timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+The integration must also address:
+
+- SDP table properties and flow-scoped write options must be resolved and passed to the Hudi writer.
+- Streaming restart must coordinate the SDP/Spark checkpoint with Hudi's sink checkpoint metadata.
+- A Hudi source used through `readStream.table` must preserve incremental offset semantics.
+- SDP must carry Hudi's persisted record-key, precombine, and record-merger settings into the existing upsert path. Deletes remain upsert input rows marked by `_hoodie_is_deleted`; Spark append output mode does not replace this contract.
+
+### Related work
+
+- Hudi's Spark 4.1 support ([issue #17663](https://github.com/apache/hudi/issues/17663), [PR #17674](https://github.com/apache/hudi/pull/17674)) provides the runtime baseline, but not an SDP contract.
+- DataSource V2 read work ([RFC-98](../rfc-98/rfc-98.md), [issue #15023](https://github.com/apache/hudi/issues/15023)) may improve the catalog read path but is not required for the initial integration.
+- The [SQL insert-overwrite discussion](https://github.com/apache/hudi/discussions/13815) shows why refresh semantics must be explicit rather than inferred from an append call.
+- [Issue #13973](https://github.com/apache/hudi/issues/13973) shows that a latest-state incremental read cannot fully propagate hard deletes. Flows that require complete delete propagation can use Hudi's existing CDC incremental format.
+- [Issue #5537](https://github.com/apache/hudi/issues/5537) tracks named-catalog support. The first version uses `HoodieCatalog` as `spark_catalog`.
+
+## Motivation
+
+### Representative use case
+
+A typical order pipeline looks like this:
+
+```text
+MySQL/PostgreSQL CDC or Kafka
+ |
+ v
+ bronze_order_events (Hudi)
+ |
+ v
+ current_orders (Hudi)
+ |
+ v
+ daily_order_metrics (Hudi)
+```
+
+The bronze table stores incoming events. The current-state table uses a record key and precombine field to reconcile duplicates and out-of-order events, including delete-marker records. Downstream streaming tables consume changed records incrementally, while batch aggregates can be materialized as views.
+
+Today this usually means several Spark applications plus custom handling for table creation, write options, checkpoints, dependencies, and restarts. With SDP, the graph lives in one project and the persisted datasets remain regular Hudi tables that other engines can query.
+
+Related user request issues include:
+
+- #19281: [multi-dataset incremental processing](https://github.com/apache/hudi/issues/19281)
+- #5189: [chained incremental Hudi tables](https://github.com/apache/hudi/issues/5189)
+- #13973: [downstream delete propagation](https://github.com/apache/hudi/issues/13973).
+
+### User value
+
+1. **Declarative pipelines.** Users define datasets as SQL or DataFrame transformations; SDP manages dependencies, checkpoints, and Hudi writes.
+2. **Managed incremental composition.** Hudi already provides incremental reads; SDP composes them into a dependency-aware pipeline, managing execution order, checkpoints, and recovery without separate Spark jobs or hand-written orchestration.
+3. **Open outputs.** Each dataset remains a standard Hudi table that can be queried independently of SDP by any Hudi-compatible engine.
+
+SDP does not make arbitrary joins or aggregates incremental; performance benefits apply only to flows that can process changes incrementally.
+
+## Goals
+
+1. Create Hudi tables from SDP datasets through `HoodieCatalog`.
+2. Write materialized views with Hudi `bulk_insert` or `insert`, following SDP's truncate-and-recompute refresh sequence.
+3. Write SDP streaming tables through Hudi's Structured Streaming sink.
+4. Support a basic incremental Hudi-to-Hudi streaming-table chain.
+5. Expose Hudi's existing keyed upsert path through SDP, including record-key, precombine, record-merger, and delete-marker handling.
+6. Keep non-SDP Hudi behavior unchanged.
+
+## Non-Goals
+
+The first version excludes:
+
+- A complete Hudi DataSource V2 read or streaming-write implementation.
+- Atomic materialized-view refresh across truncate, schema reconciliation, and the batch write.
+- Changes to Hudi's full-table truncate semantics. Timeline-preserving truncate can be developed independently.
+- Cross-table transactions across multiple SDP targets.
+- End-to-end exactly-once delivery across an arbitrary external source and Hudi. Guarantees remain scoped to Spark/Hudi micro-batch retries at one target.
+- Arbitrary named catalogs for `HoodieCatalog`; the first version uses Hudi as `spark_catalog`.
+
+## Proposed User Experience
+
+The examples below implement the representative order pipeline in dependency order, using Hudi's existing SQL table options and `SET` for flow-scoped write options.
+
+### Pipeline configuration
+
+An SDP project configures the Hudi extension and catalog like any other Hudi Spark application:
+
+```yaml
+name: hudi_orders_pipeline
+libraries:
+ - glob:
+ include: transformations/**
+storage: s3://pipelines/checkpoints/hudi_orders_pipeline
+catalog: spark_catalog
+database: lakehouse
+configuration:
+ spark.sql.extensions: org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.sql.catalog.spark_catalog: org.apache.spark.sql.hudi.catalog.HoodieCatalog
Review Comment:
**🚨** Pipeline configuration
The `configuration:` block cannot carry these settings. SDP applies it as *session* conf over Spark Connect (client `conf.set` + `CreateDataflowGraph.sql_conf`), and Spark rejects every static conf here:
```
[CANNOT_MODIFY_STATIC_CONFIG] Cannot modify the value of the static Spark config: "spark.sql.extensions"
[CANNOT_MODIFY_CONFIG] Cannot modify the value of the Spark config: "spark.serializer"
```
(same for `spark.kryo.registrator`, `spark.jars`, `spark.sql.warehouse.dir`). `--conf` on `spark-pipelines run` does not reach the auto-launched `--remote local` server either.
Suggest a "Connect server prerequisites" subsection: extension, `HoodieCatalog`, Kryo registrator and the bundle are set on the Connect server (`spark-submit --class org.apache.spark.sql.connect.service.SparkConnectServer …`), and the example invocation becomes `spark-pipelines run --remote sc://host:15002`.
##########
rfc/rfc-112/rfc-112.md:
##########
@@ -0,0 +1,429 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-112: Support Spark Declarative Pipelines
+
+## Proposers
+
+- @cshuo
+- @danny0405
+
+## Approvers
+
+- TBD
+
+## Status
+
+Issue: TBD
+
+## Abstract
+
+[Spark Declarative Pipelines (SDP)](https://spark.apache.org/docs/4.1.1/declarative-pipelines-programming-guide.html) lets users define tables and transformations while Spark builds the dependency graph and runs the batch and streaming flows. Hudi provides atomic commits, keyed updates, event ordering, deletes, incremental queries, and table services.
+
+This RFC makes Hudi a supported SDP source and target. It covers catalog-based table creation, materialized views, streaming tables, basic Hudi-to-Hudi incremental reads, and keyed upserts through Hudi's existing writer.
+
+SDP and Hudi retain their existing responsibilities:
+
+- SDP owns dataset declarations, dependency analysis, orchestration, flow checkpoints, retries, and execution order.
+- Hudi owns table storage, commit atomicity, record reconciliation, incremental read semantics, and table services.
+
+Users can build the same pipelines today with Spark jobs, Hudi writers or Hudi Streamer, and an external orchestrator. This integration removes much of that glue code while keeping the outputs as ordinary Hudi tables.
+
+## Background
+
+### Spark Declarative Pipelines
+
+Apache Spark 4.1 introduced SDP for declarative batch and streaming ETL. SDP persists two main dataset types:
+
+- A **materialized view**, which has exactly one batch flow that computes a table.
+- A **streaming table**, which has one or more streaming flows that append their output to a table.
+
+Users define pipelines in Python or SQL. SDP validates dataset references, builds the dataflow graph, runs independent flows in parallel, and stores Structured Streaming checkpoints in the pipeline storage location.
+
+In the Spark 4.1 implementation, a streaming table flow uses a Structured Streaming append-mode write to `DataStreamWriter.toTable`, and a batch table flow uses append-mode `DataFrameWriter.saveAsTable`. For an existing materialized view, SDP issues `TRUNCATE TABLE`, calls `catalog.alterTable` to reconcile schema and properties, then runs the batch flow. See Spark's [`FlowExecution`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala) and [`DatasetManager`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala).
+
+These table operations are the storage-format integration boundary; this RFC does not add a Hudi-specific execution engine.
+
+### Relevant Hudi capabilities
+
+Hudi already provides the required data-plane primitives:
+
+- `HoodieCatalog` can create and load Hudi tables through Spark's catalog APIs.
+- The Hudi batch writer supports bulk insert, insert, upsert, insert overwrite, and table insert overwrite operations. An upsert can carry delete-marker records.
+- `HoodieStreamingSink` writes Structured Streaming micro-batches and records checkpoint metadata used to avoid recommitting an already committed batch after retry.
+- Hudi's streaming source and incremental query modes can expose records changed since a prior instant.
+
+The integration exposes these capabilities through SDP's existing table and flow abstractions. Hudi Streamer, the DataFrame writer, Spark SQL DML, and direct Structured Streaming APIs remain unchanged.
+
+### Current gaps
+
+`format="hudi"` is not yet an end-to-end supported SDP contract. Some operations may already work, but table creation, refresh, restart, upsert configuration, and Hudi-to-Hudi chaining are neither documented nor covered by an integration suite.
+
+Materialized-view refresh needs end-to-end validation. SDP 4.1 truncates the target before reconciling its schema and writing the recomputed result. Hudi's current full-table `TRUNCATE TABLE` deletes the table path, including the timeline, and reinitializes its properties. This RFC reuses that behavior and documents its limitations; a timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+The integration must also address:
+
+- SDP table properties and flow-scoped write options must be resolved and passed to the Hudi writer.
+- Streaming restart must coordinate the SDP/Spark checkpoint with Hudi's sink checkpoint metadata.
+- A Hudi source used through `readStream.table` must preserve incremental offset semantics.
+- SDP must carry Hudi's persisted record-key, precombine, and record-merger settings into the existing upsert path. Deletes remain upsert input rows marked by `_hoodie_is_deleted`; Spark append output mode does not replace this contract.
+
+### Related work
+
+- Hudi's Spark 4.1 support ([issue #17663](https://github.com/apache/hudi/issues/17663), [PR #17674](https://github.com/apache/hudi/pull/17674)) provides the runtime baseline, but not an SDP contract.
+- DataSource V2 read work ([RFC-98](../rfc-98/rfc-98.md), [issue #15023](https://github.com/apache/hudi/issues/15023)) may improve the catalog read path but is not required for the initial integration.
+- The [SQL insert-overwrite discussion](https://github.com/apache/hudi/discussions/13815) shows why refresh semantics must be explicit rather than inferred from an append call.
+- [Issue #13973](https://github.com/apache/hudi/issues/13973) shows that a latest-state incremental read cannot fully propagate hard deletes. Flows that require complete delete propagation can use Hudi's existing CDC incremental format.
+- [Issue #5537](https://github.com/apache/hudi/issues/5537) tracks named-catalog support. The first version uses `HoodieCatalog` as `spark_catalog`.
+
+## Motivation
+
+### Representative use case
+
+A typical order pipeline looks like this:
+
+```text
+MySQL/PostgreSQL CDC or Kafka
+ |
+ v
+ bronze_order_events (Hudi)
+ |
+ v
+ current_orders (Hudi)
+ |
+ v
+ daily_order_metrics (Hudi)
+```
+
+The bronze table stores incoming events. The current-state table uses a record key and precombine field to reconcile duplicates and out-of-order events, including delete-marker records. Downstream streaming tables consume changed records incrementally, while batch aggregates can be materialized as views.
+
+Today this usually means several Spark applications plus custom handling for table creation, write options, checkpoints, dependencies, and restarts. With SDP, the graph lives in one project and the persisted datasets remain regular Hudi tables that other engines can query.
+
+Related user request issues include:
+
+- #19281: [multi-dataset incremental processing](https://github.com/apache/hudi/issues/19281)
+- #5189: [chained incremental Hudi tables](https://github.com/apache/hudi/issues/5189)
+- #13973: [downstream delete propagation](https://github.com/apache/hudi/issues/13973).
+
+### User value
+
+1. **Declarative pipelines.** Users define datasets as SQL or DataFrame transformations; SDP manages dependencies, checkpoints, and Hudi writes.
+2. **Managed incremental composition.** Hudi already provides incremental reads; SDP composes them into a dependency-aware pipeline, managing execution order, checkpoints, and recovery without separate Spark jobs or hand-written orchestration.
+3. **Open outputs.** Each dataset remains a standard Hudi table that can be queried independently of SDP by any Hudi-compatible engine.
+
+SDP does not make arbitrary joins or aggregates incremental; performance benefits apply only to flows that can process changes incrementally.
+
+## Goals
+
+1. Create Hudi tables from SDP datasets through `HoodieCatalog`.
+2. Write materialized views with Hudi `bulk_insert` or `insert`, following SDP's truncate-and-recompute refresh sequence.
+3. Write SDP streaming tables through Hudi's Structured Streaming sink.
+4. Support a basic incremental Hudi-to-Hudi streaming-table chain.
+5. Expose Hudi's existing keyed upsert path through SDP, including record-key, precombine, record-merger, and delete-marker handling.
+6. Keep non-SDP Hudi behavior unchanged.
+
+## Non-Goals
+
+The first version excludes:
+
+- A complete Hudi DataSource V2 read or streaming-write implementation.
+- Atomic materialized-view refresh across truncate, schema reconciliation, and the batch write.
+- Changes to Hudi's full-table truncate semantics. Timeline-preserving truncate can be developed independently.
+- Cross-table transactions across multiple SDP targets.
+- End-to-end exactly-once delivery across an arbitrary external source and Hudi. Guarantees remain scoped to Spark/Hudi micro-batch retries at one target.
+- Arbitrary named catalogs for `HoodieCatalog`; the first version uses Hudi as `spark_catalog`.
+
+## Proposed User Experience
+
+The examples below implement the representative order pipeline in dependency order, using Hudi's existing SQL table options and `SET` for flow-scoped write options.
+
+### Pipeline configuration
+
+An SDP project configures the Hudi extension and catalog like any other Hudi Spark application:
+
+```yaml
+name: hudi_orders_pipeline
+libraries:
+ - glob:
+ include: transformations/**
+storage: s3://pipelines/checkpoints/hudi_orders_pipeline
+catalog: spark_catalog
+database: lakehouse
+configuration:
+ spark.sql.extensions: org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.sql.catalog.spark_catalog: org.apache.spark.sql.hudi.catalog.HoodieCatalog
+```
+
+The SQL below assumes that `order_events_source` is an external streaming source containing parsed Kafka or database CDC events. Each `CREATE ... AS SELECT` statement defines both the target dataset and its single flow.
+
+### Streaming table with an append flow
+
+```sql
+SET spark.hoodie.datasource.write.operation = insert;
+
+CREATE STREAMING TABLE bronze_order_events
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor'
+)
+AS
+SELECT *
+FROM STREAM order_events_source;
+```
+
+SDP owns the checkpoint location and trigger; Hudi commits each micro-batch.
+
+### Streaming table with keyed upserts
+
+```sql
+SET spark.hoodie.datasource.write.operation = upsert;
+
+CREATE STREAMING TABLE current_orders
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor',
+ 'primaryKey' = 'order_id',
+ 'preCombineField' = 'event_sequence'
+)
+AS
+SELECT
+ *,
+ operation = 'DELETE' AS _hoodie_is_deleted
+FROM STREAM bronze_order_events;
+```
+
+Spark append output mode only controls how each micro-batch reaches the sink; it does not make the Hudi target append-only. `HoodieStreamingSink` still runs the configured Hudi operation, which defaults to `upsert`.
+
+For an existing Hudi table, SDP loads the record key, precombine field, partition fields, and record-merger settings from the table configuration. For a new SDP table, these settings are declared as table properties and persisted by Hudi. SDP does not implement its own deduplication, ordering, merge, or delete path. It validates the schema and configuration, then passes each micro-batch to the existing Hudi writer.
+
+### Batch materialized view
+
+```sql
+SET spark.hoodie.datasource.write.operation = bulk_insert;
+
+CREATE MATERIALIZED VIEW daily_order_metrics
+USING hudi
+PARTITIONED BY (order_date)
+TBLPROPERTIES (
+ 'type' = 'cow'
+)
+AS
+SELECT
+ order_date,
+ state,
+ COUNT(*) AS order_count
+FROM current_orders
+GROUP BY order_date, state;
+```
+
+The first run creates the table and writes the result using `bulk_insert`. Later runs truncate the target, reconcile its schema and properties, and write the recomputed result. With Hudi's current truncate implementation, each refresh removes the previous data and timeline history.
+
+## Semantics
+
+### Dataset-to-Hudi operation mapping
+
+| SDP dataset/flow | Spark 4.1 execution shape | Proposed Hudi semantics |
+| ----------------------------------- | --------------------------------------------- | -------------------------------------------------------- |
+| New materialized view | Batch append to newly created table | Hudi `bulk_insert` or `insert` |
+| Existing materialized view | Truncate, alter, then batch append | Existing Hudi truncate, then `bulk_insert` or `insert` |
+| Streaming table with an append flow | Append-mode micro-batches | Hudi `insert` or `bulk_insert`, configured per flow |
+| Hudi upsert target | Append-mode micro-batches | Existing Hudi `upsert` |
+| Hudi streaming source | Structured Streaming read with SDP checkpoint | Basic Hudi latest-state incremental source |
+
+### Materialized-view write mapping
+
+Initial materialization uses Hudi `bulk_insert` or `insert`. For an existing materialized view, Spark 4.1 executes `TRUNCATE TABLE`, reconciles schema and properties through `catalog.alterTable`, then writes the recomputed result through append-mode `saveAsTable`. Hudi uses the same configured write operation after truncate. An empty result leaves the target empty.
+
+These steps are not a single transaction. Readers may see an empty target during refresh. If schema reconciliation or the batch flow fails after truncate, the previous result is not restored, and completed catalog changes remain in effect.
+
+The initial integration reuses Hudi's current full-table truncate, which deletes old data files and timeline history. This can disrupt concurrent readers and invalidate existing incremental offsets. Timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
Review Comment:
**⚠️, design** reusing the destructive truncate
Two measured consequences that argue against "independent improvement, not a prerequisite":
1. On `--full-refresh` the sequence is: new checkpoint dir → `TRUNCATE TABLE` → `alterTable` → write. In my run `alterTable` failed after the truncate; result: 9 commits → 0, savepoints/history gone, empty table, failed run.
2. L279 relies on the `rm -rf` to "remove … sink checkpoint metadata". That is the *only* reason `canSkipBatch` (`HoodieStreamingSink.scala:308`, keyed on `hoodie.datasource.write.streaming.checkpoint.identifier` → last batch id; observed `{"default_single_writer":"5"}`) does not skip data after a refresh. The moment a replace-commit truncate ships independently, batch ids restart at 0 against a timeline that still says 5 and every full refresh silently drops the first N batches.
Proposal: key the sink checkpoint on the checkpoint location (SDP's `<flow>/<N>` increments on reset) or query id, ship that *together* with the replace-commit truncate (`insert_overwrite_table` of an empty dataset already exists), as Phase 1b; keep the destructive mode behind an advanced config.
##########
rfc/rfc-112/rfc-112.md:
##########
@@ -0,0 +1,429 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-112: Support Spark Declarative Pipelines
+
+## Proposers
+
+- @cshuo
+- @danny0405
+
+## Approvers
+
+- TBD
+
+## Status
+
+Issue: TBD
+
+## Abstract
+
+[Spark Declarative Pipelines (SDP)](https://spark.apache.org/docs/4.1.1/declarative-pipelines-programming-guide.html) lets users define tables and transformations while Spark builds the dependency graph and runs the batch and streaming flows. Hudi provides atomic commits, keyed updates, event ordering, deletes, incremental queries, and table services.
+
+This RFC makes Hudi a supported SDP source and target. It covers catalog-based table creation, materialized views, streaming tables, basic Hudi-to-Hudi incremental reads, and keyed upserts through Hudi's existing writer.
+
+SDP and Hudi retain their existing responsibilities:
+
+- SDP owns dataset declarations, dependency analysis, orchestration, flow checkpoints, retries, and execution order.
+- Hudi owns table storage, commit atomicity, record reconciliation, incremental read semantics, and table services.
+
+Users can build the same pipelines today with Spark jobs, Hudi writers or Hudi Streamer, and an external orchestrator. This integration removes much of that glue code while keeping the outputs as ordinary Hudi tables.
+
+## Background
+
+### Spark Declarative Pipelines
+
+Apache Spark 4.1 introduced SDP for declarative batch and streaming ETL. SDP persists two main dataset types:
+
+- A **materialized view**, which has exactly one batch flow that computes a table.
+- A **streaming table**, which has one or more streaming flows that append their output to a table.
+
+Users define pipelines in Python or SQL. SDP validates dataset references, builds the dataflow graph, runs independent flows in parallel, and stores Structured Streaming checkpoints in the pipeline storage location.
+
+In the Spark 4.1 implementation, a streaming table flow uses a Structured Streaming append-mode write to `DataStreamWriter.toTable`, and a batch table flow uses append-mode `DataFrameWriter.saveAsTable`. For an existing materialized view, SDP issues `TRUNCATE TABLE`, calls `catalog.alterTable` to reconcile schema and properties, then runs the batch flow. See Spark's [`FlowExecution`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala) and [`DatasetManager`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala).
+
+These table operations are the storage-format integration boundary; this RFC does not add a Hudi-specific execution engine.
+
+### Relevant Hudi capabilities
+
+Hudi already provides the required data-plane primitives:
+
+- `HoodieCatalog` can create and load Hudi tables through Spark's catalog APIs.
+- The Hudi batch writer supports bulk insert, insert, upsert, insert overwrite, and table insert overwrite operations. An upsert can carry delete-marker records.
+- `HoodieStreamingSink` writes Structured Streaming micro-batches and records checkpoint metadata used to avoid recommitting an already committed batch after retry.
+- Hudi's streaming source and incremental query modes can expose records changed since a prior instant.
+
+The integration exposes these capabilities through SDP's existing table and flow abstractions. Hudi Streamer, the DataFrame writer, Spark SQL DML, and direct Structured Streaming APIs remain unchanged.
+
+### Current gaps
+
+`format="hudi"` is not yet an end-to-end supported SDP contract. Some operations may already work, but table creation, refresh, restart, upsert configuration, and Hudi-to-Hudi chaining are neither documented nor covered by an integration suite.
+
+Materialized-view refresh needs end-to-end validation. SDP 4.1 truncates the target before reconciling its schema and writing the recomputed result. Hudi's current full-table `TRUNCATE TABLE` deletes the table path, including the timeline, and reinitializes its properties. This RFC reuses that behavior and documents its limitations; a timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+The integration must also address:
+
+- SDP table properties and flow-scoped write options must be resolved and passed to the Hudi writer.
+- Streaming restart must coordinate the SDP/Spark checkpoint with Hudi's sink checkpoint metadata.
+- A Hudi source used through `readStream.table` must preserve incremental offset semantics.
+- SDP must carry Hudi's persisted record-key, precombine, and record-merger settings into the existing upsert path. Deletes remain upsert input rows marked by `_hoodie_is_deleted`; Spark append output mode does not replace this contract.
+
+### Related work
+
+- Hudi's Spark 4.1 support ([issue #17663](https://github.com/apache/hudi/issues/17663), [PR #17674](https://github.com/apache/hudi/pull/17674)) provides the runtime baseline, but not an SDP contract.
+- DataSource V2 read work ([RFC-98](../rfc-98/rfc-98.md), [issue #15023](https://github.com/apache/hudi/issues/15023)) may improve the catalog read path but is not required for the initial integration.
+- The [SQL insert-overwrite discussion](https://github.com/apache/hudi/discussions/13815) shows why refresh semantics must be explicit rather than inferred from an append call.
+- [Issue #13973](https://github.com/apache/hudi/issues/13973) shows that a latest-state incremental read cannot fully propagate hard deletes. Flows that require complete delete propagation can use Hudi's existing CDC incremental format.
+- [Issue #5537](https://github.com/apache/hudi/issues/5537) tracks named-catalog support. The first version uses `HoodieCatalog` as `spark_catalog`.
+
+## Motivation
+
+### Representative use case
+
+A typical order pipeline looks like this:
+
+```text
+MySQL/PostgreSQL CDC or Kafka
+ |
+ v
+ bronze_order_events (Hudi)
+ |
+ v
+ current_orders (Hudi)
+ |
+ v
+ daily_order_metrics (Hudi)
+```
+
+The bronze table stores incoming events. The current-state table uses a record key and precombine field to reconcile duplicates and out-of-order events, including delete-marker records. Downstream streaming tables consume changed records incrementally, while batch aggregates can be materialized as views.
+
+Today this usually means several Spark applications plus custom handling for table creation, write options, checkpoints, dependencies, and restarts. With SDP, the graph lives in one project and the persisted datasets remain regular Hudi tables that other engines can query.
+
+Related user request issues include:
+
+- #19281: [multi-dataset incremental processing](https://github.com/apache/hudi/issues/19281)
+- #5189: [chained incremental Hudi tables](https://github.com/apache/hudi/issues/5189)
+- #13973: [downstream delete propagation](https://github.com/apache/hudi/issues/13973).
+
+### User value
+
+1. **Declarative pipelines.** Users define datasets as SQL or DataFrame transformations; SDP manages dependencies, checkpoints, and Hudi writes.
+2. **Managed incremental composition.** Hudi already provides incremental reads; SDP composes them into a dependency-aware pipeline, managing execution order, checkpoints, and recovery without separate Spark jobs or hand-written orchestration.
+3. **Open outputs.** Each dataset remains a standard Hudi table that can be queried independently of SDP by any Hudi-compatible engine.
+
+SDP does not make arbitrary joins or aggregates incremental; performance benefits apply only to flows that can process changes incrementally.
+
+## Goals
+
+1. Create Hudi tables from SDP datasets through `HoodieCatalog`.
+2. Write materialized views with Hudi `bulk_insert` or `insert`, following SDP's truncate-and-recompute refresh sequence.
+3. Write SDP streaming tables through Hudi's Structured Streaming sink.
+4. Support a basic incremental Hudi-to-Hudi streaming-table chain.
+5. Expose Hudi's existing keyed upsert path through SDP, including record-key, precombine, record-merger, and delete-marker handling.
+6. Keep non-SDP Hudi behavior unchanged.
+
+## Non-Goals
+
+The first version excludes:
+
+- A complete Hudi DataSource V2 read or streaming-write implementation.
+- Atomic materialized-view refresh across truncate, schema reconciliation, and the batch write.
+- Changes to Hudi's full-table truncate semantics. Timeline-preserving truncate can be developed independently.
+- Cross-table transactions across multiple SDP targets.
+- End-to-end exactly-once delivery across an arbitrary external source and Hudi. Guarantees remain scoped to Spark/Hudi micro-batch retries at one target.
+- Arbitrary named catalogs for `HoodieCatalog`; the first version uses Hudi as `spark_catalog`.
+
+## Proposed User Experience
+
+The examples below implement the representative order pipeline in dependency order, using Hudi's existing SQL table options and `SET` for flow-scoped write options.
+
+### Pipeline configuration
+
+An SDP project configures the Hudi extension and catalog like any other Hudi Spark application:
+
+```yaml
+name: hudi_orders_pipeline
+libraries:
+ - glob:
+ include: transformations/**
+storage: s3://pipelines/checkpoints/hudi_orders_pipeline
+catalog: spark_catalog
+database: lakehouse
+configuration:
+ spark.sql.extensions: org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.sql.catalog.spark_catalog: org.apache.spark.sql.hudi.catalog.HoodieCatalog
+```
+
+The SQL below assumes that `order_events_source` is an external streaming source containing parsed Kafka or database CDC events. Each `CREATE ... AS SELECT` statement defines both the target dataset and its single flow.
+
+### Streaming table with an append flow
+
+```sql
+SET spark.hoodie.datasource.write.operation = insert;
Review Comment:
**🚨** `SET spark.hoodie.datasource.write.operation = …`
This does not reach streaming flows today. `DataSourceOptionsHelper.collectSparkHoodieConfs` (which forwards `spark.hoodie.*` session conf) is only called from `DefaultSource.createRelation` (`DefaultSource.scala:182`, #18650). `createSink` (`:200`) passes the raw writer options to `HoodieStreamingSink`, which calls `HoodieSparkSqlWriter.write` directly. So `bronze_order_events` and `current_orders` both run with the sink default `upsert`; the bronze example (no `primaryKey`) most likely fails on upsert with auto-generated keys.
For MVs (L234) the batch path does pick it up. Please add an implementation step for the sink (merge `spark.hoodie.*` from the session in `createSink`, or make the write operation a table property), and note the behaviour change for existing `writeStream` users whose sessions carry `spark.hoodie.*`.
##########
rfc/rfc-112/rfc-112.md:
##########
@@ -0,0 +1,429 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-112: Support Spark Declarative Pipelines
+
+## Proposers
+
+- @cshuo
+- @danny0405
+
+## Approvers
+
+- TBD
+
+## Status
+
+Issue: TBD
+
+## Abstract
+
+[Spark Declarative Pipelines (SDP)](https://spark.apache.org/docs/4.1.1/declarative-pipelines-programming-guide.html) lets users define tables and transformations while Spark builds the dependency graph and runs the batch and streaming flows. Hudi provides atomic commits, keyed updates, event ordering, deletes, incremental queries, and table services.
+
+This RFC makes Hudi a supported SDP source and target. It covers catalog-based table creation, materialized views, streaming tables, basic Hudi-to-Hudi incremental reads, and keyed upserts through Hudi's existing writer.
+
+SDP and Hudi retain their existing responsibilities:
+
+- SDP owns dataset declarations, dependency analysis, orchestration, flow checkpoints, retries, and execution order.
+- Hudi owns table storage, commit atomicity, record reconciliation, incremental read semantics, and table services.
+
+Users can build the same pipelines today with Spark jobs, Hudi writers or Hudi Streamer, and an external orchestrator. This integration removes much of that glue code while keeping the outputs as ordinary Hudi tables.
+
+## Background
+
+### Spark Declarative Pipelines
+
+Apache Spark 4.1 introduced SDP for declarative batch and streaming ETL. SDP persists two main dataset types:
+
+- A **materialized view**, which has exactly one batch flow that computes a table.
+- A **streaming table**, which has one or more streaming flows that append their output to a table.
+
+Users define pipelines in Python or SQL. SDP validates dataset references, builds the dataflow graph, runs independent flows in parallel, and stores Structured Streaming checkpoints in the pipeline storage location.
+
+In the Spark 4.1 implementation, a streaming table flow uses a Structured Streaming append-mode write to `DataStreamWriter.toTable`, and a batch table flow uses append-mode `DataFrameWriter.saveAsTable`. For an existing materialized view, SDP issues `TRUNCATE TABLE`, calls `catalog.alterTable` to reconcile schema and properties, then runs the batch flow. See Spark's [`FlowExecution`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala) and [`DatasetManager`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala).
+
+These table operations are the storage-format integration boundary; this RFC does not add a Hudi-specific execution engine.
+
+### Relevant Hudi capabilities
+
+Hudi already provides the required data-plane primitives:
+
+- `HoodieCatalog` can create and load Hudi tables through Spark's catalog APIs.
+- The Hudi batch writer supports bulk insert, insert, upsert, insert overwrite, and table insert overwrite operations. An upsert can carry delete-marker records.
+- `HoodieStreamingSink` writes Structured Streaming micro-batches and records checkpoint metadata used to avoid recommitting an already committed batch after retry.
+- Hudi's streaming source and incremental query modes can expose records changed since a prior instant.
+
+The integration exposes these capabilities through SDP's existing table and flow abstractions. Hudi Streamer, the DataFrame writer, Spark SQL DML, and direct Structured Streaming APIs remain unchanged.
+
+### Current gaps
+
+`format="hudi"` is not yet an end-to-end supported SDP contract. Some operations may already work, but table creation, refresh, restart, upsert configuration, and Hudi-to-Hudi chaining are neither documented nor covered by an integration suite.
+
+Materialized-view refresh needs end-to-end validation. SDP 4.1 truncates the target before reconciling its schema and writing the recomputed result. Hudi's current full-table `TRUNCATE TABLE` deletes the table path, including the timeline, and reinitializes its properties. This RFC reuses that behavior and documents its limitations; a timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+The integration must also address:
+
+- SDP table properties and flow-scoped write options must be resolved and passed to the Hudi writer.
+- Streaming restart must coordinate the SDP/Spark checkpoint with Hudi's sink checkpoint metadata.
+- A Hudi source used through `readStream.table` must preserve incremental offset semantics.
+- SDP must carry Hudi's persisted record-key, precombine, and record-merger settings into the existing upsert path. Deletes remain upsert input rows marked by `_hoodie_is_deleted`; Spark append output mode does not replace this contract.
+
+### Related work
+
+- Hudi's Spark 4.1 support ([issue #17663](https://github.com/apache/hudi/issues/17663), [PR #17674](https://github.com/apache/hudi/pull/17674)) provides the runtime baseline, but not an SDP contract.
+- DataSource V2 read work ([RFC-98](../rfc-98/rfc-98.md), [issue #15023](https://github.com/apache/hudi/issues/15023)) may improve the catalog read path but is not required for the initial integration.
+- The [SQL insert-overwrite discussion](https://github.com/apache/hudi/discussions/13815) shows why refresh semantics must be explicit rather than inferred from an append call.
+- [Issue #13973](https://github.com/apache/hudi/issues/13973) shows that a latest-state incremental read cannot fully propagate hard deletes. Flows that require complete delete propagation can use Hudi's existing CDC incremental format.
+- [Issue #5537](https://github.com/apache/hudi/issues/5537) tracks named-catalog support. The first version uses `HoodieCatalog` as `spark_catalog`.
+
+## Motivation
+
+### Representative use case
+
+A typical order pipeline looks like this:
+
+```text
+MySQL/PostgreSQL CDC or Kafka
+ |
+ v
+ bronze_order_events (Hudi)
+ |
+ v
+ current_orders (Hudi)
+ |
+ v
+ daily_order_metrics (Hudi)
+```
+
+The bronze table stores incoming events. The current-state table uses a record key and precombine field to reconcile duplicates and out-of-order events, including delete-marker records. Downstream streaming tables consume changed records incrementally, while batch aggregates can be materialized as views.
+
+Today this usually means several Spark applications plus custom handling for table creation, write options, checkpoints, dependencies, and restarts. With SDP, the graph lives in one project and the persisted datasets remain regular Hudi tables that other engines can query.
+
+Related user request issues include:
+
+- #19281: [multi-dataset incremental processing](https://github.com/apache/hudi/issues/19281)
+- #5189: [chained incremental Hudi tables](https://github.com/apache/hudi/issues/5189)
+- #13973: [downstream delete propagation](https://github.com/apache/hudi/issues/13973).
+
+### User value
+
+1. **Declarative pipelines.** Users define datasets as SQL or DataFrame transformations; SDP manages dependencies, checkpoints, and Hudi writes.
+2. **Managed incremental composition.** Hudi already provides incremental reads; SDP composes them into a dependency-aware pipeline, managing execution order, checkpoints, and recovery without separate Spark jobs or hand-written orchestration.
+3. **Open outputs.** Each dataset remains a standard Hudi table that can be queried independently of SDP by any Hudi-compatible engine.
+
+SDP does not make arbitrary joins or aggregates incremental; performance benefits apply only to flows that can process changes incrementally.
+
+## Goals
+
+1. Create Hudi tables from SDP datasets through `HoodieCatalog`.
+2. Write materialized views with Hudi `bulk_insert` or `insert`, following SDP's truncate-and-recompute refresh sequence.
+3. Write SDP streaming tables through Hudi's Structured Streaming sink.
+4. Support a basic incremental Hudi-to-Hudi streaming-table chain.
+5. Expose Hudi's existing keyed upsert path through SDP, including record-key, precombine, record-merger, and delete-marker handling.
+6. Keep non-SDP Hudi behavior unchanged.
+
+## Non-Goals
+
+The first version excludes:
+
+- A complete Hudi DataSource V2 read or streaming-write implementation.
+- Atomic materialized-view refresh across truncate, schema reconciliation, and the batch write.
+- Changes to Hudi's full-table truncate semantics. Timeline-preserving truncate can be developed independently.
+- Cross-table transactions across multiple SDP targets.
+- End-to-end exactly-once delivery across an arbitrary external source and Hudi. Guarantees remain scoped to Spark/Hudi micro-batch retries at one target.
+- Arbitrary named catalogs for `HoodieCatalog`; the first version uses Hudi as `spark_catalog`.
+
+## Proposed User Experience
+
+The examples below implement the representative order pipeline in dependency order, using Hudi's existing SQL table options and `SET` for flow-scoped write options.
+
+### Pipeline configuration
+
+An SDP project configures the Hudi extension and catalog like any other Hudi Spark application:
+
+```yaml
+name: hudi_orders_pipeline
+libraries:
+ - glob:
+ include: transformations/**
+storage: s3://pipelines/checkpoints/hudi_orders_pipeline
+catalog: spark_catalog
+database: lakehouse
+configuration:
+ spark.sql.extensions: org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.sql.catalog.spark_catalog: org.apache.spark.sql.hudi.catalog.HoodieCatalog
+```
+
+The SQL below assumes that `order_events_source` is an external streaming source containing parsed Kafka or database CDC events. Each `CREATE ... AS SELECT` statement defines both the target dataset and its single flow.
+
+### Streaming table with an append flow
+
+```sql
+SET spark.hoodie.datasource.write.operation = insert;
+
+CREATE STREAMING TABLE bronze_order_events
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor'
+)
+AS
+SELECT *
+FROM STREAM order_events_source;
+```
+
+SDP owns the checkpoint location and trigger; Hudi commits each micro-batch.
+
+### Streaming table with keyed upserts
+
+```sql
+SET spark.hoodie.datasource.write.operation = upsert;
+
+CREATE STREAMING TABLE current_orders
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor',
+ 'primaryKey' = 'order_id',
+ 'preCombineField' = 'event_sequence'
+)
+AS
+SELECT
+ *,
+ operation = 'DELETE' AS _hoodie_is_deleted
+FROM STREAM bronze_order_events;
+```
+
+Spark append output mode only controls how each micro-batch reaches the sink; it does not make the Hudi target append-only. `HoodieStreamingSink` still runs the configured Hudi operation, which defaults to `upsert`.
+
+For an existing Hudi table, SDP loads the record key, precombine field, partition fields, and record-merger settings from the table configuration. For a new SDP table, these settings are declared as table properties and persisted by Hudi. SDP does not implement its own deduplication, ordering, merge, or delete path. It validates the schema and configuration, then passes each micro-batch to the existing Hudi writer.
+
+### Batch materialized view
+
+```sql
+SET spark.hoodie.datasource.write.operation = bulk_insert;
+
+CREATE MATERIALIZED VIEW daily_order_metrics
+USING hudi
+PARTITIONED BY (order_date)
+TBLPROPERTIES (
+ 'type' = 'cow'
+)
+AS
+SELECT
+ order_date,
+ state,
+ COUNT(*) AS order_count
+FROM current_orders
+GROUP BY order_date, state;
+```
+
+The first run creates the table and writes the result using `bulk_insert`. Later runs truncate the target, reconcile its schema and properties, and write the recomputed result. With Hudi's current truncate implementation, each refresh removes the previous data and timeline history.
+
+## Semantics
+
+### Dataset-to-Hudi operation mapping
+
+| SDP dataset/flow | Spark 4.1 execution shape | Proposed Hudi semantics |
+| ----------------------------------- | --------------------------------------------- | -------------------------------------------------------- |
+| New materialized view | Batch append to newly created table | Hudi `bulk_insert` or `insert` |
+| Existing materialized view | Truncate, alter, then batch append | Existing Hudi truncate, then `bulk_insert` or `insert` |
+| Streaming table with an append flow | Append-mode micro-batches | Hudi `insert` or `bulk_insert`, configured per flow |
+| Hudi upsert target | Append-mode micro-batches | Existing Hudi `upsert` |
+| Hudi streaming source | Structured Streaming read with SDP checkpoint | Basic Hudi latest-state incremental source |
+
+### Materialized-view write mapping
+
+Initial materialization uses Hudi `bulk_insert` or `insert`. For an existing materialized view, Spark 4.1 executes `TRUNCATE TABLE`, reconciles schema and properties through `catalog.alterTable`, then writes the recomputed result through append-mode `saveAsTable`. Hudi uses the same configured write operation after truncate. An empty result leaves the target empty.
+
+These steps are not a single transaction. Readers may see an empty target during refresh. If schema reconciliation or the batch flow fails after truncate, the previous result is not restored, and completed catalog changes remain in effect.
+
+The initial integration reuses Hudi's current full-table truncate, which deletes old data files and timeline history. This can disrupt concurrent readers and invalidate existing incremental offsets. Timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+Materialized views are supported as batch outputs, but not as incremental sources in the initial scope.
+
+### Streaming commit and restart behavior
+
+SDP manages each flow's checkpoint location and trigger. Multiple append flows can write to the same Hudi target, each with a distinct Hudi checkpoint identifier that remains stable across restarts. Concurrent flows require Hudi's existing multi-writer configuration and an appropriate lock provider. Each flow commits and retries independently.
+
+Normal restarts reuse the same Spark checkpoint and Hudi checkpoint identifier. A streaming-table full refresh starts a fresh checkpoint and uses Hudi's current full-table truncate to remove target data, timeline history, and sink checkpoint metadata before rebuilding the table. Resetting the Spark checkpoint while retaining the Hudi target is outside this RFC's scope.
+
+### Configuration ownership and precedence
+
+Configuration is split by ownership:
+
+- **Table identity:** table type, record key, precombine field, partition fields, key generator, and record-merger settings. Hudi persists these values; SDP loads them for existing tables, and a pipeline run cannot silently change them.
+- **Write behavior:** operation and writer tuning, supplied through SQL `SET spark.hoodie.*` as flow-scoped write options.
+- **Flow execution:** checkpoint location, trigger, retry, and flow identity. SDP owns these values; the Hudi integration maps each flow to a distinct sink checkpoint identifier.
+
+SDP captures the active `SET` values at each flow definition; a `SET` remains in effect for subsequent definitions until overridden. Flow settings override pipeline and Hudi defaults and must be compatible with the persisted table configuration. The integration strips the `spark.` prefix from write options and reuses `HoodieOptionConfig` to normalize table options. Flows writing to the same target must use compatible concurrency and lock settings. The sink checkpoint identifier is resolved per flow, not shared as a table property.
+
+Invalid properties should fail during SDP `dry-run` when possible, and otherwise before the first write.
+
+### Table services
+
+Table services follow the existing Hudi writer behavior, with no SDP-specific scheduling or retention defaults. Integration tests cover service shutdown, recovery, and resource cleanup when flows finish, stop, or restart, including multiple flows sharing a target.
Review Comment:
**⚠️** Table services
SDP's trigger is `Trigger.AvailableNow` unconditionally (`TriggeredGraphExecution.streamTrigger`); each run drains and stops. The sink's async compaction/clustering services are background threads with JVM shutdown hooks and will rarely finish inside that window. Please state the recommended mode for SDP (inline, or schedule-inline + execute externally) instead of "follow existing writer behaviour".
##########
rfc/rfc-112/rfc-112.md:
##########
@@ -0,0 +1,429 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-112: Support Spark Declarative Pipelines
+
+## Proposers
+
+- @cshuo
+- @danny0405
+
+## Approvers
+
+- TBD
+
+## Status
+
+Issue: TBD
+
+## Abstract
+
+[Spark Declarative Pipelines (SDP)](https://spark.apache.org/docs/4.1.1/declarative-pipelines-programming-guide.html) lets users define tables and transformations while Spark builds the dependency graph and runs the batch and streaming flows. Hudi provides atomic commits, keyed updates, event ordering, deletes, incremental queries, and table services.
+
+This RFC makes Hudi a supported SDP source and target. It covers catalog-based table creation, materialized views, streaming tables, basic Hudi-to-Hudi incremental reads, and keyed upserts through Hudi's existing writer.
+
+SDP and Hudi retain their existing responsibilities:
+
+- SDP owns dataset declarations, dependency analysis, orchestration, flow checkpoints, retries, and execution order.
+- Hudi owns table storage, commit atomicity, record reconciliation, incremental read semantics, and table services.
+
+Users can build the same pipelines today with Spark jobs, Hudi writers or Hudi Streamer, and an external orchestrator. This integration removes much of that glue code while keeping the outputs as ordinary Hudi tables.
+
+## Background
+
+### Spark Declarative Pipelines
+
+Apache Spark 4.1 introduced SDP for declarative batch and streaming ETL. SDP persists two main dataset types:
+
+- A **materialized view**, which has exactly one batch flow that computes a table.
+- A **streaming table**, which has one or more streaming flows that append their output to a table.
+
+Users define pipelines in Python or SQL. SDP validates dataset references, builds the dataflow graph, runs independent flows in parallel, and stores Structured Streaming checkpoints in the pipeline storage location.
+
+In the Spark 4.1 implementation, a streaming table flow uses a Structured Streaming append-mode write to `DataStreamWriter.toTable`, and a batch table flow uses append-mode `DataFrameWriter.saveAsTable`. For an existing materialized view, SDP issues `TRUNCATE TABLE`, calls `catalog.alterTable` to reconcile schema and properties, then runs the batch flow. See Spark's [`FlowExecution`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala) and [`DatasetManager`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala).
+
+These table operations are the storage-format integration boundary; this RFC does not add a Hudi-specific execution engine.
+
+### Relevant Hudi capabilities
+
+Hudi already provides the required data-plane primitives:
+
+- `HoodieCatalog` can create and load Hudi tables through Spark's catalog APIs.
+- The Hudi batch writer supports bulk insert, insert, upsert, insert overwrite, and table insert overwrite operations. An upsert can carry delete-marker records.
+- `HoodieStreamingSink` writes Structured Streaming micro-batches and records checkpoint metadata used to avoid recommitting an already committed batch after retry.
+- Hudi's streaming source and incremental query modes can expose records changed since a prior instant.
+
+The integration exposes these capabilities through SDP's existing table and flow abstractions. Hudi Streamer, the DataFrame writer, Spark SQL DML, and direct Structured Streaming APIs remain unchanged.
+
+### Current gaps
+
+`format="hudi"` is not yet an end-to-end supported SDP contract. Some operations may already work, but table creation, refresh, restart, upsert configuration, and Hudi-to-Hudi chaining are neither documented nor covered by an integration suite.
+
+Materialized-view refresh needs end-to-end validation. SDP 4.1 truncates the target before reconciling its schema and writing the recomputed result. Hudi's current full-table `TRUNCATE TABLE` deletes the table path, including the timeline, and reinitializes its properties. This RFC reuses that behavior and documents its limitations; a timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+The integration must also address:
+
+- SDP table properties and flow-scoped write options must be resolved and passed to the Hudi writer.
+- Streaming restart must coordinate the SDP/Spark checkpoint with Hudi's sink checkpoint metadata.
+- A Hudi source used through `readStream.table` must preserve incremental offset semantics.
+- SDP must carry Hudi's persisted record-key, precombine, and record-merger settings into the existing upsert path. Deletes remain upsert input rows marked by `_hoodie_is_deleted`; Spark append output mode does not replace this contract.
+
+### Related work
+
+- Hudi's Spark 4.1 support ([issue #17663](https://github.com/apache/hudi/issues/17663), [PR #17674](https://github.com/apache/hudi/pull/17674)) provides the runtime baseline, but not an SDP contract.
+- DataSource V2 read work ([RFC-98](../rfc-98/rfc-98.md), [issue #15023](https://github.com/apache/hudi/issues/15023)) may improve the catalog read path but is not required for the initial integration.
+- The [SQL insert-overwrite discussion](https://github.com/apache/hudi/discussions/13815) shows why refresh semantics must be explicit rather than inferred from an append call.
+- [Issue #13973](https://github.com/apache/hudi/issues/13973) shows that a latest-state incremental read cannot fully propagate hard deletes. Flows that require complete delete propagation can use Hudi's existing CDC incremental format.
+- [Issue #5537](https://github.com/apache/hudi/issues/5537) tracks named-catalog support. The first version uses `HoodieCatalog` as `spark_catalog`.
+
+## Motivation
+
+### Representative use case
+
+A typical order pipeline looks like this:
+
+```text
+MySQL/PostgreSQL CDC or Kafka
+ |
+ v
+ bronze_order_events (Hudi)
+ |
+ v
+ current_orders (Hudi)
+ |
+ v
+ daily_order_metrics (Hudi)
+```
+
+The bronze table stores incoming events. The current-state table uses a record key and precombine field to reconcile duplicates and out-of-order events, including delete-marker records. Downstream streaming tables consume changed records incrementally, while batch aggregates can be materialized as views.
+
+Today this usually means several Spark applications plus custom handling for table creation, write options, checkpoints, dependencies, and restarts. With SDP, the graph lives in one project and the persisted datasets remain regular Hudi tables that other engines can query.
+
+Related user request issues include:
+
+- #19281: [multi-dataset incremental processing](https://github.com/apache/hudi/issues/19281)
+- #5189: [chained incremental Hudi tables](https://github.com/apache/hudi/issues/5189)
+- #13973: [downstream delete propagation](https://github.com/apache/hudi/issues/13973).
+
+### User value
+
+1. **Declarative pipelines.** Users define datasets as SQL or DataFrame transformations; SDP manages dependencies, checkpoints, and Hudi writes.
+2. **Managed incremental composition.** Hudi already provides incremental reads; SDP composes them into a dependency-aware pipeline, managing execution order, checkpoints, and recovery without separate Spark jobs or hand-written orchestration.
+3. **Open outputs.** Each dataset remains a standard Hudi table that can be queried independently of SDP by any Hudi-compatible engine.
+
+SDP does not make arbitrary joins or aggregates incremental; performance benefits apply only to flows that can process changes incrementally.
+
+## Goals
+
+1. Create Hudi tables from SDP datasets through `HoodieCatalog`.
+2. Write materialized views with Hudi `bulk_insert` or `insert`, following SDP's truncate-and-recompute refresh sequence.
+3. Write SDP streaming tables through Hudi's Structured Streaming sink.
+4. Support a basic incremental Hudi-to-Hudi streaming-table chain.
+5. Expose Hudi's existing keyed upsert path through SDP, including record-key, precombine, record-merger, and delete-marker handling.
+6. Keep non-SDP Hudi behavior unchanged.
+
+## Non-Goals
+
+The first version excludes:
+
+- A complete Hudi DataSource V2 read or streaming-write implementation.
+- Atomic materialized-view refresh across truncate, schema reconciliation, and the batch write.
+- Changes to Hudi's full-table truncate semantics. Timeline-preserving truncate can be developed independently.
+- Cross-table transactions across multiple SDP targets.
+- End-to-end exactly-once delivery across an arbitrary external source and Hudi. Guarantees remain scoped to Spark/Hudi micro-batch retries at one target.
+- Arbitrary named catalogs for `HoodieCatalog`; the first version uses Hudi as `spark_catalog`.
+
+## Proposed User Experience
+
+The examples below implement the representative order pipeline in dependency order, using Hudi's existing SQL table options and `SET` for flow-scoped write options.
+
+### Pipeline configuration
+
+An SDP project configures the Hudi extension and catalog like any other Hudi Spark application:
+
+```yaml
+name: hudi_orders_pipeline
+libraries:
+ - glob:
+ include: transformations/**
+storage: s3://pipelines/checkpoints/hudi_orders_pipeline
+catalog: spark_catalog
+database: lakehouse
+configuration:
+ spark.sql.extensions: org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.sql.catalog.spark_catalog: org.apache.spark.sql.hudi.catalog.HoodieCatalog
+```
+
+The SQL below assumes that `order_events_source` is an external streaming source containing parsed Kafka or database CDC events. Each `CREATE ... AS SELECT` statement defines both the target dataset and its single flow.
+
+### Streaming table with an append flow
+
+```sql
+SET spark.hoodie.datasource.write.operation = insert;
+
+CREATE STREAMING TABLE bronze_order_events
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor'
+)
+AS
+SELECT *
+FROM STREAM order_events_source;
+```
+
+SDP owns the checkpoint location and trigger; Hudi commits each micro-batch.
+
+### Streaming table with keyed upserts
+
+```sql
+SET spark.hoodie.datasource.write.operation = upsert;
+
+CREATE STREAMING TABLE current_orders
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor',
+ 'primaryKey' = 'order_id',
+ 'preCombineField' = 'event_sequence'
+)
+AS
+SELECT
+ *,
+ operation = 'DELETE' AS _hoodie_is_deleted
+FROM STREAM bronze_order_events;
+```
+
+Spark append output mode only controls how each micro-batch reaches the sink; it does not make the Hudi target append-only. `HoodieStreamingSink` still runs the configured Hudi operation, which defaults to `upsert`.
+
+For an existing Hudi table, SDP loads the record key, precombine field, partition fields, and record-merger settings from the table configuration. For a new SDP table, these settings are declared as table properties and persisted by Hudi. SDP does not implement its own deduplication, ordering, merge, or delete path. It validates the schema and configuration, then passes each micro-batch to the existing Hudi writer.
+
+### Batch materialized view
+
+```sql
+SET spark.hoodie.datasource.write.operation = bulk_insert;
+
+CREATE MATERIALIZED VIEW daily_order_metrics
+USING hudi
+PARTITIONED BY (order_date)
+TBLPROPERTIES (
+ 'type' = 'cow'
+)
+AS
+SELECT
+ order_date,
+ state,
+ COUNT(*) AS order_count
+FROM current_orders
+GROUP BY order_date, state;
+```
+
+The first run creates the table and writes the result using `bulk_insert`. Later runs truncate the target, reconcile its schema and properties, and write the recomputed result. With Hudi's current truncate implementation, each refresh removes the previous data and timeline history.
Review Comment:
**🚨** "Later runs truncate the target, reconcile its schema and properties, and write"
Ran this. The write step fails today independent of truncate: SDP calls `df.write.format("hudi").mode("append").saveAsTable(name)`; because `HoodieCatalog` is a `CatalogExtension`, Spark takes `saveAsV1TableCommand` → `CreateTable(..., Append)` → `CreateHoodieTableAsSelectCommand`, which handles `ErrorIfExists`/`Ignore` (and asserts not-`Overwrite`) for an existing table but falls through for `Append` into `HoodieCatalogTable.initHoodieTable()`:
```
HoodieAnalysisException: Can not create the managed table('spark_catalog.default.rate_by_bucket').
The associated location('…/warehouse/rate_by_bucket') already exists.
at HoodieCatalogTable.parseSchemaAndConfigs(HoodieCatalogTable.scala:277)
at CreateHoodieTableAsSelectCommand.run(CreateHoodieTableAsSelectCommand.scala:101)
at org.apache.spark.sql.classic.DataFrameWriter.saveAsTable(DataFrameWriter.scala:432)
at org.apache.spark.sql.pipelines.graph.BatchTableWrite.$anonfun$executeInternal$5(FlowExecution.scala:272)
```
Spark's own `CreateDataSourceTableAsSelectCommand` treats exists+Append as an insert. This is a general `saveAsTable(append)` bug under `HoodieCatalog`; it should be a listed Phase 2 prerequisite. Filed as #20369.
##########
rfc/rfc-112/rfc-112.md:
##########
@@ -0,0 +1,429 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-112: Support Spark Declarative Pipelines
+
+## Proposers
+
+- @cshuo
+- @danny0405
+
+## Approvers
+
+- TBD
+
+## Status
+
+Issue: TBD
+
+## Abstract
+
+[Spark Declarative Pipelines (SDP)](https://spark.apache.org/docs/4.1.1/declarative-pipelines-programming-guide.html) lets users define tables and transformations while Spark builds the dependency graph and runs the batch and streaming flows. Hudi provides atomic commits, keyed updates, event ordering, deletes, incremental queries, and table services.
+
+This RFC makes Hudi a supported SDP source and target. It covers catalog-based table creation, materialized views, streaming tables, basic Hudi-to-Hudi incremental reads, and keyed upserts through Hudi's existing writer.
+
+SDP and Hudi retain their existing responsibilities:
+
+- SDP owns dataset declarations, dependency analysis, orchestration, flow checkpoints, retries, and execution order.
+- Hudi owns table storage, commit atomicity, record reconciliation, incremental read semantics, and table services.
+
+Users can build the same pipelines today with Spark jobs, Hudi writers or Hudi Streamer, and an external orchestrator. This integration removes much of that glue code while keeping the outputs as ordinary Hudi tables.
+
+## Background
+
+### Spark Declarative Pipelines
+
+Apache Spark 4.1 introduced SDP for declarative batch and streaming ETL. SDP persists two main dataset types:
+
+- A **materialized view**, which has exactly one batch flow that computes a table.
+- A **streaming table**, which has one or more streaming flows that append their output to a table.
+
+Users define pipelines in Python or SQL. SDP validates dataset references, builds the dataflow graph, runs independent flows in parallel, and stores Structured Streaming checkpoints in the pipeline storage location.
+
+In the Spark 4.1 implementation, a streaming table flow uses a Structured Streaming append-mode write to `DataStreamWriter.toTable`, and a batch table flow uses append-mode `DataFrameWriter.saveAsTable`. For an existing materialized view, SDP issues `TRUNCATE TABLE`, calls `catalog.alterTable` to reconcile schema and properties, then runs the batch flow. See Spark's [`FlowExecution`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala) and [`DatasetManager`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala).
+
+These table operations are the storage-format integration boundary; this RFC does not add a Hudi-specific execution engine.
+
+### Relevant Hudi capabilities
+
+Hudi already provides the required data-plane primitives:
+
+- `HoodieCatalog` can create and load Hudi tables through Spark's catalog APIs.
+- The Hudi batch writer supports bulk insert, insert, upsert, insert overwrite, and table insert overwrite operations. An upsert can carry delete-marker records.
+- `HoodieStreamingSink` writes Structured Streaming micro-batches and records checkpoint metadata used to avoid recommitting an already committed batch after retry.
+- Hudi's streaming source and incremental query modes can expose records changed since a prior instant.
+
+The integration exposes these capabilities through SDP's existing table and flow abstractions. Hudi Streamer, the DataFrame writer, Spark SQL DML, and direct Structured Streaming APIs remain unchanged.
+
+### Current gaps
+
+`format="hudi"` is not yet an end-to-end supported SDP contract. Some operations may already work, but table creation, refresh, restart, upsert configuration, and Hudi-to-Hudi chaining are neither documented nor covered by an integration suite.
+
+Materialized-view refresh needs end-to-end validation. SDP 4.1 truncates the target before reconciling its schema and writing the recomputed result. Hudi's current full-table `TRUNCATE TABLE` deletes the table path, including the timeline, and reinitializes its properties. This RFC reuses that behavior and documents its limitations; a timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+The integration must also address:
+
+- SDP table properties and flow-scoped write options must be resolved and passed to the Hudi writer.
+- Streaming restart must coordinate the SDP/Spark checkpoint with Hudi's sink checkpoint metadata.
+- A Hudi source used through `readStream.table` must preserve incremental offset semantics.
+- SDP must carry Hudi's persisted record-key, precombine, and record-merger settings into the existing upsert path. Deletes remain upsert input rows marked by `_hoodie_is_deleted`; Spark append output mode does not replace this contract.
+
+### Related work
+
+- Hudi's Spark 4.1 support ([issue #17663](https://github.com/apache/hudi/issues/17663), [PR #17674](https://github.com/apache/hudi/pull/17674)) provides the runtime baseline, but not an SDP contract.
+- DataSource V2 read work ([RFC-98](../rfc-98/rfc-98.md), [issue #15023](https://github.com/apache/hudi/issues/15023)) may improve the catalog read path but is not required for the initial integration.
+- The [SQL insert-overwrite discussion](https://github.com/apache/hudi/discussions/13815) shows why refresh semantics must be explicit rather than inferred from an append call.
+- [Issue #13973](https://github.com/apache/hudi/issues/13973) shows that a latest-state incremental read cannot fully propagate hard deletes. Flows that require complete delete propagation can use Hudi's existing CDC incremental format.
+- [Issue #5537](https://github.com/apache/hudi/issues/5537) tracks named-catalog support. The first version uses `HoodieCatalog` as `spark_catalog`.
+
+## Motivation
+
+### Representative use case
+
+A typical order pipeline looks like this:
+
+```text
+MySQL/PostgreSQL CDC or Kafka
+ |
+ v
+ bronze_order_events (Hudi)
+ |
+ v
+ current_orders (Hudi)
+ |
+ v
+ daily_order_metrics (Hudi)
+```
+
+The bronze table stores incoming events. The current-state table uses a record key and precombine field to reconcile duplicates and out-of-order events, including delete-marker records. Downstream streaming tables consume changed records incrementally, while batch aggregates can be materialized as views.
+
+Today this usually means several Spark applications plus custom handling for table creation, write options, checkpoints, dependencies, and restarts. With SDP, the graph lives in one project and the persisted datasets remain regular Hudi tables that other engines can query.
+
+Related user request issues include:
+
+- #19281: [multi-dataset incremental processing](https://github.com/apache/hudi/issues/19281)
+- #5189: [chained incremental Hudi tables](https://github.com/apache/hudi/issues/5189)
+- #13973: [downstream delete propagation](https://github.com/apache/hudi/issues/13973).
+
+### User value
+
+1. **Declarative pipelines.** Users define datasets as SQL or DataFrame transformations; SDP manages dependencies, checkpoints, and Hudi writes.
+2. **Managed incremental composition.** Hudi already provides incremental reads; SDP composes them into a dependency-aware pipeline, managing execution order, checkpoints, and recovery without separate Spark jobs or hand-written orchestration.
+3. **Open outputs.** Each dataset remains a standard Hudi table that can be queried independently of SDP by any Hudi-compatible engine.
+
+SDP does not make arbitrary joins or aggregates incremental; performance benefits apply only to flows that can process changes incrementally.
+
+## Goals
+
+1. Create Hudi tables from SDP datasets through `HoodieCatalog`.
+2. Write materialized views with Hudi `bulk_insert` or `insert`, following SDP's truncate-and-recompute refresh sequence.
+3. Write SDP streaming tables through Hudi's Structured Streaming sink.
+4. Support a basic incremental Hudi-to-Hudi streaming-table chain.
+5. Expose Hudi's existing keyed upsert path through SDP, including record-key, precombine, record-merger, and delete-marker handling.
+6. Keep non-SDP Hudi behavior unchanged.
+
+## Non-Goals
+
+The first version excludes:
+
+- A complete Hudi DataSource V2 read or streaming-write implementation.
+- Atomic materialized-view refresh across truncate, schema reconciliation, and the batch write.
+- Changes to Hudi's full-table truncate semantics. Timeline-preserving truncate can be developed independently.
+- Cross-table transactions across multiple SDP targets.
+- End-to-end exactly-once delivery across an arbitrary external source and Hudi. Guarantees remain scoped to Spark/Hudi micro-batch retries at one target.
+- Arbitrary named catalogs for `HoodieCatalog`; the first version uses Hudi as `spark_catalog`.
+
+## Proposed User Experience
+
+The examples below implement the representative order pipeline in dependency order, using Hudi's existing SQL table options and `SET` for flow-scoped write options.
+
+### Pipeline configuration
+
+An SDP project configures the Hudi extension and catalog like any other Hudi Spark application:
+
+```yaml
+name: hudi_orders_pipeline
+libraries:
+ - glob:
+ include: transformations/**
+storage: s3://pipelines/checkpoints/hudi_orders_pipeline
+catalog: spark_catalog
+database: lakehouse
+configuration:
+ spark.sql.extensions: org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.sql.catalog.spark_catalog: org.apache.spark.sql.hudi.catalog.HoodieCatalog
+```
+
+The SQL below assumes that `order_events_source` is an external streaming source containing parsed Kafka or database CDC events. Each `CREATE ... AS SELECT` statement defines both the target dataset and its single flow.
+
+### Streaming table with an append flow
+
+```sql
+SET spark.hoodie.datasource.write.operation = insert;
+
+CREATE STREAMING TABLE bronze_order_events
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor'
+)
+AS
+SELECT *
+FROM STREAM order_events_source;
+```
+
+SDP owns the checkpoint location and trigger; Hudi commits each micro-batch.
+
+### Streaming table with keyed upserts
+
+```sql
+SET spark.hoodie.datasource.write.operation = upsert;
+
+CREATE STREAMING TABLE current_orders
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor',
+ 'primaryKey' = 'order_id',
+ 'preCombineField' = 'event_sequence'
+)
+AS
+SELECT
+ *,
+ operation = 'DELETE' AS _hoodie_is_deleted
+FROM STREAM bronze_order_events;
+```
+
+Spark append output mode only controls how each micro-batch reaches the sink; it does not make the Hudi target append-only. `HoodieStreamingSink` still runs the configured Hudi operation, which defaults to `upsert`.
+
+For an existing Hudi table, SDP loads the record key, precombine field, partition fields, and record-merger settings from the table configuration. For a new SDP table, these settings are declared as table properties and persisted by Hudi. SDP does not implement its own deduplication, ordering, merge, or delete path. It validates the schema and configuration, then passes each micro-batch to the existing Hudi writer.
+
+### Batch materialized view
+
+```sql
+SET spark.hoodie.datasource.write.operation = bulk_insert;
+
+CREATE MATERIALIZED VIEW daily_order_metrics
+USING hudi
+PARTITIONED BY (order_date)
+TBLPROPERTIES (
+ 'type' = 'cow'
+)
+AS
+SELECT
+ order_date,
+ state,
+ COUNT(*) AS order_count
+FROM current_orders
+GROUP BY order_date, state;
+```
+
+The first run creates the table and writes the result using `bulk_insert`. Later runs truncate the target, reconcile its schema and properties, and write the recomputed result. With Hudi's current truncate implementation, each refresh removes the previous data and timeline history.
+
+## Semantics
+
+### Dataset-to-Hudi operation mapping
+
+| SDP dataset/flow | Spark 4.1 execution shape | Proposed Hudi semantics |
+| ----------------------------------- | --------------------------------------------- | -------------------------------------------------------- |
+| New materialized view | Batch append to newly created table | Hudi `bulk_insert` or `insert` |
+| Existing materialized view | Truncate, alter, then batch append | Existing Hudi truncate, then `bulk_insert` or `insert` |
+| Streaming table with an append flow | Append-mode micro-batches | Hudi `insert` or `bulk_insert`, configured per flow |
+| Hudi upsert target | Append-mode micro-batches | Existing Hudi `upsert` |
+| Hudi streaming source | Structured Streaming read with SDP checkpoint | Basic Hudi latest-state incremental source |
+
+### Materialized-view write mapping
+
+Initial materialization uses Hudi `bulk_insert` or `insert`. For an existing materialized view, Spark 4.1 executes `TRUNCATE TABLE`, reconciles schema and properties through `catalog.alterTable`, then writes the recomputed result through append-mode `saveAsTable`. Hudi uses the same configured write operation after truncate. An empty result leaves the target empty.
+
+These steps are not a single transaction. Readers may see an empty target during refresh. If schema reconciliation or the batch flow fails after truncate, the previous result is not restored, and completed catalog changes remain in effect.
+
+The initial integration reuses Hudi's current full-table truncate, which deletes old data files and timeline history. This can disrupt concurrent readers and invalidate existing incremental offsets. Timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+Materialized views are supported as batch outputs, but not as incremental sources in the initial scope.
+
+### Streaming commit and restart behavior
+
+SDP manages each flow's checkpoint location and trigger. Multiple append flows can write to the same Hudi target, each with a distinct Hudi checkpoint identifier that remains stable across restarts. Concurrent flows require Hudi's existing multi-writer configuration and an appropriate lock provider. Each flow commits and retries independently.
+
+Normal restarts reuse the same Spark checkpoint and Hudi checkpoint identifier. A streaming-table full refresh starts a fresh checkpoint and uses Hudi's current full-table truncate to remove target data, timeline history, and sink checkpoint metadata before rebuilding the table. Resetting the Spark checkpoint while retaining the Hudi target is outside this RFC's scope.
+
+### Configuration ownership and precedence
+
+Configuration is split by ownership:
+
+- **Table identity:** table type, record key, precombine field, partition fields, key generator, and record-merger settings. Hudi persists these values; SDP loads them for existing tables, and a pipeline run cannot silently change them.
+- **Write behavior:** operation and writer tuning, supplied through SQL `SET spark.hoodie.*` as flow-scoped write options.
+- **Flow execution:** checkpoint location, trigger, retry, and flow identity. SDP owns these values; the Hudi integration maps each flow to a distinct sink checkpoint identifier.
+
+SDP captures the active `SET` values at each flow definition; a `SET` remains in effect for subsequent definitions until overridden. Flow settings override pipeline and Hudi defaults and must be compatible with the persisted table configuration. The integration strips the `spark.` prefix from write options and reuses `HoodieOptionConfig` to normalize table options. Flows writing to the same target must use compatible concurrency and lock settings. The sink checkpoint identifier is resolved per flow, not shared as a table property.
+
+Invalid properties should fail during SDP `dry-run` when possible, and otherwise before the first write.
+
+### Table services
+
+Table services follow the existing Hudi writer behavior, with no SDP-specific scheduling or retention defaults. Integration tests cover service shutdown, recovery, and resource cleanup when flows finish, stop, or restart, including multiple flows sharing a target.
+
+### Schema and partition evolution
+
+SDP follows Hudi's existing schema and partition evolution rules.
+
+- Hudi-supported additive schema changes may be accepted for streaming tables.
+- Dropping, renaming, or changing columns follows Hudi's configured schema evolution behavior.
+- Record-key, precombine, and partition columns must exist in the target schema.
+- Partitioning changes fail; SDP does not turn them into an implicit table replacement.
+
+`HoodieCatalog.alterTable` validates the complete set of requested schema and property changes before applying supported changes. It accepts unchanged properties and rejects unsupported changes or changes to table identity. The Hudi writer validates the actual input schema against the persisted table configuration using its existing schema reconciliation rules.
Review Comment:
**⚠️** `HoodieCatalog.alterTable` contract
Measured: `SetProperty` → `UnsupportedOperationException` (`HoodieCatalog.scala:255`) on the very first re-run of any table; `DeleteColumn` → `MatchError` (`HoodieCatalog.scala:238`) on full refresh, because `Table.columns()` exposes the five `_hoodie_*` columns and SDP's `diffSchemas(existing, flowSchema)` asks to delete them (same on every MV run after the first). "Accepts unchanged properties and rejects unsupported changes" should become an explicit list: idempotent `SetProperty`/`RemoveProperty` (reject only identity changes), `UpdateColumnNullability` no-op when loosening, and either hide meta fields from `columns()` or treat `DeleteColumn` on `_hoodie_*` as a no-op. Filed as #20371.
##########
rfc/rfc-112/rfc-112.md:
##########
@@ -0,0 +1,429 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-112: Support Spark Declarative Pipelines
+
+## Proposers
+
+- @cshuo
+- @danny0405
+
+## Approvers
+
+- TBD
+
+## Status
+
+Issue: TBD
Review Comment:
**đź’…**
`Issue: TBD` — link #19993. Approvers TBD.
##########
rfc/rfc-112/rfc-112.md:
##########
@@ -0,0 +1,429 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-112: Support Spark Declarative Pipelines
+
+## Proposers
+
+- @cshuo
+- @danny0405
+
+## Approvers
+
+- TBD
+
+## Status
+
+Issue: TBD
+
+## Abstract
+
+[Spark Declarative Pipelines (SDP)](https://spark.apache.org/docs/4.1.1/declarative-pipelines-programming-guide.html) lets users define tables and transformations while Spark builds the dependency graph and runs the batch and streaming flows. Hudi provides atomic commits, keyed updates, event ordering, deletes, incremental queries, and table services.
+
+This RFC makes Hudi a supported SDP source and target. It covers catalog-based table creation, materialized views, streaming tables, basic Hudi-to-Hudi incremental reads, and keyed upserts through Hudi's existing writer.
+
+SDP and Hudi retain their existing responsibilities:
+
+- SDP owns dataset declarations, dependency analysis, orchestration, flow checkpoints, retries, and execution order.
+- Hudi owns table storage, commit atomicity, record reconciliation, incremental read semantics, and table services.
+
+Users can build the same pipelines today with Spark jobs, Hudi writers or Hudi Streamer, and an external orchestrator. This integration removes much of that glue code while keeping the outputs as ordinary Hudi tables.
+
+## Background
+
+### Spark Declarative Pipelines
+
+Apache Spark 4.1 introduced SDP for declarative batch and streaming ETL. SDP persists two main dataset types:
+
+- A **materialized view**, which has exactly one batch flow that computes a table.
+- A **streaming table**, which has one or more streaming flows that append their output to a table.
+
+Users define pipelines in Python or SQL. SDP validates dataset references, builds the dataflow graph, runs independent flows in parallel, and stores Structured Streaming checkpoints in the pipeline storage location.
+
+In the Spark 4.1 implementation, a streaming table flow uses a Structured Streaming append-mode write to `DataStreamWriter.toTable`, and a batch table flow uses append-mode `DataFrameWriter.saveAsTable`. For an existing materialized view, SDP issues `TRUNCATE TABLE`, calls `catalog.alterTable` to reconcile schema and properties, then runs the batch flow. See Spark's [`FlowExecution`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala) and [`DatasetManager`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala).
+
+These table operations are the storage-format integration boundary; this RFC does not add a Hudi-specific execution engine.
+
+### Relevant Hudi capabilities
+
+Hudi already provides the required data-plane primitives:
+
+- `HoodieCatalog` can create and load Hudi tables through Spark's catalog APIs.
+- The Hudi batch writer supports bulk insert, insert, upsert, insert overwrite, and table insert overwrite operations. An upsert can carry delete-marker records.
+- `HoodieStreamingSink` writes Structured Streaming micro-batches and records checkpoint metadata used to avoid recommitting an already committed batch after retry.
+- Hudi's streaming source and incremental query modes can expose records changed since a prior instant.
+
+The integration exposes these capabilities through SDP's existing table and flow abstractions. Hudi Streamer, the DataFrame writer, Spark SQL DML, and direct Structured Streaming APIs remain unchanged.
+
+### Current gaps
+
+`format="hudi"` is not yet an end-to-end supported SDP contract. Some operations may already work, but table creation, refresh, restart, upsert configuration, and Hudi-to-Hudi chaining are neither documented nor covered by an integration suite.
+
+Materialized-view refresh needs end-to-end validation. SDP 4.1 truncates the target before reconciling its schema and writing the recomputed result. Hudi's current full-table `TRUNCATE TABLE` deletes the table path, including the timeline, and reinitializes its properties. This RFC reuses that behavior and documents its limitations; a timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+The integration must also address:
+
+- SDP table properties and flow-scoped write options must be resolved and passed to the Hudi writer.
+- Streaming restart must coordinate the SDP/Spark checkpoint with Hudi's sink checkpoint metadata.
+- A Hudi source used through `readStream.table` must preserve incremental offset semantics.
+- SDP must carry Hudi's persisted record-key, precombine, and record-merger settings into the existing upsert path. Deletes remain upsert input rows marked by `_hoodie_is_deleted`; Spark append output mode does not replace this contract.
+
+### Related work
+
+- Hudi's Spark 4.1 support ([issue #17663](https://github.com/apache/hudi/issues/17663), [PR #17674](https://github.com/apache/hudi/pull/17674)) provides the runtime baseline, but not an SDP contract.
+- DataSource V2 read work ([RFC-98](../rfc-98/rfc-98.md), [issue #15023](https://github.com/apache/hudi/issues/15023)) may improve the catalog read path but is not required for the initial integration.
+- The [SQL insert-overwrite discussion](https://github.com/apache/hudi/discussions/13815) shows why refresh semantics must be explicit rather than inferred from an append call.
+- [Issue #13973](https://github.com/apache/hudi/issues/13973) shows that a latest-state incremental read cannot fully propagate hard deletes. Flows that require complete delete propagation can use Hudi's existing CDC incremental format.
+- [Issue #5537](https://github.com/apache/hudi/issues/5537) tracks named-catalog support. The first version uses `HoodieCatalog` as `spark_catalog`.
+
+## Motivation
+
+### Representative use case
+
+A typical order pipeline looks like this:
+
+```text
+MySQL/PostgreSQL CDC or Kafka
+ |
+ v
+ bronze_order_events (Hudi)
+ |
+ v
+ current_orders (Hudi)
+ |
+ v
+ daily_order_metrics (Hudi)
+```
+
+The bronze table stores incoming events. The current-state table uses a record key and precombine field to reconcile duplicates and out-of-order events, including delete-marker records. Downstream streaming tables consume changed records incrementally, while batch aggregates can be materialized as views.
+
+Today this usually means several Spark applications plus custom handling for table creation, write options, checkpoints, dependencies, and restarts. With SDP, the graph lives in one project and the persisted datasets remain regular Hudi tables that other engines can query.
+
+Related user request issues include:
+
+- #19281: [multi-dataset incremental processing](https://github.com/apache/hudi/issues/19281)
+- #5189: [chained incremental Hudi tables](https://github.com/apache/hudi/issues/5189)
+- #13973: [downstream delete propagation](https://github.com/apache/hudi/issues/13973).
+
+### User value
+
+1. **Declarative pipelines.** Users define datasets as SQL or DataFrame transformations; SDP manages dependencies, checkpoints, and Hudi writes.
+2. **Managed incremental composition.** Hudi already provides incremental reads; SDP composes them into a dependency-aware pipeline, managing execution order, checkpoints, and recovery without separate Spark jobs or hand-written orchestration.
+3. **Open outputs.** Each dataset remains a standard Hudi table that can be queried independently of SDP by any Hudi-compatible engine.
+
+SDP does not make arbitrary joins or aggregates incremental; performance benefits apply only to flows that can process changes incrementally.
+
+## Goals
+
+1. Create Hudi tables from SDP datasets through `HoodieCatalog`.
+2. Write materialized views with Hudi `bulk_insert` or `insert`, following SDP's truncate-and-recompute refresh sequence.
+3. Write SDP streaming tables through Hudi's Structured Streaming sink.
+4. Support a basic incremental Hudi-to-Hudi streaming-table chain.
+5. Expose Hudi's existing keyed upsert path through SDP, including record-key, precombine, record-merger, and delete-marker handling.
+6. Keep non-SDP Hudi behavior unchanged.
+
+## Non-Goals
+
+The first version excludes:
+
+- A complete Hudi DataSource V2 read or streaming-write implementation.
+- Atomic materialized-view refresh across truncate, schema reconciliation, and the batch write.
+- Changes to Hudi's full-table truncate semantics. Timeline-preserving truncate can be developed independently.
+- Cross-table transactions across multiple SDP targets.
+- End-to-end exactly-once delivery across an arbitrary external source and Hudi. Guarantees remain scoped to Spark/Hudi micro-batch retries at one target.
+- Arbitrary named catalogs for `HoodieCatalog`; the first version uses Hudi as `spark_catalog`.
+
+## Proposed User Experience
+
+The examples below implement the representative order pipeline in dependency order, using Hudi's existing SQL table options and `SET` for flow-scoped write options.
+
+### Pipeline configuration
+
+An SDP project configures the Hudi extension and catalog like any other Hudi Spark application:
+
+```yaml
+name: hudi_orders_pipeline
+libraries:
+ - glob:
+ include: transformations/**
+storage: s3://pipelines/checkpoints/hudi_orders_pipeline
+catalog: spark_catalog
+database: lakehouse
+configuration:
+ spark.sql.extensions: org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.sql.catalog.spark_catalog: org.apache.spark.sql.hudi.catalog.HoodieCatalog
+```
+
+The SQL below assumes that `order_events_source` is an external streaming source containing parsed Kafka or database CDC events. Each `CREATE ... AS SELECT` statement defines both the target dataset and its single flow.
+
+### Streaming table with an append flow
+
+```sql
+SET spark.hoodie.datasource.write.operation = insert;
+
+CREATE STREAMING TABLE bronze_order_events
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor'
+)
+AS
+SELECT *
+FROM STREAM order_events_source;
+```
+
+SDP owns the checkpoint location and trigger; Hudi commits each micro-batch.
+
+### Streaming table with keyed upserts
+
+```sql
+SET spark.hoodie.datasource.write.operation = upsert;
+
+CREATE STREAMING TABLE current_orders
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor',
+ 'primaryKey' = 'order_id',
+ 'preCombineField' = 'event_sequence'
+)
+AS
+SELECT
+ *,
+ operation = 'DELETE' AS _hoodie_is_deleted
+FROM STREAM bronze_order_events;
+```
+
+Spark append output mode only controls how each micro-batch reaches the sink; it does not make the Hudi target append-only. `HoodieStreamingSink` still runs the configured Hudi operation, which defaults to `upsert`.
+
+For an existing Hudi table, SDP loads the record key, precombine field, partition fields, and record-merger settings from the table configuration. For a new SDP table, these settings are declared as table properties and persisted by Hudi. SDP does not implement its own deduplication, ordering, merge, or delete path. It validates the schema and configuration, then passes each micro-batch to the existing Hudi writer.
+
+### Batch materialized view
+
+```sql
+SET spark.hoodie.datasource.write.operation = bulk_insert;
+
+CREATE MATERIALIZED VIEW daily_order_metrics
+USING hudi
+PARTITIONED BY (order_date)
+TBLPROPERTIES (
+ 'type' = 'cow'
+)
+AS
+SELECT
+ order_date,
+ state,
+ COUNT(*) AS order_count
+FROM current_orders
+GROUP BY order_date, state;
+```
+
+The first run creates the table and writes the result using `bulk_insert`. Later runs truncate the target, reconcile its schema and properties, and write the recomputed result. With Hudi's current truncate implementation, each refresh removes the previous data and timeline history.
+
+## Semantics
+
+### Dataset-to-Hudi operation mapping
+
+| SDP dataset/flow | Spark 4.1 execution shape | Proposed Hudi semantics |
+| ----------------------------------- | --------------------------------------------- | -------------------------------------------------------- |
+| New materialized view | Batch append to newly created table | Hudi `bulk_insert` or `insert` |
+| Existing materialized view | Truncate, alter, then batch append | Existing Hudi truncate, then `bulk_insert` or `insert` |
+| Streaming table with an append flow | Append-mode micro-batches | Hudi `insert` or `bulk_insert`, configured per flow |
+| Hudi upsert target | Append-mode micro-batches | Existing Hudi `upsert` |
+| Hudi streaming source | Structured Streaming read with SDP checkpoint | Basic Hudi latest-state incremental source |
+
+### Materialized-view write mapping
+
+Initial materialization uses Hudi `bulk_insert` or `insert`. For an existing materialized view, Spark 4.1 executes `TRUNCATE TABLE`, reconciles schema and properties through `catalog.alterTable`, then writes the recomputed result through append-mode `saveAsTable`. Hudi uses the same configured write operation after truncate. An empty result leaves the target empty.
+
+These steps are not a single transaction. Readers may see an empty target during refresh. If schema reconciliation or the batch flow fails after truncate, the previous result is not restored, and completed catalog changes remain in effect.
+
+The initial integration reuses Hudi's current full-table truncate, which deletes old data files and timeline history. This can disrupt concurrent readers and invalidate existing incremental offsets. Timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+Materialized views are supported as batch outputs, but not as incremental sources in the initial scope.
+
+### Streaming commit and restart behavior
+
+SDP manages each flow's checkpoint location and trigger. Multiple append flows can write to the same Hudi target, each with a distinct Hudi checkpoint identifier that remains stable across restarts. Concurrent flows require Hudi's existing multi-writer configuration and an appropriate lock provider. Each flow commits and retries independently.
Review Comment:
**⚠️** "each with a distinct Hudi checkpoint identifier"
No mechanism is given, and the two available ones don't work: SDP passes no writer options to `toTable`, and per-flow `SET` doesn't reach the sink (see the comment on the `SET` examples). Concrete option: default the identifier to the Structured Streaming `queryName`, which SDP sets to the flow name. Also, when a second identifier shows up on a table without OCC configured, fail fast rather than "requires an appropriate lock provider".
##########
rfc/rfc-112/rfc-112.md:
##########
@@ -0,0 +1,429 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-112: Support Spark Declarative Pipelines
+
+## Proposers
+
+- @cshuo
+- @danny0405
+
+## Approvers
+
+- TBD
+
+## Status
+
+Issue: TBD
+
+## Abstract
+
+[Spark Declarative Pipelines (SDP)](https://spark.apache.org/docs/4.1.1/declarative-pipelines-programming-guide.html) lets users define tables and transformations while Spark builds the dependency graph and runs the batch and streaming flows. Hudi provides atomic commits, keyed updates, event ordering, deletes, incremental queries, and table services.
+
+This RFC makes Hudi a supported SDP source and target. It covers catalog-based table creation, materialized views, streaming tables, basic Hudi-to-Hudi incremental reads, and keyed upserts through Hudi's existing writer.
+
+SDP and Hudi retain their existing responsibilities:
+
+- SDP owns dataset declarations, dependency analysis, orchestration, flow checkpoints, retries, and execution order.
+- Hudi owns table storage, commit atomicity, record reconciliation, incremental read semantics, and table services.
+
+Users can build the same pipelines today with Spark jobs, Hudi writers or Hudi Streamer, and an external orchestrator. This integration removes much of that glue code while keeping the outputs as ordinary Hudi tables.
+
+## Background
+
+### Spark Declarative Pipelines
+
+Apache Spark 4.1 introduced SDP for declarative batch and streaming ETL. SDP persists two main dataset types:
+
+- A **materialized view**, which has exactly one batch flow that computes a table.
+- A **streaming table**, which has one or more streaming flows that append their output to a table.
+
+Users define pipelines in Python or SQL. SDP validates dataset references, builds the dataflow graph, runs independent flows in parallel, and stores Structured Streaming checkpoints in the pipeline storage location.
+
+In the Spark 4.1 implementation, a streaming table flow uses a Structured Streaming append-mode write to `DataStreamWriter.toTable`, and a batch table flow uses append-mode `DataFrameWriter.saveAsTable`. For an existing materialized view, SDP issues `TRUNCATE TABLE`, calls `catalog.alterTable` to reconcile schema and properties, then runs the batch flow. See Spark's [`FlowExecution`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala) and [`DatasetManager`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala).
+
+These table operations are the storage-format integration boundary; this RFC does not add a Hudi-specific execution engine.
+
+### Relevant Hudi capabilities
+
+Hudi already provides the required data-plane primitives:
+
+- `HoodieCatalog` can create and load Hudi tables through Spark's catalog APIs.
+- The Hudi batch writer supports bulk insert, insert, upsert, insert overwrite, and table insert overwrite operations. An upsert can carry delete-marker records.
+- `HoodieStreamingSink` writes Structured Streaming micro-batches and records checkpoint metadata used to avoid recommitting an already committed batch after retry.
+- Hudi's streaming source and incremental query modes can expose records changed since a prior instant.
+
+The integration exposes these capabilities through SDP's existing table and flow abstractions. Hudi Streamer, the DataFrame writer, Spark SQL DML, and direct Structured Streaming APIs remain unchanged.
+
+### Current gaps
+
+`format="hudi"` is not yet an end-to-end supported SDP contract. Some operations may already work, but table creation, refresh, restart, upsert configuration, and Hudi-to-Hudi chaining are neither documented nor covered by an integration suite.
+
+Materialized-view refresh needs end-to-end validation. SDP 4.1 truncates the target before reconciling its schema and writing the recomputed result. Hudi's current full-table `TRUNCATE TABLE` deletes the table path, including the timeline, and reinitializes its properties. This RFC reuses that behavior and documents its limitations; a timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+The integration must also address:
+
+- SDP table properties and flow-scoped write options must be resolved and passed to the Hudi writer.
+- Streaming restart must coordinate the SDP/Spark checkpoint with Hudi's sink checkpoint metadata.
+- A Hudi source used through `readStream.table` must preserve incremental offset semantics.
+- SDP must carry Hudi's persisted record-key, precombine, and record-merger settings into the existing upsert path. Deletes remain upsert input rows marked by `_hoodie_is_deleted`; Spark append output mode does not replace this contract.
+
+### Related work
+
+- Hudi's Spark 4.1 support ([issue #17663](https://github.com/apache/hudi/issues/17663), [PR #17674](https://github.com/apache/hudi/pull/17674)) provides the runtime baseline, but not an SDP contract.
+- DataSource V2 read work ([RFC-98](../rfc-98/rfc-98.md), [issue #15023](https://github.com/apache/hudi/issues/15023)) may improve the catalog read path but is not required for the initial integration.
+- The [SQL insert-overwrite discussion](https://github.com/apache/hudi/discussions/13815) shows why refresh semantics must be explicit rather than inferred from an append call.
+- [Issue #13973](https://github.com/apache/hudi/issues/13973) shows that a latest-state incremental read cannot fully propagate hard deletes. Flows that require complete delete propagation can use Hudi's existing CDC incremental format.
+- [Issue #5537](https://github.com/apache/hudi/issues/5537) tracks named-catalog support. The first version uses `HoodieCatalog` as `spark_catalog`.
+
+## Motivation
+
+### Representative use case
+
+A typical order pipeline looks like this:
+
+```text
+MySQL/PostgreSQL CDC or Kafka
+ |
+ v
+ bronze_order_events (Hudi)
+ |
+ v
+ current_orders (Hudi)
+ |
+ v
+ daily_order_metrics (Hudi)
+```
+
+The bronze table stores incoming events. The current-state table uses a record key and precombine field to reconcile duplicates and out-of-order events, including delete-marker records. Downstream streaming tables consume changed records incrementally, while batch aggregates can be materialized as views.
+
+Today this usually means several Spark applications plus custom handling for table creation, write options, checkpoints, dependencies, and restarts. With SDP, the graph lives in one project and the persisted datasets remain regular Hudi tables that other engines can query.
+
+Related user request issues include:
+
+- #19281: [multi-dataset incremental processing](https://github.com/apache/hudi/issues/19281)
+- #5189: [chained incremental Hudi tables](https://github.com/apache/hudi/issues/5189)
+- #13973: [downstream delete propagation](https://github.com/apache/hudi/issues/13973).
+
+### User value
+
+1. **Declarative pipelines.** Users define datasets as SQL or DataFrame transformations; SDP manages dependencies, checkpoints, and Hudi writes.
+2. **Managed incremental composition.** Hudi already provides incremental reads; SDP composes them into a dependency-aware pipeline, managing execution order, checkpoints, and recovery without separate Spark jobs or hand-written orchestration.
+3. **Open outputs.** Each dataset remains a standard Hudi table that can be queried independently of SDP by any Hudi-compatible engine.
+
+SDP does not make arbitrary joins or aggregates incremental; performance benefits apply only to flows that can process changes incrementally.
+
+## Goals
+
+1. Create Hudi tables from SDP datasets through `HoodieCatalog`.
+2. Write materialized views with Hudi `bulk_insert` or `insert`, following SDP's truncate-and-recompute refresh sequence.
+3. Write SDP streaming tables through Hudi's Structured Streaming sink.
+4. Support a basic incremental Hudi-to-Hudi streaming-table chain.
+5. Expose Hudi's existing keyed upsert path through SDP, including record-key, precombine, record-merger, and delete-marker handling.
+6. Keep non-SDP Hudi behavior unchanged.
+
+## Non-Goals
+
+The first version excludes:
+
+- A complete Hudi DataSource V2 read or streaming-write implementation.
+- Atomic materialized-view refresh across truncate, schema reconciliation, and the batch write.
+- Changes to Hudi's full-table truncate semantics. Timeline-preserving truncate can be developed independently.
+- Cross-table transactions across multiple SDP targets.
+- End-to-end exactly-once delivery across an arbitrary external source and Hudi. Guarantees remain scoped to Spark/Hudi micro-batch retries at one target.
+- Arbitrary named catalogs for `HoodieCatalog`; the first version uses Hudi as `spark_catalog`.
+
+## Proposed User Experience
+
+The examples below implement the representative order pipeline in dependency order, using Hudi's existing SQL table options and `SET` for flow-scoped write options.
+
+### Pipeline configuration
+
+An SDP project configures the Hudi extension and catalog like any other Hudi Spark application:
+
+```yaml
+name: hudi_orders_pipeline
+libraries:
+ - glob:
+ include: transformations/**
+storage: s3://pipelines/checkpoints/hudi_orders_pipeline
+catalog: spark_catalog
+database: lakehouse
+configuration:
+ spark.sql.extensions: org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.sql.catalog.spark_catalog: org.apache.spark.sql.hudi.catalog.HoodieCatalog
+```
+
+The SQL below assumes that `order_events_source` is an external streaming source containing parsed Kafka or database CDC events. Each `CREATE ... AS SELECT` statement defines both the target dataset and its single flow.
+
+### Streaming table with an append flow
+
+```sql
+SET spark.hoodie.datasource.write.operation = insert;
+
+CREATE STREAMING TABLE bronze_order_events
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor'
+)
+AS
+SELECT *
+FROM STREAM order_events_source;
+```
+
+SDP owns the checkpoint location and trigger; Hudi commits each micro-batch.
+
+### Streaming table with keyed upserts
+
+```sql
+SET spark.hoodie.datasource.write.operation = upsert;
+
+CREATE STREAMING TABLE current_orders
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor',
+ 'primaryKey' = 'order_id',
+ 'preCombineField' = 'event_sequence'
+)
+AS
+SELECT
+ *,
+ operation = 'DELETE' AS _hoodie_is_deleted
+FROM STREAM bronze_order_events;
+```
+
+Spark append output mode only controls how each micro-batch reaches the sink; it does not make the Hudi target append-only. `HoodieStreamingSink` still runs the configured Hudi operation, which defaults to `upsert`.
+
+For an existing Hudi table, SDP loads the record key, precombine field, partition fields, and record-merger settings from the table configuration. For a new SDP table, these settings are declared as table properties and persisted by Hudi. SDP does not implement its own deduplication, ordering, merge, or delete path. It validates the schema and configuration, then passes each micro-batch to the existing Hudi writer.
+
+### Batch materialized view
+
+```sql
+SET spark.hoodie.datasource.write.operation = bulk_insert;
+
+CREATE MATERIALIZED VIEW daily_order_metrics
+USING hudi
+PARTITIONED BY (order_date)
+TBLPROPERTIES (
+ 'type' = 'cow'
+)
+AS
+SELECT
+ order_date,
+ state,
+ COUNT(*) AS order_count
+FROM current_orders
+GROUP BY order_date, state;
+```
+
+The first run creates the table and writes the result using `bulk_insert`. Later runs truncate the target, reconcile its schema and properties, and write the recomputed result. With Hudi's current truncate implementation, each refresh removes the previous data and timeline history.
+
+## Semantics
+
+### Dataset-to-Hudi operation mapping
+
+| SDP dataset/flow | Spark 4.1 execution shape | Proposed Hudi semantics |
+| ----------------------------------- | --------------------------------------------- | -------------------------------------------------------- |
+| New materialized view | Batch append to newly created table | Hudi `bulk_insert` or `insert` |
+| Existing materialized view | Truncate, alter, then batch append | Existing Hudi truncate, then `bulk_insert` or `insert` |
+| Streaming table with an append flow | Append-mode micro-batches | Hudi `insert` or `bulk_insert`, configured per flow |
+| Hudi upsert target | Append-mode micro-batches | Existing Hudi `upsert` |
+| Hudi streaming source | Structured Streaming read with SDP checkpoint | Basic Hudi latest-state incremental source |
+
+### Materialized-view write mapping
+
+Initial materialization uses Hudi `bulk_insert` or `insert`. For an existing materialized view, Spark 4.1 executes `TRUNCATE TABLE`, reconciles schema and properties through `catalog.alterTable`, then writes the recomputed result through append-mode `saveAsTable`. Hudi uses the same configured write operation after truncate. An empty result leaves the target empty.
+
+These steps are not a single transaction. Readers may see an empty target during refresh. If schema reconciliation or the batch flow fails after truncate, the previous result is not restored, and completed catalog changes remain in effect.
+
+The initial integration reuses Hudi's current full-table truncate, which deletes old data files and timeline history. This can disrupt concurrent readers and invalidate existing incremental offsets. Timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+Materialized views are supported as batch outputs, but not as incremental sources in the initial scope.
+
+### Streaming commit and restart behavior
+
+SDP manages each flow's checkpoint location and trigger. Multiple append flows can write to the same Hudi target, each with a distinct Hudi checkpoint identifier that remains stable across restarts. Concurrent flows require Hudi's existing multi-writer configuration and an appropriate lock provider. Each flow commits and retries independently.
+
+Normal restarts reuse the same Spark checkpoint and Hudi checkpoint identifier. A streaming-table full refresh starts a fresh checkpoint and uses Hudi's current full-table truncate to remove target data, timeline history, and sink checkpoint metadata before rebuilding the table. Resetting the Spark checkpoint while retaining the Hudi target is outside this RFC's scope.
+
+### Configuration ownership and precedence
+
+Configuration is split by ownership:
+
+- **Table identity:** table type, record key, precombine field, partition fields, key generator, and record-merger settings. Hudi persists these values; SDP loads them for existing tables, and a pipeline run cannot silently change them.
+- **Write behavior:** operation and writer tuning, supplied through SQL `SET spark.hoodie.*` as flow-scoped write options.
+- **Flow execution:** checkpoint location, trigger, retry, and flow identity. SDP owns these values; the Hudi integration maps each flow to a distinct sink checkpoint identifier.
+
+SDP captures the active `SET` values at each flow definition; a `SET` remains in effect for subsequent definitions until overridden. Flow settings override pipeline and Hudi defaults and must be compatible with the persisted table configuration. The integration strips the `spark.` prefix from write options and reuses `HoodieOptionConfig` to normalize table options. Flows writing to the same target must use compatible concurrency and lock settings. The sink checkpoint identifier is resolved per flow, not shared as a table property.
+
+Invalid properties should fail during SDP `dry-run` when possible, and otherwise before the first write.
+
+### Table services
+
+Table services follow the existing Hudi writer behavior, with no SDP-specific scheduling or retention defaults. Integration tests cover service shutdown, recovery, and resource cleanup when flows finish, stop, or restart, including multiple flows sharing a target.
+
+### Schema and partition evolution
+
+SDP follows Hudi's existing schema and partition evolution rules.
+
+- Hudi-supported additive schema changes may be accepted for streaming tables.
+- Dropping, renaming, or changing columns follows Hudi's configured schema evolution behavior.
+- Record-key, precombine, and partition columns must exist in the target schema.
+- Partitioning changes fail; SDP does not turn them into an implicit table replacement.
+
+`HoodieCatalog.alterTable` validates the complete set of requested schema and property changes before applying supported changes. It accepts unchanged properties and rejects unsupported changes or changes to table identity. The Hudi writer validates the actual input schema against the persisted table configuration using its existing schema reconciliation rules.
+
+For materialized views, Spark invokes `alterTable` after truncate. A schema validation failure can therefore leave the target empty; it does not preserve the previous result. If metadata changes succeed but the batch write fails, those changes remain in effect.
+
+## Implementation
+
+### 1. Catalog integration
+
+Extend and test [HoodieCatalog](../../hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala) so SDP can:
+
+- Create a table whose provider is `hudi` from SDP's table information.
+- Persist schema, location, partition transforms, comments, and Hudi table properties.
+- Load the created table for both batch and streaming execution.
+- Validate the complete alter request before applying supported property and schema changes, preserving Hudi table identity and accepting unchanged properties.
+- Reject unsupported partition transforms and incompatible property changes with actionable errors.
+
+The first version configures `HoodieCatalog` as `spark_catalog`; named catalogs are follow-up work.
+
+This path may continue to use Hudi's V1 fallback. Full DataSource V2 support is not required as long as the SDP table contract is met.
+
+### 2. Materialized-view write path
+
+Use Hudi `bulk_insert` or `insert` for the batch result. For refresh, reuse [TruncateHoodieTableCommand](../../hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/TruncateHoodieTableCommand.scala), support the subsequent catalog reconciliation, and pass the configured operation to the append-mode writer. No SDP-specific truncate mode or refresh-detection mechanism is introduced. Validate this sequence through the integration harness before declaring materialized-view refresh supported.
+
+### 3. Streaming target integration
+
+Route SDP's append-mode `toTable` call through [HoodieStreamingSink](../../hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieStreamingSink.scala), including table properties supplied by `HoodieCatalog`.
+
+The sink integration will:
+
+- Pass the flow's write operation and persisted keyed-write settings to the existing Hudi writer.
+- Preserve the SDP checkpoint location rather than creating a competing checkpoint tree.
+- Resolve a stable, per-flow `hoodie.datasource.write.streaming.checkpoint.identifier` and use it for both retry checks and commit metadata.
+- Pass through and validate Hudi's existing multi-writer and lock configuration for concurrent flows sharing a target.
+- Reuse Hudi's existing streaming-sink checkpoint and retry behavior.
+
+No SDP-specific upsert writer, merge logic, or delete path is introduced.
+
+### 4. Incremental source integration
+
+Validate both catalog-table and explicit Hudi sources in SDP streaming flows. Source progress is restored from the SDP-managed checkpoint and advances with the downstream flow.
Review Comment:
**đź’¬** Incremental source via `readStream.table`
Works through the V1 fallback, but `HoodieStreamSourceV1/V2` implement neither `SupportsAdmissionControl` nor `SupportsTriggerAvailableNow`. Under `AvailableNow` the end offset is captured at start and the whole backlog becomes one micro-batch, so a Hudi→Hudi flow that was idle for a day upserts a day of instants in one commit. Not a v1 blocker, but it belongs in the source contract and is the most valuable piece of DSv2 work (ahead of `STREAMING_WRITE`).
##########
rfc/rfc-112/rfc-112.md:
##########
@@ -0,0 +1,429 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-112: Support Spark Declarative Pipelines
+
+## Proposers
+
+- @cshuo
+- @danny0405
+
+## Approvers
+
+- TBD
+
+## Status
+
+Issue: TBD
+
+## Abstract
+
+[Spark Declarative Pipelines (SDP)](https://spark.apache.org/docs/4.1.1/declarative-pipelines-programming-guide.html) lets users define tables and transformations while Spark builds the dependency graph and runs the batch and streaming flows. Hudi provides atomic commits, keyed updates, event ordering, deletes, incremental queries, and table services.
+
+This RFC makes Hudi a supported SDP source and target. It covers catalog-based table creation, materialized views, streaming tables, basic Hudi-to-Hudi incremental reads, and keyed upserts through Hudi's existing writer.
+
+SDP and Hudi retain their existing responsibilities:
+
+- SDP owns dataset declarations, dependency analysis, orchestration, flow checkpoints, retries, and execution order.
+- Hudi owns table storage, commit atomicity, record reconciliation, incremental read semantics, and table services.
+
+Users can build the same pipelines today with Spark jobs, Hudi writers or Hudi Streamer, and an external orchestrator. This integration removes much of that glue code while keeping the outputs as ordinary Hudi tables.
+
+## Background
+
+### Spark Declarative Pipelines
+
+Apache Spark 4.1 introduced SDP for declarative batch and streaming ETL. SDP persists two main dataset types:
+
+- A **materialized view**, which has exactly one batch flow that computes a table.
+- A **streaming table**, which has one or more streaming flows that append their output to a table.
+
+Users define pipelines in Python or SQL. SDP validates dataset references, builds the dataflow graph, runs independent flows in parallel, and stores Structured Streaming checkpoints in the pipeline storage location.
+
+In the Spark 4.1 implementation, a streaming table flow uses a Structured Streaming append-mode write to `DataStreamWriter.toTable`, and a batch table flow uses append-mode `DataFrameWriter.saveAsTable`. For an existing materialized view, SDP issues `TRUNCATE TABLE`, calls `catalog.alterTable` to reconcile schema and properties, then runs the batch flow. See Spark's [`FlowExecution`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala) and [`DatasetManager`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala).
+
+These table operations are the storage-format integration boundary; this RFC does not add a Hudi-specific execution engine.
+
+### Relevant Hudi capabilities
+
+Hudi already provides the required data-plane primitives:
+
+- `HoodieCatalog` can create and load Hudi tables through Spark's catalog APIs.
+- The Hudi batch writer supports bulk insert, insert, upsert, insert overwrite, and table insert overwrite operations. An upsert can carry delete-marker records.
+- `HoodieStreamingSink` writes Structured Streaming micro-batches and records checkpoint metadata used to avoid recommitting an already committed batch after retry.
+- Hudi's streaming source and incremental query modes can expose records changed since a prior instant.
+
+The integration exposes these capabilities through SDP's existing table and flow abstractions. Hudi Streamer, the DataFrame writer, Spark SQL DML, and direct Structured Streaming APIs remain unchanged.
+
+### Current gaps
+
+`format="hudi"` is not yet an end-to-end supported SDP contract. Some operations may already work, but table creation, refresh, restart, upsert configuration, and Hudi-to-Hudi chaining are neither documented nor covered by an integration suite.
+
+Materialized-view refresh needs end-to-end validation. SDP 4.1 truncates the target before reconciling its schema and writing the recomputed result. Hudi's current full-table `TRUNCATE TABLE` deletes the table path, including the timeline, and reinitializes its properties. This RFC reuses that behavior and documents its limitations; a timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+The integration must also address:
+
+- SDP table properties and flow-scoped write options must be resolved and passed to the Hudi writer.
+- Streaming restart must coordinate the SDP/Spark checkpoint with Hudi's sink checkpoint metadata.
+- A Hudi source used through `readStream.table` must preserve incremental offset semantics.
+- SDP must carry Hudi's persisted record-key, precombine, and record-merger settings into the existing upsert path. Deletes remain upsert input rows marked by `_hoodie_is_deleted`; Spark append output mode does not replace this contract.
+
+### Related work
+
+- Hudi's Spark 4.1 support ([issue #17663](https://github.com/apache/hudi/issues/17663), [PR #17674](https://github.com/apache/hudi/pull/17674)) provides the runtime baseline, but not an SDP contract.
+- DataSource V2 read work ([RFC-98](../rfc-98/rfc-98.md), [issue #15023](https://github.com/apache/hudi/issues/15023)) may improve the catalog read path but is not required for the initial integration.
+- The [SQL insert-overwrite discussion](https://github.com/apache/hudi/discussions/13815) shows why refresh semantics must be explicit rather than inferred from an append call.
+- [Issue #13973](https://github.com/apache/hudi/issues/13973) shows that a latest-state incremental read cannot fully propagate hard deletes. Flows that require complete delete propagation can use Hudi's existing CDC incremental format.
+- [Issue #5537](https://github.com/apache/hudi/issues/5537) tracks named-catalog support. The first version uses `HoodieCatalog` as `spark_catalog`.
+
+## Motivation
+
+### Representative use case
+
+A typical order pipeline looks like this:
+
+```text
+MySQL/PostgreSQL CDC or Kafka
+ |
+ v
+ bronze_order_events (Hudi)
+ |
+ v
+ current_orders (Hudi)
+ |
+ v
+ daily_order_metrics (Hudi)
+```
+
+The bronze table stores incoming events. The current-state table uses a record key and precombine field to reconcile duplicates and out-of-order events, including delete-marker records. Downstream streaming tables consume changed records incrementally, while batch aggregates can be materialized as views.
+
+Today this usually means several Spark applications plus custom handling for table creation, write options, checkpoints, dependencies, and restarts. With SDP, the graph lives in one project and the persisted datasets remain regular Hudi tables that other engines can query.
+
+Related user request issues include:
+
+- #19281: [multi-dataset incremental processing](https://github.com/apache/hudi/issues/19281)
+- #5189: [chained incremental Hudi tables](https://github.com/apache/hudi/issues/5189)
+- #13973: [downstream delete propagation](https://github.com/apache/hudi/issues/13973).
+
+### User value
+
+1. **Declarative pipelines.** Users define datasets as SQL or DataFrame transformations; SDP manages dependencies, checkpoints, and Hudi writes.
+2. **Managed incremental composition.** Hudi already provides incremental reads; SDP composes them into a dependency-aware pipeline, managing execution order, checkpoints, and recovery without separate Spark jobs or hand-written orchestration.
+3. **Open outputs.** Each dataset remains a standard Hudi table that can be queried independently of SDP by any Hudi-compatible engine.
+
+SDP does not make arbitrary joins or aggregates incremental; performance benefits apply only to flows that can process changes incrementally.
+
+## Goals
+
+1. Create Hudi tables from SDP datasets through `HoodieCatalog`.
+2. Write materialized views with Hudi `bulk_insert` or `insert`, following SDP's truncate-and-recompute refresh sequence.
+3. Write SDP streaming tables through Hudi's Structured Streaming sink.
+4. Support a basic incremental Hudi-to-Hudi streaming-table chain.
+5. Expose Hudi's existing keyed upsert path through SDP, including record-key, precombine, record-merger, and delete-marker handling.
+6. Keep non-SDP Hudi behavior unchanged.
+
+## Non-Goals
+
+The first version excludes:
+
+- A complete Hudi DataSource V2 read or streaming-write implementation.
+- Atomic materialized-view refresh across truncate, schema reconciliation, and the batch write.
+- Changes to Hudi's full-table truncate semantics. Timeline-preserving truncate can be developed independently.
+- Cross-table transactions across multiple SDP targets.
+- End-to-end exactly-once delivery across an arbitrary external source and Hudi. Guarantees remain scoped to Spark/Hudi micro-batch retries at one target.
+- Arbitrary named catalogs for `HoodieCatalog`; the first version uses Hudi as `spark_catalog`.
+
+## Proposed User Experience
+
+The examples below implement the representative order pipeline in dependency order, using Hudi's existing SQL table options and `SET` for flow-scoped write options.
+
+### Pipeline configuration
+
+An SDP project configures the Hudi extension and catalog like any other Hudi Spark application:
+
+```yaml
+name: hudi_orders_pipeline
+libraries:
+ - glob:
+ include: transformations/**
+storage: s3://pipelines/checkpoints/hudi_orders_pipeline
+catalog: spark_catalog
+database: lakehouse
+configuration:
+ spark.sql.extensions: org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.sql.catalog.spark_catalog: org.apache.spark.sql.hudi.catalog.HoodieCatalog
+```
+
+The SQL below assumes that `order_events_source` is an external streaming source containing parsed Kafka or database CDC events. Each `CREATE ... AS SELECT` statement defines both the target dataset and its single flow.
+
+### Streaming table with an append flow
+
+```sql
+SET spark.hoodie.datasource.write.operation = insert;
+
+CREATE STREAMING TABLE bronze_order_events
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor'
+)
+AS
+SELECT *
+FROM STREAM order_events_source;
+```
+
+SDP owns the checkpoint location and trigger; Hudi commits each micro-batch.
+
+### Streaming table with keyed upserts
+
+```sql
+SET spark.hoodie.datasource.write.operation = upsert;
+
+CREATE STREAMING TABLE current_orders
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor',
+ 'primaryKey' = 'order_id',
+ 'preCombineField' = 'event_sequence'
+)
+AS
+SELECT
+ *,
+ operation = 'DELETE' AS _hoodie_is_deleted
+FROM STREAM bronze_order_events;
+```
+
+Spark append output mode only controls how each micro-batch reaches the sink; it does not make the Hudi target append-only. `HoodieStreamingSink` still runs the configured Hudi operation, which defaults to `upsert`.
+
+For an existing Hudi table, SDP loads the record key, precombine field, partition fields, and record-merger settings from the table configuration. For a new SDP table, these settings are declared as table properties and persisted by Hudi. SDP does not implement its own deduplication, ordering, merge, or delete path. It validates the schema and configuration, then passes each micro-batch to the existing Hudi writer.
+
+### Batch materialized view
+
+```sql
+SET spark.hoodie.datasource.write.operation = bulk_insert;
+
+CREATE MATERIALIZED VIEW daily_order_metrics
+USING hudi
+PARTITIONED BY (order_date)
+TBLPROPERTIES (
+ 'type' = 'cow'
+)
+AS
+SELECT
+ order_date,
+ state,
+ COUNT(*) AS order_count
+FROM current_orders
+GROUP BY order_date, state;
+```
+
+The first run creates the table and writes the result using `bulk_insert`. Later runs truncate the target, reconcile its schema and properties, and write the recomputed result. With Hudi's current truncate implementation, each refresh removes the previous data and timeline history.
+
+## Semantics
+
+### Dataset-to-Hudi operation mapping
+
+| SDP dataset/flow | Spark 4.1 execution shape | Proposed Hudi semantics |
+| ----------------------------------- | --------------------------------------------- | -------------------------------------------------------- |
+| New materialized view | Batch append to newly created table | Hudi `bulk_insert` or `insert` |
+| Existing materialized view | Truncate, alter, then batch append | Existing Hudi truncate, then `bulk_insert` or `insert` |
+| Streaming table with an append flow | Append-mode micro-batches | Hudi `insert` or `bulk_insert`, configured per flow |
+| Hudi upsert target | Append-mode micro-batches | Existing Hudi `upsert` |
+| Hudi streaming source | Structured Streaming read with SDP checkpoint | Basic Hudi latest-state incremental source |
+
+### Materialized-view write mapping
+
+Initial materialization uses Hudi `bulk_insert` or `insert`. For an existing materialized view, Spark 4.1 executes `TRUNCATE TABLE`, reconciles schema and properties through `catalog.alterTable`, then writes the recomputed result through append-mode `saveAsTable`. Hudi uses the same configured write operation after truncate. An empty result leaves the target empty.
+
+These steps are not a single transaction. Readers may see an empty target during refresh. If schema reconciliation or the batch flow fails after truncate, the previous result is not restored, and completed catalog changes remain in effect.
+
+The initial integration reuses Hudi's current full-table truncate, which deletes old data files and timeline history. This can disrupt concurrent readers and invalidate existing incremental offsets. Timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+Materialized views are supported as batch outputs, but not as incremental sources in the initial scope.
+
+### Streaming commit and restart behavior
+
+SDP manages each flow's checkpoint location and trigger. Multiple append flows can write to the same Hudi target, each with a distinct Hudi checkpoint identifier that remains stable across restarts. Concurrent flows require Hudi's existing multi-writer configuration and an appropriate lock provider. Each flow commits and retries independently.
+
+Normal restarts reuse the same Spark checkpoint and Hudi checkpoint identifier. A streaming-table full refresh starts a fresh checkpoint and uses Hudi's current full-table truncate to remove target data, timeline history, and sink checkpoint metadata before rebuilding the table. Resetting the Spark checkpoint while retaining the Hudi target is outside this RFC's scope.
+
+### Configuration ownership and precedence
+
+Configuration is split by ownership:
+
+- **Table identity:** table type, record key, precombine field, partition fields, key generator, and record-merger settings. Hudi persists these values; SDP loads them for existing tables, and a pipeline run cannot silently change them.
+- **Write behavior:** operation and writer tuning, supplied through SQL `SET spark.hoodie.*` as flow-scoped write options.
+- **Flow execution:** checkpoint location, trigger, retry, and flow identity. SDP owns these values; the Hudi integration maps each flow to a distinct sink checkpoint identifier.
+
+SDP captures the active `SET` values at each flow definition; a `SET` remains in effect for subsequent definitions until overridden. Flow settings override pipeline and Hudi defaults and must be compatible with the persisted table configuration. The integration strips the `spark.` prefix from write options and reuses `HoodieOptionConfig` to normalize table options. Flows writing to the same target must use compatible concurrency and lock settings. The sink checkpoint identifier is resolved per flow, not shared as a table property.
+
+Invalid properties should fail during SDP `dry-run` when possible, and otherwise before the first write.
+
+### Table services
+
+Table services follow the existing Hudi writer behavior, with no SDP-specific scheduling or retention defaults. Integration tests cover service shutdown, recovery, and resource cleanup when flows finish, stop, or restart, including multiple flows sharing a target.
+
+### Schema and partition evolution
+
+SDP follows Hudi's existing schema and partition evolution rules.
+
+- Hudi-supported additive schema changes may be accepted for streaming tables.
+- Dropping, renaming, or changing columns follows Hudi's configured schema evolution behavior.
+- Record-key, precombine, and partition columns must exist in the target schema.
+- Partitioning changes fail; SDP does not turn them into an implicit table replacement.
+
+`HoodieCatalog.alterTable` validates the complete set of requested schema and property changes before applying supported changes. It accepts unchanged properties and rejects unsupported changes or changes to table identity. The Hudi writer validates the actual input schema against the persisted table configuration using its existing schema reconciliation rules.
+
+For materialized views, Spark invokes `alterTable` after truncate. A schema validation failure can therefore leave the target empty; it does not preserve the previous result. If metadata changes succeed but the batch write fails, those changes remain in effect.
+
+## Implementation
+
+### 1. Catalog integration
+
+Extend and test [HoodieCatalog](../../hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala) so SDP can:
+
+- Create a table whose provider is `hudi` from SDP's table information.
+- Persist schema, location, partition transforms, comments, and Hudi table properties.
+- Load the created table for both batch and streaming execution.
+- Validate the complete alter request before applying supported property and schema changes, preserving Hudi table identity and accepting unchanged properties.
+- Reject unsupported partition transforms and incompatible property changes with actionable errors.
+
+The first version configures `HoodieCatalog` as `spark_catalog`; named catalogs are follow-up work.
+
+This path may continue to use Hudi's V1 fallback. Full DataSource V2 support is not required as long as the SDP table contract is met.
+
+### 2. Materialized-view write path
+
+Use Hudi `bulk_insert` or `insert` for the batch result. For refresh, reuse [TruncateHoodieTableCommand](../../hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/TruncateHoodieTableCommand.scala), support the subsequent catalog reconciliation, and pass the configured operation to the append-mode writer. No SDP-specific truncate mode or refresh-detection mechanism is introduced. Validate this sequence through the integration harness before declaring materialized-view refresh supported.
+
+### 3. Streaming target integration
+
+Route SDP's append-mode `toTable` call through [HoodieStreamingSink](../../hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieStreamingSink.scala), including table properties supplied by `HoodieCatalog`.
+
+The sink integration will:
+
+- Pass the flow's write operation and persisted keyed-write settings to the existing Hudi writer.
+- Preserve the SDP checkpoint location rather than creating a competing checkpoint tree.
+- Resolve a stable, per-flow `hoodie.datasource.write.streaming.checkpoint.identifier` and use it for both retry checks and commit metadata.
+- Pass through and validate Hudi's existing multi-writer and lock configuration for concurrent flows sharing a target.
+- Reuse Hudi's existing streaming-sink checkpoint and retry behavior.
+
+No SDP-specific upsert writer, merge logic, or delete path is introduced.
+
+### 4. Incremental source integration
+
+Validate both catalog-table and explicit Hudi sources in SDP streaming flows. Source progress is restored from the SDP-managed checkpoint and advances with the downstream flow.
+
+Document:
+
+- Initial position behavior (`earliest`, `latest`, or an explicit instant).
+- The basic `latest-state` incremental output contract.
+- Behavior when required source instants have been archived or cleaned.
+- COW and MOR differences, if any, in supported streaming query modes.
+- Metadata-column handling when one Hudi table feeds another.
+
+If the checkpoint refers to source history that is no longer available, fail with a recovery error rather than skipping data.
+
+### 5. Upsert configuration and validation
+
+For an upsert target, validate that:
+
+- At least one record-key field is configured and present.
+- All partition fields are present.
+- Configured precombine fields are present and type-compatible.
+- `_hoodie_is_deleted`, when present, is boolean.
+
+These checks only validate the input contract. Hudi's existing writer, precombine, and record-merger logic perform the upsert.
+
+## Failure and Consistency Model
+
+| Failure point | Required outcome |
+| ------------------------------------------------------ | ------------------------------------------------------------------------------ |
+| Validation fails before truncate starts | Existing target remains unchanged |
+| Full-table truncate fails | Files may already have been removed; no automatic restoration is guaranteed |
+| Schema reconciliation or MV computation fails after truncate | Previous result is not restored; completed metadata changes remain in effect |
+| MV batch write fails before commit | No completed commit for that write; prior data removed by truncate is not restored |
+| Streaming task fails before Hudi commit | Spark retries the micro-batch |
+| Streaming flow retries or restarts with its existing checkpoint | Existing Hudi streaming-sink checkpoint behavior is preserved |
+| Source checkpoint points to expired Hudi history | Flow fails with an explicit recovery error |
+| Pipeline definition changes record key or partitioning | Validation fails; no implicit migration occurs, but an earlier truncate is not undone |
+
+Hudi writes retain per-commit atomicity. The current full-table truncate is a filesystem operation, not a Hudi commit, and the refresh sequence has no transaction or rollback spanning truncate, catalog changes, and the batch write. Commits from independent SDP flows are not atomic as a group.
+
+## Rollout/Adoption Plan
+
+### Phase 1: Basic catalog target
+
+- Support Spark 4.1.x with `HoodieCatalog` configured as `spark_catalog`.
+- Materialize a new, trivial SDP dataset as a Hudi-readable table.
+- Add the SDP integration harness and documentation.
+
+### Phase 2: Materialized-view refresh
+
+- Write the initial materialized-view result through Hudi `bulk_insert` or `insert`.
+- Refresh existing targets through Hudi's current truncate, catalog reconciliation, and batch write.
+- Validate repeated runs and document the history, reader, and failure limitations.
+
+### Phase 3: Streaming tables and chaining
+
+- Support one or more append streaming flows writing to the same target through the Hudi streaming sink.
+- Validate source-to-Hudi streaming-table writes.
+- Validate an upstream Hudi streaming table feeding a downstream streaming table through a basic incremental read.
+
+### Phase 4: Upsert configuration and verification
+
+- Verify that record-key, precombine, partition, and record-merger settings reach the existing Hudi writer.
+- Verify that `_hoodie_is_deleted` records use the same upsert path.
+- Add functional coverage for the resulting Hudi table state.
+
+Each phase is independently testable. Existing Hudi jobs are unaffected; the integration is used only by SDP projects that declare Hudi-backed datasets.
+
+## Test Plan
+
+The integration suite covers the four milestones:
Review Comment:
**đź’¬** Test plan
Please name the harness: the real `spark-pipelines` CLI from a Python venv against a Connect server (the only thing that exercises the configuration issue above) vs. Scala tests against `sql/pipelines`' `DataflowGraph`. Also `spark4.1` is currently commented out in `.github/workflows/bot.yml`; Phase 1 needs it on. For what it's worth, a reproducer needs no Hudi build: `pip install "pyspark[pipelines,connect]==4.1.1"` + the 4.1 bundle from Maven Central.
##########
rfc/rfc-112/rfc-112.md:
##########
@@ -0,0 +1,429 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-112: Support Spark Declarative Pipelines
+
+## Proposers
+
+- @cshuo
+- @danny0405
+
+## Approvers
+
+- TBD
+
+## Status
+
+Issue: TBD
+
+## Abstract
+
+[Spark Declarative Pipelines (SDP)](https://spark.apache.org/docs/4.1.1/declarative-pipelines-programming-guide.html) lets users define tables and transformations while Spark builds the dependency graph and runs the batch and streaming flows. Hudi provides atomic commits, keyed updates, event ordering, deletes, incremental queries, and table services.
+
+This RFC makes Hudi a supported SDP source and target. It covers catalog-based table creation, materialized views, streaming tables, basic Hudi-to-Hudi incremental reads, and keyed upserts through Hudi's existing writer.
+
+SDP and Hudi retain their existing responsibilities:
+
+- SDP owns dataset declarations, dependency analysis, orchestration, flow checkpoints, retries, and execution order.
+- Hudi owns table storage, commit atomicity, record reconciliation, incremental read semantics, and table services.
+
+Users can build the same pipelines today with Spark jobs, Hudi writers or Hudi Streamer, and an external orchestrator. This integration removes much of that glue code while keeping the outputs as ordinary Hudi tables.
+
+## Background
+
+### Spark Declarative Pipelines
+
+Apache Spark 4.1 introduced SDP for declarative batch and streaming ETL. SDP persists two main dataset types:
+
+- A **materialized view**, which has exactly one batch flow that computes a table.
+- A **streaming table**, which has one or more streaming flows that append their output to a table.
+
+Users define pipelines in Python or SQL. SDP validates dataset references, builds the dataflow graph, runs independent flows in parallel, and stores Structured Streaming checkpoints in the pipeline storage location.
+
+In the Spark 4.1 implementation, a streaming table flow uses a Structured Streaming append-mode write to `DataStreamWriter.toTable`, and a batch table flow uses append-mode `DataFrameWriter.saveAsTable`. For an existing materialized view, SDP issues `TRUNCATE TABLE`, calls `catalog.alterTable` to reconcile schema and properties, then runs the batch flow. See Spark's [`FlowExecution`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala) and [`DatasetManager`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala).
+
+These table operations are the storage-format integration boundary; this RFC does not add a Hudi-specific execution engine.
+
+### Relevant Hudi capabilities
+
+Hudi already provides the required data-plane primitives:
+
+- `HoodieCatalog` can create and load Hudi tables through Spark's catalog APIs.
+- The Hudi batch writer supports bulk insert, insert, upsert, insert overwrite, and table insert overwrite operations. An upsert can carry delete-marker records.
+- `HoodieStreamingSink` writes Structured Streaming micro-batches and records checkpoint metadata used to avoid recommitting an already committed batch after retry.
+- Hudi's streaming source and incremental query modes can expose records changed since a prior instant.
+
+The integration exposes these capabilities through SDP's existing table and flow abstractions. Hudi Streamer, the DataFrame writer, Spark SQL DML, and direct Structured Streaming APIs remain unchanged.
+
+### Current gaps
+
+`format="hudi"` is not yet an end-to-end supported SDP contract. Some operations may already work, but table creation, refresh, restart, upsert configuration, and Hudi-to-Hudi chaining are neither documented nor covered by an integration suite.
+
+Materialized-view refresh needs end-to-end validation. SDP 4.1 truncates the target before reconciling its schema and writing the recomputed result. Hudi's current full-table `TRUNCATE TABLE` deletes the table path, including the timeline, and reinitializes its properties. This RFC reuses that behavior and documents its limitations; a timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+The integration must also address:
+
+- SDP table properties and flow-scoped write options must be resolved and passed to the Hudi writer.
+- Streaming restart must coordinate the SDP/Spark checkpoint with Hudi's sink checkpoint metadata.
+- A Hudi source used through `readStream.table` must preserve incremental offset semantics.
+- SDP must carry Hudi's persisted record-key, precombine, and record-merger settings into the existing upsert path. Deletes remain upsert input rows marked by `_hoodie_is_deleted`; Spark append output mode does not replace this contract.
+
+### Related work
+
+- Hudi's Spark 4.1 support ([issue #17663](https://github.com/apache/hudi/issues/17663), [PR #17674](https://github.com/apache/hudi/pull/17674)) provides the runtime baseline, but not an SDP contract.
+- DataSource V2 read work ([RFC-98](../rfc-98/rfc-98.md), [issue #15023](https://github.com/apache/hudi/issues/15023)) may improve the catalog read path but is not required for the initial integration.
+- The [SQL insert-overwrite discussion](https://github.com/apache/hudi/discussions/13815) shows why refresh semantics must be explicit rather than inferred from an append call.
+- [Issue #13973](https://github.com/apache/hudi/issues/13973) shows that a latest-state incremental read cannot fully propagate hard deletes. Flows that require complete delete propagation can use Hudi's existing CDC incremental format.
+- [Issue #5537](https://github.com/apache/hudi/issues/5537) tracks named-catalog support. The first version uses `HoodieCatalog` as `spark_catalog`.
+
+## Motivation
+
+### Representative use case
+
+A typical order pipeline looks like this:
+
+```text
+MySQL/PostgreSQL CDC or Kafka
+ |
+ v
+ bronze_order_events (Hudi)
+ |
+ v
+ current_orders (Hudi)
+ |
+ v
+ daily_order_metrics (Hudi)
+```
+
+The bronze table stores incoming events. The current-state table uses a record key and precombine field to reconcile duplicates and out-of-order events, including delete-marker records. Downstream streaming tables consume changed records incrementally, while batch aggregates can be materialized as views.
+
+Today this usually means several Spark applications plus custom handling for table creation, write options, checkpoints, dependencies, and restarts. With SDP, the graph lives in one project and the persisted datasets remain regular Hudi tables that other engines can query.
+
+Related user request issues include:
+
+- #19281: [multi-dataset incremental processing](https://github.com/apache/hudi/issues/19281)
+- #5189: [chained incremental Hudi tables](https://github.com/apache/hudi/issues/5189)
+- #13973: [downstream delete propagation](https://github.com/apache/hudi/issues/13973).
+
+### User value
+
+1. **Declarative pipelines.** Users define datasets as SQL or DataFrame transformations; SDP manages dependencies, checkpoints, and Hudi writes.
+2. **Managed incremental composition.** Hudi already provides incremental reads; SDP composes them into a dependency-aware pipeline, managing execution order, checkpoints, and recovery without separate Spark jobs or hand-written orchestration.
+3. **Open outputs.** Each dataset remains a standard Hudi table that can be queried independently of SDP by any Hudi-compatible engine.
+
+SDP does not make arbitrary joins or aggregates incremental; performance benefits apply only to flows that can process changes incrementally.
+
+## Goals
+
+1. Create Hudi tables from SDP datasets through `HoodieCatalog`.
+2. Write materialized views with Hudi `bulk_insert` or `insert`, following SDP's truncate-and-recompute refresh sequence.
+3. Write SDP streaming tables through Hudi's Structured Streaming sink.
+4. Support a basic incremental Hudi-to-Hudi streaming-table chain.
+5. Expose Hudi's existing keyed upsert path through SDP, including record-key, precombine, record-merger, and delete-marker handling.
+6. Keep non-SDP Hudi behavior unchanged.
+
+## Non-Goals
+
+The first version excludes:
+
+- A complete Hudi DataSource V2 read or streaming-write implementation.
+- Atomic materialized-view refresh across truncate, schema reconciliation, and the batch write.
+- Changes to Hudi's full-table truncate semantics. Timeline-preserving truncate can be developed independently.
+- Cross-table transactions across multiple SDP targets.
+- End-to-end exactly-once delivery across an arbitrary external source and Hudi. Guarantees remain scoped to Spark/Hudi micro-batch retries at one target.
+- Arbitrary named catalogs for `HoodieCatalog`; the first version uses Hudi as `spark_catalog`.
+
+## Proposed User Experience
+
+The examples below implement the representative order pipeline in dependency order, using Hudi's existing SQL table options and `SET` for flow-scoped write options.
+
+### Pipeline configuration
+
+An SDP project configures the Hudi extension and catalog like any other Hudi Spark application:
+
+```yaml
+name: hudi_orders_pipeline
+libraries:
+ - glob:
+ include: transformations/**
+storage: s3://pipelines/checkpoints/hudi_orders_pipeline
+catalog: spark_catalog
+database: lakehouse
+configuration:
+ spark.sql.extensions: org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.sql.catalog.spark_catalog: org.apache.spark.sql.hudi.catalog.HoodieCatalog
+```
+
+The SQL below assumes that `order_events_source` is an external streaming source containing parsed Kafka or database CDC events. Each `CREATE ... AS SELECT` statement defines both the target dataset and its single flow.
+
+### Streaming table with an append flow
+
+```sql
+SET spark.hoodie.datasource.write.operation = insert;
+
+CREATE STREAMING TABLE bronze_order_events
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor'
+)
+AS
+SELECT *
+FROM STREAM order_events_source;
+```
+
+SDP owns the checkpoint location and trigger; Hudi commits each micro-batch.
+
+### Streaming table with keyed upserts
+
+```sql
+SET spark.hoodie.datasource.write.operation = upsert;
+
+CREATE STREAMING TABLE current_orders
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor',
+ 'primaryKey' = 'order_id',
+ 'preCombineField' = 'event_sequence'
+)
+AS
+SELECT
+ *,
+ operation = 'DELETE' AS _hoodie_is_deleted
+FROM STREAM bronze_order_events;
+```
+
+Spark append output mode only controls how each micro-batch reaches the sink; it does not make the Hudi target append-only. `HoodieStreamingSink` still runs the configured Hudi operation, which defaults to `upsert`.
+
+For an existing Hudi table, SDP loads the record key, precombine field, partition fields, and record-merger settings from the table configuration. For a new SDP table, these settings are declared as table properties and persisted by Hudi. SDP does not implement its own deduplication, ordering, merge, or delete path. It validates the schema and configuration, then passes each micro-batch to the existing Hudi writer.
+
+### Batch materialized view
+
+```sql
+SET spark.hoodie.datasource.write.operation = bulk_insert;
+
+CREATE MATERIALIZED VIEW daily_order_metrics
+USING hudi
+PARTITIONED BY (order_date)
+TBLPROPERTIES (
+ 'type' = 'cow'
+)
+AS
+SELECT
+ order_date,
+ state,
+ COUNT(*) AS order_count
+FROM current_orders
+GROUP BY order_date, state;
+```
+
+The first run creates the table and writes the result using `bulk_insert`. Later runs truncate the target, reconcile its schema and properties, and write the recomputed result. With Hudi's current truncate implementation, each refresh removes the previous data and timeline history.
+
+## Semantics
+
+### Dataset-to-Hudi operation mapping
+
+| SDP dataset/flow | Spark 4.1 execution shape | Proposed Hudi semantics |
+| ----------------------------------- | --------------------------------------------- | -------------------------------------------------------- |
+| New materialized view | Batch append to newly created table | Hudi `bulk_insert` or `insert` |
+| Existing materialized view | Truncate, alter, then batch append | Existing Hudi truncate, then `bulk_insert` or `insert` |
+| Streaming table with an append flow | Append-mode micro-batches | Hudi `insert` or `bulk_insert`, configured per flow |
+| Hudi upsert target | Append-mode micro-batches | Existing Hudi `upsert` |
+| Hudi streaming source | Structured Streaming read with SDP checkpoint | Basic Hudi latest-state incremental source |
+
+### Materialized-view write mapping
+
+Initial materialization uses Hudi `bulk_insert` or `insert`. For an existing materialized view, Spark 4.1 executes `TRUNCATE TABLE`, reconciles schema and properties through `catalog.alterTable`, then writes the recomputed result through append-mode `saveAsTable`. Hudi uses the same configured write operation after truncate. An empty result leaves the target empty.
+
+These steps are not a single transaction. Readers may see an empty target during refresh. If schema reconciliation or the batch flow fails after truncate, the previous result is not restored, and completed catalog changes remain in effect.
+
+The initial integration reuses Hudi's current full-table truncate, which deletes old data files and timeline history. This can disrupt concurrent readers and invalidate existing incremental offsets. Timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+Materialized views are supported as batch outputs, but not as incremental sources in the initial scope.
+
+### Streaming commit and restart behavior
+
+SDP manages each flow's checkpoint location and trigger. Multiple append flows can write to the same Hudi target, each with a distinct Hudi checkpoint identifier that remains stable across restarts. Concurrent flows require Hudi's existing multi-writer configuration and an appropriate lock provider. Each flow commits and retries independently.
+
+Normal restarts reuse the same Spark checkpoint and Hudi checkpoint identifier. A streaming-table full refresh starts a fresh checkpoint and uses Hudi's current full-table truncate to remove target data, timeline history, and sink checkpoint metadata before rebuilding the table. Resetting the Spark checkpoint while retaining the Hudi target is outside this RFC's scope.
+
+### Configuration ownership and precedence
+
+Configuration is split by ownership:
+
+- **Table identity:** table type, record key, precombine field, partition fields, key generator, and record-merger settings. Hudi persists these values; SDP loads them for existing tables, and a pipeline run cannot silently change them.
+- **Write behavior:** operation and writer tuning, supplied through SQL `SET spark.hoodie.*` as flow-scoped write options.
+- **Flow execution:** checkpoint location, trigger, retry, and flow identity. SDP owns these values; the Hudi integration maps each flow to a distinct sink checkpoint identifier.
+
+SDP captures the active `SET` values at each flow definition; a `SET` remains in effect for subsequent definitions until overridden. Flow settings override pipeline and Hudi defaults and must be compatible with the persisted table configuration. The integration strips the `spark.` prefix from write options and reuses `HoodieOptionConfig` to normalize table options. Flows writing to the same target must use compatible concurrency and lock settings. The sink checkpoint identifier is resolved per flow, not shared as a table property.
+
+Invalid properties should fail during SDP `dry-run` when possible, and otherwise before the first write.
+
+### Table services
+
+Table services follow the existing Hudi writer behavior, with no SDP-specific scheduling or retention defaults. Integration tests cover service shutdown, recovery, and resource cleanup when flows finish, stop, or restart, including multiple flows sharing a target.
+
+### Schema and partition evolution
+
+SDP follows Hudi's existing schema and partition evolution rules.
+
+- Hudi-supported additive schema changes may be accepted for streaming tables.
+- Dropping, renaming, or changing columns follows Hudi's configured schema evolution behavior.
+- Record-key, precombine, and partition columns must exist in the target schema.
+- Partitioning changes fail; SDP does not turn them into an implicit table replacement.
+
+`HoodieCatalog.alterTable` validates the complete set of requested schema and property changes before applying supported changes. It accepts unchanged properties and rejects unsupported changes or changes to table identity. The Hudi writer validates the actual input schema against the persisted table configuration using its existing schema reconciliation rules.
+
+For materialized views, Spark invokes `alterTable` after truncate. A schema validation failure can therefore leave the target empty; it does not preserve the previous result. If metadata changes succeed but the batch write fails, those changes remain in effect.
+
+## Implementation
+
+### 1. Catalog integration
+
+Extend and test [HoodieCatalog](../../hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala) so SDP can:
+
+- Create a table whose provider is `hudi` from SDP's table information.
+- Persist schema, location, partition transforms, comments, and Hudi table properties.
+- Load the created table for both batch and streaming execution.
+- Validate the complete alter request before applying supported property and schema changes, preserving Hudi table identity and accepting unchanged properties.
+- Reject unsupported partition transforms and incompatible property changes with actionable errors.
+
+The first version configures `HoodieCatalog` as `spark_catalog`; named catalogs are follow-up work.
+
+This path may continue to use Hudi's V1 fallback. Full DataSource V2 support is not required as long as the SDP table contract is met.
+
+### 2. Materialized-view write path
+
+Use Hudi `bulk_insert` or `insert` for the batch result. For refresh, reuse [TruncateHoodieTableCommand](../../hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/TruncateHoodieTableCommand.scala), support the subsequent catalog reconciliation, and pass the configured operation to the append-mode writer. No SDP-specific truncate mode or refresh-detection mechanism is introduced. Validate this sequence through the integration harness before declaring materialized-view refresh supported.
+
+### 3. Streaming target integration
+
+Route SDP's append-mode `toTable` call through [HoodieStreamingSink](../../hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieStreamingSink.scala), including table properties supplied by `HoodieCatalog`.
+
+The sink integration will:
+
+- Pass the flow's write operation and persisted keyed-write settings to the existing Hudi writer.
+- Preserve the SDP checkpoint location rather than creating a competing checkpoint tree.
+- Resolve a stable, per-flow `hoodie.datasource.write.streaming.checkpoint.identifier` and use it for both retry checks and commit metadata.
+- Pass through and validate Hudi's existing multi-writer and lock configuration for concurrent flows sharing a target.
+- Reuse Hudi's existing streaming-sink checkpoint and retry behavior.
Review Comment:
**🚨** "Reuse Hudi's existing streaming-sink checkpoint and retry behavior"
The existing behaviour ends in `System.exit(1)` after `streaming.retry.count` attempts (`HoodieStreamingSink.scala:210`). Under SDP that JVM is the Connect server shared by every flow and client session; in my run the server died and the `spark-pipelines` client hung indefinitely. The sink needs to throw and defer to SDP's retry (`spark.sql.pipelines.maxFlowRetryAttempts`). Filed as #20370.
##########
rfc/rfc-112/rfc-112.md:
##########
@@ -0,0 +1,429 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-112: Support Spark Declarative Pipelines
+
+## Proposers
+
+- @cshuo
+- @danny0405
+
+## Approvers
+
+- TBD
+
+## Status
+
+Issue: TBD
+
+## Abstract
+
+[Spark Declarative Pipelines (SDP)](https://spark.apache.org/docs/4.1.1/declarative-pipelines-programming-guide.html) lets users define tables and transformations while Spark builds the dependency graph and runs the batch and streaming flows. Hudi provides atomic commits, keyed updates, event ordering, deletes, incremental queries, and table services.
+
+This RFC makes Hudi a supported SDP source and target. It covers catalog-based table creation, materialized views, streaming tables, basic Hudi-to-Hudi incremental reads, and keyed upserts through Hudi's existing writer.
+
+SDP and Hudi retain their existing responsibilities:
+
+- SDP owns dataset declarations, dependency analysis, orchestration, flow checkpoints, retries, and execution order.
+- Hudi owns table storage, commit atomicity, record reconciliation, incremental read semantics, and table services.
+
+Users can build the same pipelines today with Spark jobs, Hudi writers or Hudi Streamer, and an external orchestrator. This integration removes much of that glue code while keeping the outputs as ordinary Hudi tables.
+
+## Background
+
+### Spark Declarative Pipelines
+
+Apache Spark 4.1 introduced SDP for declarative batch and streaming ETL. SDP persists two main dataset types:
+
+- A **materialized view**, which has exactly one batch flow that computes a table.
+- A **streaming table**, which has one or more streaming flows that append their output to a table.
+
+Users define pipelines in Python or SQL. SDP validates dataset references, builds the dataflow graph, runs independent flows in parallel, and stores Structured Streaming checkpoints in the pipeline storage location.
+
+In the Spark 4.1 implementation, a streaming table flow uses a Structured Streaming append-mode write to `DataStreamWriter.toTable`, and a batch table flow uses append-mode `DataFrameWriter.saveAsTable`. For an existing materialized view, SDP issues `TRUNCATE TABLE`, calls `catalog.alterTable` to reconcile schema and properties, then runs the batch flow. See Spark's [`FlowExecution`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala) and [`DatasetManager`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala).
+
+These table operations are the storage-format integration boundary; this RFC does not add a Hudi-specific execution engine.
+
+### Relevant Hudi capabilities
+
+Hudi already provides the required data-plane primitives:
+
+- `HoodieCatalog` can create and load Hudi tables through Spark's catalog APIs.
+- The Hudi batch writer supports bulk insert, insert, upsert, insert overwrite, and table insert overwrite operations. An upsert can carry delete-marker records.
+- `HoodieStreamingSink` writes Structured Streaming micro-batches and records checkpoint metadata used to avoid recommitting an already committed batch after retry.
+- Hudi's streaming source and incremental query modes can expose records changed since a prior instant.
+
+The integration exposes these capabilities through SDP's existing table and flow abstractions. Hudi Streamer, the DataFrame writer, Spark SQL DML, and direct Structured Streaming APIs remain unchanged.
+
+### Current gaps
+
+`format="hudi"` is not yet an end-to-end supported SDP contract. Some operations may already work, but table creation, refresh, restart, upsert configuration, and Hudi-to-Hudi chaining are neither documented nor covered by an integration suite.
+
+Materialized-view refresh needs end-to-end validation. SDP 4.1 truncates the target before reconciling its schema and writing the recomputed result. Hudi's current full-table `TRUNCATE TABLE` deletes the table path, including the timeline, and reinitializes its properties. This RFC reuses that behavior and documents its limitations; a timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+The integration must also address:
+
+- SDP table properties and flow-scoped write options must be resolved and passed to the Hudi writer.
+- Streaming restart must coordinate the SDP/Spark checkpoint with Hudi's sink checkpoint metadata.
+- A Hudi source used through `readStream.table` must preserve incremental offset semantics.
+- SDP must carry Hudi's persisted record-key, precombine, and record-merger settings into the existing upsert path. Deletes remain upsert input rows marked by `_hoodie_is_deleted`; Spark append output mode does not replace this contract.
+
+### Related work
+
+- Hudi's Spark 4.1 support ([issue #17663](https://github.com/apache/hudi/issues/17663), [PR #17674](https://github.com/apache/hudi/pull/17674)) provides the runtime baseline, but not an SDP contract.
+- DataSource V2 read work ([RFC-98](../rfc-98/rfc-98.md), [issue #15023](https://github.com/apache/hudi/issues/15023)) may improve the catalog read path but is not required for the initial integration.
+- The [SQL insert-overwrite discussion](https://github.com/apache/hudi/discussions/13815) shows why refresh semantics must be explicit rather than inferred from an append call.
+- [Issue #13973](https://github.com/apache/hudi/issues/13973) shows that a latest-state incremental read cannot fully propagate hard deletes. Flows that require complete delete propagation can use Hudi's existing CDC incremental format.
+- [Issue #5537](https://github.com/apache/hudi/issues/5537) tracks named-catalog support. The first version uses `HoodieCatalog` as `spark_catalog`.
+
+## Motivation
+
+### Representative use case
+
+A typical order pipeline looks like this:
+
+```text
+MySQL/PostgreSQL CDC or Kafka
+ |
+ v
+ bronze_order_events (Hudi)
+ |
+ v
+ current_orders (Hudi)
+ |
+ v
+ daily_order_metrics (Hudi)
+```
+
+The bronze table stores incoming events. The current-state table uses a record key and precombine field to reconcile duplicates and out-of-order events, including delete-marker records. Downstream streaming tables consume changed records incrementally, while batch aggregates can be materialized as views.
+
+Today this usually means several Spark applications plus custom handling for table creation, write options, checkpoints, dependencies, and restarts. With SDP, the graph lives in one project and the persisted datasets remain regular Hudi tables that other engines can query.
+
+Related user request issues include:
+
+- #19281: [multi-dataset incremental processing](https://github.com/apache/hudi/issues/19281)
+- #5189: [chained incremental Hudi tables](https://github.com/apache/hudi/issues/5189)
+- #13973: [downstream delete propagation](https://github.com/apache/hudi/issues/13973).
+
+### User value
+
+1. **Declarative pipelines.** Users define datasets as SQL or DataFrame transformations; SDP manages dependencies, checkpoints, and Hudi writes.
+2. **Managed incremental composition.** Hudi already provides incremental reads; SDP composes them into a dependency-aware pipeline, managing execution order, checkpoints, and recovery without separate Spark jobs or hand-written orchestration.
+3. **Open outputs.** Each dataset remains a standard Hudi table that can be queried independently of SDP by any Hudi-compatible engine.
+
+SDP does not make arbitrary joins or aggregates incremental; performance benefits apply only to flows that can process changes incrementally.
+
+## Goals
+
+1. Create Hudi tables from SDP datasets through `HoodieCatalog`.
+2. Write materialized views with Hudi `bulk_insert` or `insert`, following SDP's truncate-and-recompute refresh sequence.
+3. Write SDP streaming tables through Hudi's Structured Streaming sink.
+4. Support a basic incremental Hudi-to-Hudi streaming-table chain.
+5. Expose Hudi's existing keyed upsert path through SDP, including record-key, precombine, record-merger, and delete-marker handling.
+6. Keep non-SDP Hudi behavior unchanged.
+
+## Non-Goals
+
+The first version excludes:
+
+- A complete Hudi DataSource V2 read or streaming-write implementation.
+- Atomic materialized-view refresh across truncate, schema reconciliation, and the batch write.
+- Changes to Hudi's full-table truncate semantics. Timeline-preserving truncate can be developed independently.
+- Cross-table transactions across multiple SDP targets.
+- End-to-end exactly-once delivery across an arbitrary external source and Hudi. Guarantees remain scoped to Spark/Hudi micro-batch retries at one target.
+- Arbitrary named catalogs for `HoodieCatalog`; the first version uses Hudi as `spark_catalog`.
+
+## Proposed User Experience
+
+The examples below implement the representative order pipeline in dependency order, using Hudi's existing SQL table options and `SET` for flow-scoped write options.
+
+### Pipeline configuration
+
+An SDP project configures the Hudi extension and catalog like any other Hudi Spark application:
+
+```yaml
+name: hudi_orders_pipeline
+libraries:
+ - glob:
+ include: transformations/**
+storage: s3://pipelines/checkpoints/hudi_orders_pipeline
+catalog: spark_catalog
+database: lakehouse
+configuration:
+ spark.sql.extensions: org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.sql.catalog.spark_catalog: org.apache.spark.sql.hudi.catalog.HoodieCatalog
+```
+
+The SQL below assumes that `order_events_source` is an external streaming source containing parsed Kafka or database CDC events. Each `CREATE ... AS SELECT` statement defines both the target dataset and its single flow.
+
+### Streaming table with an append flow
+
+```sql
+SET spark.hoodie.datasource.write.operation = insert;
+
+CREATE STREAMING TABLE bronze_order_events
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor'
+)
+AS
+SELECT *
+FROM STREAM order_events_source;
+```
+
+SDP owns the checkpoint location and trigger; Hudi commits each micro-batch.
+
+### Streaming table with keyed upserts
+
+```sql
+SET spark.hoodie.datasource.write.operation = upsert;
+
+CREATE STREAMING TABLE current_orders
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor',
+ 'primaryKey' = 'order_id',
+ 'preCombineField' = 'event_sequence'
+)
+AS
+SELECT
+ *,
+ operation = 'DELETE' AS _hoodie_is_deleted
+FROM STREAM bronze_order_events;
+```
+
+Spark append output mode only controls how each micro-batch reaches the sink; it does not make the Hudi target append-only. `HoodieStreamingSink` still runs the configured Hudi operation, which defaults to `upsert`.
+
+For an existing Hudi table, SDP loads the record key, precombine field, partition fields, and record-merger settings from the table configuration. For a new SDP table, these settings are declared as table properties and persisted by Hudi. SDP does not implement its own deduplication, ordering, merge, or delete path. It validates the schema and configuration, then passes each micro-batch to the existing Hudi writer.
+
+### Batch materialized view
+
+```sql
+SET spark.hoodie.datasource.write.operation = bulk_insert;
+
+CREATE MATERIALIZED VIEW daily_order_metrics
+USING hudi
+PARTITIONED BY (order_date)
+TBLPROPERTIES (
+ 'type' = 'cow'
+)
+AS
+SELECT
+ order_date,
+ state,
+ COUNT(*) AS order_count
+FROM current_orders
+GROUP BY order_date, state;
+```
+
+The first run creates the table and writes the result using `bulk_insert`. Later runs truncate the target, reconcile its schema and properties, and write the recomputed result. With Hudi's current truncate implementation, each refresh removes the previous data and timeline history.
+
+## Semantics
+
+### Dataset-to-Hudi operation mapping
+
+| SDP dataset/flow | Spark 4.1 execution shape | Proposed Hudi semantics |
+| ----------------------------------- | --------------------------------------------- | -------------------------------------------------------- |
+| New materialized view | Batch append to newly created table | Hudi `bulk_insert` or `insert` |
+| Existing materialized view | Truncate, alter, then batch append | Existing Hudi truncate, then `bulk_insert` or `insert` |
+| Streaming table with an append flow | Append-mode micro-batches | Hudi `insert` or `bulk_insert`, configured per flow |
+| Hudi upsert target | Append-mode micro-batches | Existing Hudi `upsert` |
+| Hudi streaming source | Structured Streaming read with SDP checkpoint | Basic Hudi latest-state incremental source |
+
+### Materialized-view write mapping
+
+Initial materialization uses Hudi `bulk_insert` or `insert`. For an existing materialized view, Spark 4.1 executes `TRUNCATE TABLE`, reconciles schema and properties through `catalog.alterTable`, then writes the recomputed result through append-mode `saveAsTable`. Hudi uses the same configured write operation after truncate. An empty result leaves the target empty.
+
+These steps are not a single transaction. Readers may see an empty target during refresh. If schema reconciliation or the batch flow fails after truncate, the previous result is not restored, and completed catalog changes remain in effect.
+
+The initial integration reuses Hudi's current full-table truncate, which deletes old data files and timeline history. This can disrupt concurrent readers and invalidate existing incremental offsets. Timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+Materialized views are supported as batch outputs, but not as incremental sources in the initial scope.
+
+### Streaming commit and restart behavior
+
+SDP manages each flow's checkpoint location and trigger. Multiple append flows can write to the same Hudi target, each with a distinct Hudi checkpoint identifier that remains stable across restarts. Concurrent flows require Hudi's existing multi-writer configuration and an appropriate lock provider. Each flow commits and retries independently.
+
+Normal restarts reuse the same Spark checkpoint and Hudi checkpoint identifier. A streaming-table full refresh starts a fresh checkpoint and uses Hudi's current full-table truncate to remove target data, timeline history, and sink checkpoint metadata before rebuilding the table. Resetting the Spark checkpoint while retaining the Hudi target is outside this RFC's scope.
+
+### Configuration ownership and precedence
+
+Configuration is split by ownership:
+
+- **Table identity:** table type, record key, precombine field, partition fields, key generator, and record-merger settings. Hudi persists these values; SDP loads them for existing tables, and a pipeline run cannot silently change them.
+- **Write behavior:** operation and writer tuning, supplied through SQL `SET spark.hoodie.*` as flow-scoped write options.
+- **Flow execution:** checkpoint location, trigger, retry, and flow identity. SDP owns these values; the Hudi integration maps each flow to a distinct sink checkpoint identifier.
+
+SDP captures the active `SET` values at each flow definition; a `SET` remains in effect for subsequent definitions until overridden. Flow settings override pipeline and Hudi defaults and must be compatible with the persisted table configuration. The integration strips the `spark.` prefix from write options and reuses `HoodieOptionConfig` to normalize table options. Flows writing to the same target must use compatible concurrency and lock settings. The sink checkpoint identifier is resolved per flow, not shared as a table property.
+
+Invalid properties should fail during SDP `dry-run` when possible, and otherwise before the first write.
+
+### Table services
+
+Table services follow the existing Hudi writer behavior, with no SDP-specific scheduling or retention defaults. Integration tests cover service shutdown, recovery, and resource cleanup when flows finish, stop, or restart, including multiple flows sharing a target.
+
+### Schema and partition evolution
+
+SDP follows Hudi's existing schema and partition evolution rules.
+
+- Hudi-supported additive schema changes may be accepted for streaming tables.
+- Dropping, renaming, or changing columns follows Hudi's configured schema evolution behavior.
+- Record-key, precombine, and partition columns must exist in the target schema.
+- Partitioning changes fail; SDP does not turn them into an implicit table replacement.
+
+`HoodieCatalog.alterTable` validates the complete set of requested schema and property changes before applying supported changes. It accepts unchanged properties and rejects unsupported changes or changes to table identity. The Hudi writer validates the actual input schema against the persisted table configuration using its existing schema reconciliation rules.
+
+For materialized views, Spark invokes `alterTable` after truncate. A schema validation failure can therefore leave the target empty; it does not preserve the previous result. If metadata changes succeed but the batch write fails, those changes remain in effect.
+
+## Implementation
+
+### 1. Catalog integration
+
+Extend and test [HoodieCatalog](../../hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala) so SDP can:
+
+- Create a table whose provider is `hudi` from SDP's table information.
Review Comment:
**🚨** "Create a table whose provider is hudi from SDP's table information"
SDP calls `TableCatalog.createTable(ident, TableInfo)`; the default implementation routes to the `Column[]` overload, which `DelegatingCatalogExtension` forwards to the session catalog. `HoodieCatalog` overrides only the `StructType` overload (`HoodieCatalog.scala:173`). Observed: catalog entry with `provider=hudi`, no `hoodie.properties`, and the sink then fails with `'hoodie.table.name' must be set`. One-line fix (override the `Column[]` overload), but Phase 1 cannot pass without it, so worth naming here. (Tracked in #20371.)
##########
rfc/rfc-112/rfc-112.md:
##########
@@ -0,0 +1,429 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-112: Support Spark Declarative Pipelines
+
+## Proposers
+
+- @cshuo
+- @danny0405
+
+## Approvers
+
+- TBD
+
+## Status
+
+Issue: TBD
+
+## Abstract
+
+[Spark Declarative Pipelines (SDP)](https://spark.apache.org/docs/4.1.1/declarative-pipelines-programming-guide.html) lets users define tables and transformations while Spark builds the dependency graph and runs the batch and streaming flows. Hudi provides atomic commits, keyed updates, event ordering, deletes, incremental queries, and table services.
+
+This RFC makes Hudi a supported SDP source and target. It covers catalog-based table creation, materialized views, streaming tables, basic Hudi-to-Hudi incremental reads, and keyed upserts through Hudi's existing writer.
+
+SDP and Hudi retain their existing responsibilities:
+
+- SDP owns dataset declarations, dependency analysis, orchestration, flow checkpoints, retries, and execution order.
+- Hudi owns table storage, commit atomicity, record reconciliation, incremental read semantics, and table services.
+
+Users can build the same pipelines today with Spark jobs, Hudi writers or Hudi Streamer, and an external orchestrator. This integration removes much of that glue code while keeping the outputs as ordinary Hudi tables.
+
+## Background
+
+### Spark Declarative Pipelines
+
+Apache Spark 4.1 introduced SDP for declarative batch and streaming ETL. SDP persists two main dataset types:
+
+- A **materialized view**, which has exactly one batch flow that computes a table.
+- A **streaming table**, which has one or more streaming flows that append their output to a table.
+
+Users define pipelines in Python or SQL. SDP validates dataset references, builds the dataflow graph, runs independent flows in parallel, and stores Structured Streaming checkpoints in the pipeline storage location.
+
+In the Spark 4.1 implementation, a streaming table flow uses a Structured Streaming append-mode write to `DataStreamWriter.toTable`, and a batch table flow uses append-mode `DataFrameWriter.saveAsTable`. For an existing materialized view, SDP issues `TRUNCATE TABLE`, calls `catalog.alterTable` to reconcile schema and properties, then runs the batch flow. See Spark's [`FlowExecution`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala) and [`DatasetManager`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala).
+
+These table operations are the storage-format integration boundary; this RFC does not add a Hudi-specific execution engine.
+
+### Relevant Hudi capabilities
+
+Hudi already provides the required data-plane primitives:
+
+- `HoodieCatalog` can create and load Hudi tables through Spark's catalog APIs.
+- The Hudi batch writer supports bulk insert, insert, upsert, insert overwrite, and table insert overwrite operations. An upsert can carry delete-marker records.
+- `HoodieStreamingSink` writes Structured Streaming micro-batches and records checkpoint metadata used to avoid recommitting an already committed batch after retry.
+- Hudi's streaming source and incremental query modes can expose records changed since a prior instant.
+
+The integration exposes these capabilities through SDP's existing table and flow abstractions. Hudi Streamer, the DataFrame writer, Spark SQL DML, and direct Structured Streaming APIs remain unchanged.
+
+### Current gaps
+
+`format="hudi"` is not yet an end-to-end supported SDP contract. Some operations may already work, but table creation, refresh, restart, upsert configuration, and Hudi-to-Hudi chaining are neither documented nor covered by an integration suite.
+
+Materialized-view refresh needs end-to-end validation. SDP 4.1 truncates the target before reconciling its schema and writing the recomputed result. Hudi's current full-table `TRUNCATE TABLE` deletes the table path, including the timeline, and reinitializes its properties. This RFC reuses that behavior and documents its limitations; a timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+The integration must also address:
+
+- SDP table properties and flow-scoped write options must be resolved and passed to the Hudi writer.
+- Streaming restart must coordinate the SDP/Spark checkpoint with Hudi's sink checkpoint metadata.
+- A Hudi source used through `readStream.table` must preserve incremental offset semantics.
+- SDP must carry Hudi's persisted record-key, precombine, and record-merger settings into the existing upsert path. Deletes remain upsert input rows marked by `_hoodie_is_deleted`; Spark append output mode does not replace this contract.
+
+### Related work
+
+- Hudi's Spark 4.1 support ([issue #17663](https://github.com/apache/hudi/issues/17663), [PR #17674](https://github.com/apache/hudi/pull/17674)) provides the runtime baseline, but not an SDP contract.
+- DataSource V2 read work ([RFC-98](../rfc-98/rfc-98.md), [issue #15023](https://github.com/apache/hudi/issues/15023)) may improve the catalog read path but is not required for the initial integration.
+- The [SQL insert-overwrite discussion](https://github.com/apache/hudi/discussions/13815) shows why refresh semantics must be explicit rather than inferred from an append call.
+- [Issue #13973](https://github.com/apache/hudi/issues/13973) shows that a latest-state incremental read cannot fully propagate hard deletes. Flows that require complete delete propagation can use Hudi's existing CDC incremental format.
+- [Issue #5537](https://github.com/apache/hudi/issues/5537) tracks named-catalog support. The first version uses `HoodieCatalog` as `spark_catalog`.
+
+## Motivation
+
+### Representative use case
+
+A typical order pipeline looks like this:
+
+```text
+MySQL/PostgreSQL CDC or Kafka
+ |
+ v
+ bronze_order_events (Hudi)
+ |
+ v
+ current_orders (Hudi)
+ |
+ v
+ daily_order_metrics (Hudi)
+```
+
+The bronze table stores incoming events. The current-state table uses a record key and precombine field to reconcile duplicates and out-of-order events, including delete-marker records. Downstream streaming tables consume changed records incrementally, while batch aggregates can be materialized as views.
+
+Today this usually means several Spark applications plus custom handling for table creation, write options, checkpoints, dependencies, and restarts. With SDP, the graph lives in one project and the persisted datasets remain regular Hudi tables that other engines can query.
+
+Related user request issues include:
+
+- #19281: [multi-dataset incremental processing](https://github.com/apache/hudi/issues/19281)
+- #5189: [chained incremental Hudi tables](https://github.com/apache/hudi/issues/5189)
+- #13973: [downstream delete propagation](https://github.com/apache/hudi/issues/13973).
+
+### User value
+
+1. **Declarative pipelines.** Users define datasets as SQL or DataFrame transformations; SDP manages dependencies, checkpoints, and Hudi writes.
+2. **Managed incremental composition.** Hudi already provides incremental reads; SDP composes them into a dependency-aware pipeline, managing execution order, checkpoints, and recovery without separate Spark jobs or hand-written orchestration.
+3. **Open outputs.** Each dataset remains a standard Hudi table that can be queried independently of SDP by any Hudi-compatible engine.
+
+SDP does not make arbitrary joins or aggregates incremental; performance benefits apply only to flows that can process changes incrementally.
+
+## Goals
+
+1. Create Hudi tables from SDP datasets through `HoodieCatalog`.
+2. Write materialized views with Hudi `bulk_insert` or `insert`, following SDP's truncate-and-recompute refresh sequence.
+3. Write SDP streaming tables through Hudi's Structured Streaming sink.
+4. Support a basic incremental Hudi-to-Hudi streaming-table chain.
+5. Expose Hudi's existing keyed upsert path through SDP, including record-key, precombine, record-merger, and delete-marker handling.
+6. Keep non-SDP Hudi behavior unchanged.
+
+## Non-Goals
+
+The first version excludes:
+
+- A complete Hudi DataSource V2 read or streaming-write implementation.
+- Atomic materialized-view refresh across truncate, schema reconciliation, and the batch write.
+- Changes to Hudi's full-table truncate semantics. Timeline-preserving truncate can be developed independently.
+- Cross-table transactions across multiple SDP targets.
+- End-to-end exactly-once delivery across an arbitrary external source and Hudi. Guarantees remain scoped to Spark/Hudi micro-batch retries at one target.
+- Arbitrary named catalogs for `HoodieCatalog`; the first version uses Hudi as `spark_catalog`.
+
+## Proposed User Experience
+
+The examples below implement the representative order pipeline in dependency order, using Hudi's existing SQL table options and `SET` for flow-scoped write options.
+
+### Pipeline configuration
+
+An SDP project configures the Hudi extension and catalog like any other Hudi Spark application:
+
+```yaml
+name: hudi_orders_pipeline
+libraries:
+ - glob:
+ include: transformations/**
+storage: s3://pipelines/checkpoints/hudi_orders_pipeline
+catalog: spark_catalog
+database: lakehouse
+configuration:
+ spark.sql.extensions: org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.sql.catalog.spark_catalog: org.apache.spark.sql.hudi.catalog.HoodieCatalog
+```
+
+The SQL below assumes that `order_events_source` is an external streaming source containing parsed Kafka or database CDC events. Each `CREATE ... AS SELECT` statement defines both the target dataset and its single flow.
+
+### Streaming table with an append flow
+
+```sql
+SET spark.hoodie.datasource.write.operation = insert;
+
+CREATE STREAMING TABLE bronze_order_events
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor'
+)
+AS
+SELECT *
+FROM STREAM order_events_source;
+```
+
+SDP owns the checkpoint location and trigger; Hudi commits each micro-batch.
+
+### Streaming table with keyed upserts
+
+```sql
+SET spark.hoodie.datasource.write.operation = upsert;
+
+CREATE STREAMING TABLE current_orders
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor',
+ 'primaryKey' = 'order_id',
+ 'preCombineField' = 'event_sequence'
+)
+AS
+SELECT
+ *,
+ operation = 'DELETE' AS _hoodie_is_deleted
+FROM STREAM bronze_order_events;
+```
+
+Spark append output mode only controls how each micro-batch reaches the sink; it does not make the Hudi target append-only. `HoodieStreamingSink` still runs the configured Hudi operation, which defaults to `upsert`.
+
+For an existing Hudi table, SDP loads the record key, precombine field, partition fields, and record-merger settings from the table configuration. For a new SDP table, these settings are declared as table properties and persisted by Hudi. SDP does not implement its own deduplication, ordering, merge, or delete path. It validates the schema and configuration, then passes each micro-batch to the existing Hudi writer.
+
+### Batch materialized view
+
+```sql
+SET spark.hoodie.datasource.write.operation = bulk_insert;
+
+CREATE MATERIALIZED VIEW daily_order_metrics
+USING hudi
+PARTITIONED BY (order_date)
+TBLPROPERTIES (
+ 'type' = 'cow'
+)
+AS
+SELECT
+ order_date,
+ state,
+ COUNT(*) AS order_count
+FROM current_orders
+GROUP BY order_date, state;
+```
+
+The first run creates the table and writes the result using `bulk_insert`. Later runs truncate the target, reconcile its schema and properties, and write the recomputed result. With Hudi's current truncate implementation, each refresh removes the previous data and timeline history.
+
+## Semantics
+
+### Dataset-to-Hudi operation mapping
+
+| SDP dataset/flow | Spark 4.1 execution shape | Proposed Hudi semantics |
+| ----------------------------------- | --------------------------------------------- | -------------------------------------------------------- |
+| New materialized view | Batch append to newly created table | Hudi `bulk_insert` or `insert` |
+| Existing materialized view | Truncate, alter, then batch append | Existing Hudi truncate, then `bulk_insert` or `insert` |
+| Streaming table with an append flow | Append-mode micro-batches | Hudi `insert` or `bulk_insert`, configured per flow |
+| Hudi upsert target | Append-mode micro-batches | Existing Hudi `upsert` |
+| Hudi streaming source | Structured Streaming read with SDP checkpoint | Basic Hudi latest-state incremental source |
+
+### Materialized-view write mapping
+
+Initial materialization uses Hudi `bulk_insert` or `insert`. For an existing materialized view, Spark 4.1 executes `TRUNCATE TABLE`, reconciles schema and properties through `catalog.alterTable`, then writes the recomputed result through append-mode `saveAsTable`. Hudi uses the same configured write operation after truncate. An empty result leaves the target empty.
+
+These steps are not a single transaction. Readers may see an empty target during refresh. If schema reconciliation or the batch flow fails after truncate, the previous result is not restored, and completed catalog changes remain in effect.
+
+The initial integration reuses Hudi's current full-table truncate, which deletes old data files and timeline history. This can disrupt concurrent readers and invalidate existing incremental offsets. Timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+Materialized views are supported as batch outputs, but not as incremental sources in the initial scope.
+
+### Streaming commit and restart behavior
+
+SDP manages each flow's checkpoint location and trigger. Multiple append flows can write to the same Hudi target, each with a distinct Hudi checkpoint identifier that remains stable across restarts. Concurrent flows require Hudi's existing multi-writer configuration and an appropriate lock provider. Each flow commits and retries independently.
+
+Normal restarts reuse the same Spark checkpoint and Hudi checkpoint identifier. A streaming-table full refresh starts a fresh checkpoint and uses Hudi's current full-table truncate to remove target data, timeline history, and sink checkpoint metadata before rebuilding the table. Resetting the Spark checkpoint while retaining the Hudi target is outside this RFC's scope.
+
+### Configuration ownership and precedence
+
+Configuration is split by ownership:
+
+- **Table identity:** table type, record key, precombine field, partition fields, key generator, and record-merger settings. Hudi persists these values; SDP loads them for existing tables, and a pipeline run cannot silently change them.
+- **Write behavior:** operation and writer tuning, supplied through SQL `SET spark.hoodie.*` as flow-scoped write options.
+- **Flow execution:** checkpoint location, trigger, retry, and flow identity. SDP owns these values; the Hudi integration maps each flow to a distinct sink checkpoint identifier.
+
+SDP captures the active `SET` values at each flow definition; a `SET` remains in effect for subsequent definitions until overridden. Flow settings override pipeline and Hudi defaults and must be compatible with the persisted table configuration. The integration strips the `spark.` prefix from write options and reuses `HoodieOptionConfig` to normalize table options. Flows writing to the same target must use compatible concurrency and lock settings. The sink checkpoint identifier is resolved per flow, not shared as a table property.
+
+Invalid properties should fail during SDP `dry-run` when possible, and otherwise before the first write.
+
+### Table services
+
+Table services follow the existing Hudi writer behavior, with no SDP-specific scheduling or retention defaults. Integration tests cover service shutdown, recovery, and resource cleanup when flows finish, stop, or restart, including multiple flows sharing a target.
+
+### Schema and partition evolution
+
+SDP follows Hudi's existing schema and partition evolution rules.
+
+- Hudi-supported additive schema changes may be accepted for streaming tables.
+- Dropping, renaming, or changing columns follows Hudi's configured schema evolution behavior.
+- Record-key, precombine, and partition columns must exist in the target schema.
+- Partitioning changes fail; SDP does not turn them into an implicit table replacement.
+
+`HoodieCatalog.alterTable` validates the complete set of requested schema and property changes before applying supported changes. It accepts unchanged properties and rejects unsupported changes or changes to table identity. The Hudi writer validates the actual input schema against the persisted table configuration using its existing schema reconciliation rules.
+
+For materialized views, Spark invokes `alterTable` after truncate. A schema validation failure can therefore leave the target empty; it does not preserve the previous result. If metadata changes succeed but the batch write fails, those changes remain in effect.
+
+## Implementation
+
+### 1. Catalog integration
+
+Extend and test [HoodieCatalog](../../hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala) so SDP can:
+
+- Create a table whose provider is `hudi` from SDP's table information.
+- Persist schema, location, partition transforms, comments, and Hudi table properties.
+- Load the created table for both batch and streaming execution.
+- Validate the complete alter request before applying supported property and schema changes, preserving Hudi table identity and accepting unchanged properties.
+- Reject unsupported partition transforms and incompatible property changes with actionable errors.
+
+The first version configures `HoodieCatalog` as `spark_catalog`; named catalogs are follow-up work.
+
+This path may continue to use Hudi's V1 fallback. Full DataSource V2 support is not required as long as the SDP table contract is met.
+
+### 2. Materialized-view write path
+
+Use Hudi `bulk_insert` or `insert` for the batch result. For refresh, reuse [TruncateHoodieTableCommand](../../hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/TruncateHoodieTableCommand.scala), support the subsequent catalog reconciliation, and pass the configured operation to the append-mode writer. No SDP-specific truncate mode or refresh-detection mechanism is introduced. Validate this sequence through the integration harness before declaring materialized-view refresh supported.
+
+### 3. Streaming target integration
+
+Route SDP's append-mode `toTable` call through [HoodieStreamingSink](../../hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieStreamingSink.scala), including table properties supplied by `HoodieCatalog`.
+
+The sink integration will:
+
+- Pass the flow's write operation and persisted keyed-write settings to the existing Hudi writer.
+- Preserve the SDP checkpoint location rather than creating a competing checkpoint tree.
+- Resolve a stable, per-flow `hoodie.datasource.write.streaming.checkpoint.identifier` and use it for both retry checks and commit metadata.
+- Pass through and validate Hudi's existing multi-writer and lock configuration for concurrent flows sharing a target.
+- Reuse Hudi's existing streaming-sink checkpoint and retry behavior.
+
+No SDP-specific upsert writer, merge logic, or delete path is introduced.
+
+### 4. Incremental source integration
+
+Validate both catalog-table and explicit Hudi sources in SDP streaming flows. Source progress is restored from the SDP-managed checkpoint and advances with the downstream flow.
+
+Document:
+
+- Initial position behavior (`earliest`, `latest`, or an explicit instant).
+- The basic `latest-state` incremental output contract.
+- Behavior when required source instants have been archived or cleaned.
+- COW and MOR differences, if any, in supported streaming query modes.
+- Metadata-column handling when one Hudi table feeds another.
+
+If the checkpoint refers to source history that is no longer available, fail with a recovery error rather than skipping data.
+
+### 5. Upsert configuration and validation
+
+For an upsert target, validate that:
+
+- At least one record-key field is configured and present.
+- All partition fields are present.
+- Configured precombine fields are present and type-compatible.
+- `_hoodie_is_deleted`, when present, is boolean.
+
+These checks only validate the input contract. Hudi's existing writer, precombine, and record-merger logic perform the upsert.
+
+## Failure and Consistency Model
+
+| Failure point | Required outcome |
+| ------------------------------------------------------ | ------------------------------------------------------------------------------ |
+| Validation fails before truncate starts | Existing target remains unchanged |
+| Full-table truncate fails | Files may already have been removed; no automatic restoration is guaranteed |
+| Schema reconciliation or MV computation fails after truncate | Previous result is not restored; completed metadata changes remain in effect |
+| MV batch write fails before commit | No completed commit for that write; prior data removed by truncate is not restored |
+| Streaming task fails before Hudi commit | Spark retries the micro-batch |
+| Streaming flow retries or restarts with its existing checkpoint | Existing Hudi streaming-sink checkpoint behavior is preserved |
+| Source checkpoint points to expired Hudi history | Flow fails with an explicit recovery error |
+| Pipeline definition changes record key or partitioning | Validation fails; no implicit migration occurs, but an earlier truncate is not undone |
+
+Hudi writes retain per-commit atomicity. The current full-table truncate is a filesystem operation, not a Hudi commit, and the refresh sequence has no transaction or rollback spanning truncate, catalog changes, and the batch write. Commits from independent SDP flows are not atomic as a group.
+
+## Rollout/Adoption Plan
+
+### Phase 1: Basic catalog target
+
+- Support Spark 4.1.x with `HoodieCatalog` configured as `spark_catalog`.
+- Materialize a new, trivial SDP dataset as a Hudi-readable table.
+- Add the SDP integration harness and documentation.
+
+### Phase 2: Materialized-view refresh
+
+- Write the initial materialized-view result through Hudi `bulk_insert` or `insert`.
+- Refresh existing targets through Hudi's current truncate, catalog reconciliation, and batch write.
+- Validate repeated runs and document the history, reader, and failure limitations.
+
+### Phase 3: Streaming tables and chaining
+
+- Support one or more append streaming flows writing to the same target through the Hudi streaming sink.
+- Validate source-to-Hudi streaming-table writes.
+- Validate an upstream Hudi streaming table feeding a downstream streaming table through a basic incremental read.
+
+### Phase 4: Upsert configuration and verification
+
+- Verify that record-key, precombine, partition, and record-merger settings reach the existing Hudi writer.
+- Verify that `_hoodie_is_deleted` records use the same upsert path.
+- Add functional coverage for the resulting Hudi table state.
+
+Each phase is independently testable. Existing Hudi jobs are unaffected; the integration is used only by SDP projects that declare Hudi-backed datasets.
+
+## Test Plan
+
+The integration suite covers the four milestones:
+
+1. A trivial SDP pipeline materializes a table that can be read as Hudi.
+2. A materialized-view flow supports initial writes and repeated refreshes, including schema validation, empty results, and the documented failure behavior.
+3. Append streaming flows support source-to-Hudi writes, multiple flows sharing a target, and basic incremental Hudi-to-Hudi chaining, with isolated write options and correct results across flow restarts.
+4. A keyed flow preserves Hudi's existing upsert semantics, including delete-marker records, and produces the expected table state.
+
+Existing DataFrame, Spark SQL, and Structured Streaming tests provide regression coverage.
+
+## Alternatives Considered
+
+### Require timeline-preserving truncate before SDP support
+
+A logical truncate implemented through a replace commit would preserve timeline history and benefit Hudi SQL users as well as SDP. It can be developed and adopted independently; the SDP integration does not depend on it. Retaining the timeline would not make the truncate–alter–append sequence atomic.
+
+### Treat truncate as a no-op and overwrite on append
+
+Not adopted. This requires distinguishing SDP refresh from ordinary SQL truncate, handling empty results in the overwrite path, and coordinating separately applied catalog changes. The initial integration keeps the existing truncate behavior instead of introducing a refresh-specific protocol. A DataSource V2 writer's `truncate()` hook does not by itself replace SDP's preceding SQL `TRUNCATE TABLE` call.
+
+### Wait for complete DataSource V2 support
Review Comment:
**đź’¬** The DSv2 dependency that matters
Agree that DSv2 reads/writes are not needed for this scope. One dependency worth naming for the roadmap: Spark master's `create_auto_cdc_flow` (SCD1/2) runs the DataFrame `mergeInto(...).merge()` API in `foreachBatch`, which needs `SupportsRowLevelOperations` on the target; Hudi's SQL-rule `MergeIntoHoodieTableCommand` does not serve that API. That is the piece of DSv2 work the next SDP feature will actually block on.
##########
rfc/rfc-112/rfc-112.md:
##########
@@ -0,0 +1,429 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-112: Support Spark Declarative Pipelines
+
+## Proposers
+
+- @cshuo
+- @danny0405
+
+## Approvers
+
+- TBD
+
+## Status
+
+Issue: TBD
+
+## Abstract
+
+[Spark Declarative Pipelines (SDP)](https://spark.apache.org/docs/4.1.1/declarative-pipelines-programming-guide.html) lets users define tables and transformations while Spark builds the dependency graph and runs the batch and streaming flows. Hudi provides atomic commits, keyed updates, event ordering, deletes, incremental queries, and table services.
+
+This RFC makes Hudi a supported SDP source and target. It covers catalog-based table creation, materialized views, streaming tables, basic Hudi-to-Hudi incremental reads, and keyed upserts through Hudi's existing writer.
+
+SDP and Hudi retain their existing responsibilities:
+
+- SDP owns dataset declarations, dependency analysis, orchestration, flow checkpoints, retries, and execution order.
+- Hudi owns table storage, commit atomicity, record reconciliation, incremental read semantics, and table services.
+
+Users can build the same pipelines today with Spark jobs, Hudi writers or Hudi Streamer, and an external orchestrator. This integration removes much of that glue code while keeping the outputs as ordinary Hudi tables.
+
+## Background
+
+### Spark Declarative Pipelines
+
+Apache Spark 4.1 introduced SDP for declarative batch and streaming ETL. SDP persists two main dataset types:
+
+- A **materialized view**, which has exactly one batch flow that computes a table.
+- A **streaming table**, which has one or more streaming flows that append their output to a table.
+
+Users define pipelines in Python or SQL. SDP validates dataset references, builds the dataflow graph, runs independent flows in parallel, and stores Structured Streaming checkpoints in the pipeline storage location.
+
+In the Spark 4.1 implementation, a streaming table flow uses a Structured Streaming append-mode write to `DataStreamWriter.toTable`, and a batch table flow uses append-mode `DataFrameWriter.saveAsTable`. For an existing materialized view, SDP issues `TRUNCATE TABLE`, calls `catalog.alterTable` to reconcile schema and properties, then runs the batch flow. See Spark's [`FlowExecution`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala) and [`DatasetManager`](https://github.com/apache/spark/blob/branch-4.1/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala).
+
+These table operations are the storage-format integration boundary; this RFC does not add a Hudi-specific execution engine.
+
+### Relevant Hudi capabilities
+
+Hudi already provides the required data-plane primitives:
+
+- `HoodieCatalog` can create and load Hudi tables through Spark's catalog APIs.
+- The Hudi batch writer supports bulk insert, insert, upsert, insert overwrite, and table insert overwrite operations. An upsert can carry delete-marker records.
+- `HoodieStreamingSink` writes Structured Streaming micro-batches and records checkpoint metadata used to avoid recommitting an already committed batch after retry.
+- Hudi's streaming source and incremental query modes can expose records changed since a prior instant.
+
+The integration exposes these capabilities through SDP's existing table and flow abstractions. Hudi Streamer, the DataFrame writer, Spark SQL DML, and direct Structured Streaming APIs remain unchanged.
+
+### Current gaps
+
+`format="hudi"` is not yet an end-to-end supported SDP contract. Some operations may already work, but table creation, refresh, restart, upsert configuration, and Hudi-to-Hudi chaining are neither documented nor covered by an integration suite.
+
+Materialized-view refresh needs end-to-end validation. SDP 4.1 truncates the target before reconciling its schema and writing the recomputed result. Hudi's current full-table `TRUNCATE TABLE` deletes the table path, including the timeline, and reinitializes its properties. This RFC reuses that behavior and documents its limitations; a timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+The integration must also address:
+
+- SDP table properties and flow-scoped write options must be resolved and passed to the Hudi writer.
+- Streaming restart must coordinate the SDP/Spark checkpoint with Hudi's sink checkpoint metadata.
+- A Hudi source used through `readStream.table` must preserve incremental offset semantics.
+- SDP must carry Hudi's persisted record-key, precombine, and record-merger settings into the existing upsert path. Deletes remain upsert input rows marked by `_hoodie_is_deleted`; Spark append output mode does not replace this contract.
+
+### Related work
+
+- Hudi's Spark 4.1 support ([issue #17663](https://github.com/apache/hudi/issues/17663), [PR #17674](https://github.com/apache/hudi/pull/17674)) provides the runtime baseline, but not an SDP contract.
+- DataSource V2 read work ([RFC-98](../rfc-98/rfc-98.md), [issue #15023](https://github.com/apache/hudi/issues/15023)) may improve the catalog read path but is not required for the initial integration.
+- The [SQL insert-overwrite discussion](https://github.com/apache/hudi/discussions/13815) shows why refresh semantics must be explicit rather than inferred from an append call.
+- [Issue #13973](https://github.com/apache/hudi/issues/13973) shows that a latest-state incremental read cannot fully propagate hard deletes. Flows that require complete delete propagation can use Hudi's existing CDC incremental format.
+- [Issue #5537](https://github.com/apache/hudi/issues/5537) tracks named-catalog support. The first version uses `HoodieCatalog` as `spark_catalog`.
+
+## Motivation
+
+### Representative use case
+
+A typical order pipeline looks like this:
+
+```text
+MySQL/PostgreSQL CDC or Kafka
+ |
+ v
+ bronze_order_events (Hudi)
+ |
+ v
+ current_orders (Hudi)
+ |
+ v
+ daily_order_metrics (Hudi)
+```
+
+The bronze table stores incoming events. The current-state table uses a record key and precombine field to reconcile duplicates and out-of-order events, including delete-marker records. Downstream streaming tables consume changed records incrementally, while batch aggregates can be materialized as views.
+
+Today this usually means several Spark applications plus custom handling for table creation, write options, checkpoints, dependencies, and restarts. With SDP, the graph lives in one project and the persisted datasets remain regular Hudi tables that other engines can query.
+
+Related user request issues include:
+
+- #19281: [multi-dataset incremental processing](https://github.com/apache/hudi/issues/19281)
+- #5189: [chained incremental Hudi tables](https://github.com/apache/hudi/issues/5189)
+- #13973: [downstream delete propagation](https://github.com/apache/hudi/issues/13973).
+
+### User value
+
+1. **Declarative pipelines.** Users define datasets as SQL or DataFrame transformations; SDP manages dependencies, checkpoints, and Hudi writes.
+2. **Managed incremental composition.** Hudi already provides incremental reads; SDP composes them into a dependency-aware pipeline, managing execution order, checkpoints, and recovery without separate Spark jobs or hand-written orchestration.
+3. **Open outputs.** Each dataset remains a standard Hudi table that can be queried independently of SDP by any Hudi-compatible engine.
+
+SDP does not make arbitrary joins or aggregates incremental; performance benefits apply only to flows that can process changes incrementally.
+
+## Goals
+
+1. Create Hudi tables from SDP datasets through `HoodieCatalog`.
+2. Write materialized views with Hudi `bulk_insert` or `insert`, following SDP's truncate-and-recompute refresh sequence.
+3. Write SDP streaming tables through Hudi's Structured Streaming sink.
+4. Support a basic incremental Hudi-to-Hudi streaming-table chain.
+5. Expose Hudi's existing keyed upsert path through SDP, including record-key, precombine, record-merger, and delete-marker handling.
+6. Keep non-SDP Hudi behavior unchanged.
+
+## Non-Goals
+
+The first version excludes:
+
+- A complete Hudi DataSource V2 read or streaming-write implementation.
+- Atomic materialized-view refresh across truncate, schema reconciliation, and the batch write.
+- Changes to Hudi's full-table truncate semantics. Timeline-preserving truncate can be developed independently.
+- Cross-table transactions across multiple SDP targets.
+- End-to-end exactly-once delivery across an arbitrary external source and Hudi. Guarantees remain scoped to Spark/Hudi micro-batch retries at one target.
+- Arbitrary named catalogs for `HoodieCatalog`; the first version uses Hudi as `spark_catalog`.
+
+## Proposed User Experience
+
+The examples below implement the representative order pipeline in dependency order, using Hudi's existing SQL table options and `SET` for flow-scoped write options.
+
+### Pipeline configuration
+
+An SDP project configures the Hudi extension and catalog like any other Hudi Spark application:
+
+```yaml
+name: hudi_orders_pipeline
+libraries:
+ - glob:
+ include: transformations/**
+storage: s3://pipelines/checkpoints/hudi_orders_pipeline
+catalog: spark_catalog
+database: lakehouse
+configuration:
+ spark.sql.extensions: org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.sql.catalog.spark_catalog: org.apache.spark.sql.hudi.catalog.HoodieCatalog
+```
+
+The SQL below assumes that `order_events_source` is an external streaming source containing parsed Kafka or database CDC events. Each `CREATE ... AS SELECT` statement defines both the target dataset and its single flow.
+
+### Streaming table with an append flow
+
+```sql
+SET spark.hoodie.datasource.write.operation = insert;
+
+CREATE STREAMING TABLE bronze_order_events
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor'
+)
+AS
+SELECT *
+FROM STREAM order_events_source;
+```
+
+SDP owns the checkpoint location and trigger; Hudi commits each micro-batch.
+
+### Streaming table with keyed upserts
+
+```sql
+SET spark.hoodie.datasource.write.operation = upsert;
+
+CREATE STREAMING TABLE current_orders
+USING hudi
+PARTITIONED BY (event_date)
+TBLPROPERTIES (
+ 'type' = 'mor',
+ 'primaryKey' = 'order_id',
+ 'preCombineField' = 'event_sequence'
+)
+AS
+SELECT
+ *,
+ operation = 'DELETE' AS _hoodie_is_deleted
+FROM STREAM bronze_order_events;
+```
+
+Spark append output mode only controls how each micro-batch reaches the sink; it does not make the Hudi target append-only. `HoodieStreamingSink` still runs the configured Hudi operation, which defaults to `upsert`.
+
+For an existing Hudi table, SDP loads the record key, precombine field, partition fields, and record-merger settings from the table configuration. For a new SDP table, these settings are declared as table properties and persisted by Hudi. SDP does not implement its own deduplication, ordering, merge, or delete path. It validates the schema and configuration, then passes each micro-batch to the existing Hudi writer.
+
+### Batch materialized view
+
+```sql
+SET spark.hoodie.datasource.write.operation = bulk_insert;
+
+CREATE MATERIALIZED VIEW daily_order_metrics
+USING hudi
+PARTITIONED BY (order_date)
+TBLPROPERTIES (
+ 'type' = 'cow'
+)
+AS
+SELECT
+ order_date,
+ state,
+ COUNT(*) AS order_count
+FROM current_orders
+GROUP BY order_date, state;
+```
+
+The first run creates the table and writes the result using `bulk_insert`. Later runs truncate the target, reconcile its schema and properties, and write the recomputed result. With Hudi's current truncate implementation, each refresh removes the previous data and timeline history.
+
+## Semantics
+
+### Dataset-to-Hudi operation mapping
+
+| SDP dataset/flow | Spark 4.1 execution shape | Proposed Hudi semantics |
+| ----------------------------------- | --------------------------------------------- | -------------------------------------------------------- |
+| New materialized view | Batch append to newly created table | Hudi `bulk_insert` or `insert` |
+| Existing materialized view | Truncate, alter, then batch append | Existing Hudi truncate, then `bulk_insert` or `insert` |
+| Streaming table with an append flow | Append-mode micro-batches | Hudi `insert` or `bulk_insert`, configured per flow |
+| Hudi upsert target | Append-mode micro-batches | Existing Hudi `upsert` |
+| Hudi streaming source | Structured Streaming read with SDP checkpoint | Basic Hudi latest-state incremental source |
+
+### Materialized-view write mapping
+
+Initial materialization uses Hudi `bulk_insert` or `insert`. For an existing materialized view, Spark 4.1 executes `TRUNCATE TABLE`, reconciles schema and properties through `catalog.alterTable`, then writes the recomputed result through append-mode `saveAsTable`. Hudi uses the same configured write operation after truncate. An empty result leaves the target empty.
+
+These steps are not a single transaction. Readers may see an empty target during refresh. If schema reconciliation or the batch flow fails after truncate, the previous result is not restored, and completed catalog changes remain in effect.
+
+The initial integration reuses Hudi's current full-table truncate, which deletes old data files and timeline history. This can disrupt concurrent readers and invalidate existing incremental offsets. Timeline-preserving truncate is an independent Hudi improvement, not a prerequisite for SDP support.
+
+Materialized views are supported as batch outputs, but not as incremental sources in the initial scope.
+
+### Streaming commit and restart behavior
+
+SDP manages each flow's checkpoint location and trigger. Multiple append flows can write to the same Hudi target, each with a distinct Hudi checkpoint identifier that remains stable across restarts. Concurrent flows require Hudi's existing multi-writer configuration and an appropriate lock provider. Each flow commits and retries independently.
+
+Normal restarts reuse the same Spark checkpoint and Hudi checkpoint identifier. A streaming-table full refresh starts a fresh checkpoint and uses Hudi's current full-table truncate to remove target data, timeline history, and sink checkpoint metadata before rebuilding the table. Resetting the Spark checkpoint while retaining the Hudi target is outside this RFC's scope.
+
+### Configuration ownership and precedence
+
+Configuration is split by ownership:
+
+- **Table identity:** table type, record key, precombine field, partition fields, key generator, and record-merger settings. Hudi persists these values; SDP loads them for existing tables, and a pipeline run cannot silently change them.
+- **Write behavior:** operation and writer tuning, supplied through SQL `SET spark.hoodie.*` as flow-scoped write options.
+- **Flow execution:** checkpoint location, trigger, retry, and flow identity. SDP owns these values; the Hudi integration maps each flow to a distinct sink checkpoint identifier.
+
+SDP captures the active `SET` values at each flow definition; a `SET` remains in effect for subsequent definitions until overridden. Flow settings override pipeline and Hudi defaults and must be compatible with the persisted table configuration. The integration strips the `spark.` prefix from write options and reuses `HoodieOptionConfig` to normalize table options. Flows writing to the same target must use compatible concurrency and lock settings. The sink checkpoint identifier is resolved per flow, not shared as a table property.
+
+Invalid properties should fail during SDP `dry-run` when possible, and otherwise before the first write.
+
+### Table services
+
+Table services follow the existing Hudi writer behavior, with no SDP-specific scheduling or retention defaults. Integration tests cover service shutdown, recovery, and resource cleanup when flows finish, stop, or restart, including multiple flows sharing a target.
+
+### Schema and partition evolution
+
+SDP follows Hudi's existing schema and partition evolution rules.
+
+- Hudi-supported additive schema changes may be accepted for streaming tables.
+- Dropping, renaming, or changing columns follows Hudi's configured schema evolution behavior.
+- Record-key, precombine, and partition columns must exist in the target schema.
+- Partitioning changes fail; SDP does not turn them into an implicit table replacement.
+
+`HoodieCatalog.alterTable` validates the complete set of requested schema and property changes before applying supported changes. It accepts unchanged properties and rejects unsupported changes or changes to table identity. The Hudi writer validates the actual input schema against the persisted table configuration using its existing schema reconciliation rules.
+
+For materialized views, Spark invokes `alterTable` after truncate. A schema validation failure can therefore leave the target empty; it does not preserve the previous result. If metadata changes succeed but the batch write fails, those changes remain in effect.
+
+## Implementation
+
+### 1. Catalog integration
+
+Extend and test [HoodieCatalog](../../hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala) so SDP can:
+
+- Create a table whose provider is `hudi` from SDP's table information.
+- Persist schema, location, partition transforms, comments, and Hudi table properties.
+- Load the created table for both batch and streaming execution.
+- Validate the complete alter request before applying supported property and schema changes, preserving Hudi table identity and accepting unchanged properties.
+- Reject unsupported partition transforms and incompatible property changes with actionable errors.
+
+The first version configures `HoodieCatalog` as `spark_catalog`; named catalogs are follow-up work.
+
+This path may continue to use Hudi's V1 fallback. Full DataSource V2 support is not required as long as the SDP table contract is met.
+
+### 2. Materialized-view write path
+
+Use Hudi `bulk_insert` or `insert` for the batch result. For refresh, reuse [TruncateHoodieTableCommand](../../hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/TruncateHoodieTableCommand.scala), support the subsequent catalog reconciliation, and pass the configured operation to the append-mode writer. No SDP-specific truncate mode or refresh-detection mechanism is introduced. Validate this sequence through the integration harness before declaring materialized-view refresh supported.
+
+### 3. Streaming target integration
+
+Route SDP's append-mode `toTable` call through [HoodieStreamingSink](../../hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieStreamingSink.scala), including table properties supplied by `HoodieCatalog`.
+
+The sink integration will:
+
+- Pass the flow's write operation and persisted keyed-write settings to the existing Hudi writer.
+- Preserve the SDP checkpoint location rather than creating a competing checkpoint tree.
+- Resolve a stable, per-flow `hoodie.datasource.write.streaming.checkpoint.identifier` and use it for both retry checks and commit metadata.
+- Pass through and validate Hudi's existing multi-writer and lock configuration for concurrent flows sharing a target.
+- Reuse Hudi's existing streaming-sink checkpoint and retry behavior.
+
+No SDP-specific upsert writer, merge logic, or delete path is introduced.
+
+### 4. Incremental source integration
+
+Validate both catalog-table and explicit Hudi sources in SDP streaming flows. Source progress is restored from the SDP-managed checkpoint and advances with the downstream flow.
+
+Document:
+
+- Initial position behavior (`earliest`, `latest`, or an explicit instant).
+- The basic `latest-state` incremental output contract.
+- Behavior when required source instants have been archived or cleaned.
+- COW and MOR differences, if any, in supported streaming query modes.
+- Metadata-column handling when one Hudi table feeds another.
+
+If the checkpoint refers to source history that is no longer available, fail with a recovery error rather than skipping data.
+
+### 5. Upsert configuration and validation
+
+For an upsert target, validate that:
+
+- At least one record-key field is configured and present.
+- All partition fields are present.
+- Configured precombine fields are present and type-compatible.
+- `_hoodie_is_deleted`, when present, is boolean.
+
+These checks only validate the input contract. Hudi's existing writer, precombine, and record-merger logic perform the upsert.
+
+## Failure and Consistency Model
+
+| Failure point | Required outcome |
+| ------------------------------------------------------ | ------------------------------------------------------------------------------ |
+| Validation fails before truncate starts | Existing target remains unchanged |
+| Full-table truncate fails | Files may already have been removed; no automatic restoration is guaranteed |
+| Schema reconciliation or MV computation fails after truncate | Previous result is not restored; completed metadata changes remain in effect |
+| MV batch write fails before commit | No completed commit for that write; prior data removed by truncate is not restored |
+| Streaming task fails before Hudi commit | Spark retries the micro-batch |
+| Streaming flow retries or restarts with its existing checkpoint | Existing Hudi streaming-sink checkpoint behavior is preserved |
+| Source checkpoint points to expired Hudi history | Flow fails with an explicit recovery error |
+| Pipeline definition changes record key or partitioning | Validation fails; no implicit migration occurs, but an earlier truncate is not undone |
+
+Hudi writes retain per-commit atomicity. The current full-table truncate is a filesystem operation, not a Hudi commit, and the refresh sequence has no transaction or rollback spanning truncate, catalog changes, and the batch write. Commits from independent SDP flows are not atomic as a group.
+
+## Rollout/Adoption Plan
+
+### Phase 1: Basic catalog target
+
+- Support Spark 4.1.x with `HoodieCatalog` configured as `spark_catalog`.
+- Materialize a new, trivial SDP dataset as a Hudi-readable table.
+- Add the SDP integration harness and documentation.
+
+### Phase 2: Materialized-view refresh
+
+- Write the initial materialized-view result through Hudi `bulk_insert` or `insert`.
+- Refresh existing targets through Hudi's current truncate, catalog reconciliation, and batch write.
+- Validate repeated runs and document the history, reader, and failure limitations.
+
+### Phase 3: Streaming tables and chaining
+
+- Support one or more append streaming flows writing to the same target through the Hudi streaming sink.
+- Validate source-to-Hudi streaming-table writes.
+- Validate an upstream Hudi streaming table feeding a downstream streaming table through a basic incremental read.
+
+### Phase 4: Upsert configuration and verification
+
+- Verify that record-key, precombine, partition, and record-merger settings reach the existing Hudi writer.
+- Verify that `_hoodie_is_deleted` records use the same upsert path.
+- Add functional coverage for the resulting Hudi table state.
+
+Each phase is independently testable. Existing Hudi jobs are unaffected; the integration is used only by SDP projects that declare Hudi-backed datasets.
+
+## Test Plan
+
+The integration suite covers the four milestones:
+
+1. A trivial SDP pipeline materializes a table that can be read as Hudi.
+2. A materialized-view flow supports initial writes and repeated refreshes, including schema validation, empty results, and the documented failure behavior.
+3. Append streaming flows support source-to-Hudi writes, multiple flows sharing a target, and basic incremental Hudi-to-Hudi chaining, with isolated write options and correct results across flow restarts.
+4. A keyed flow preserves Hudi's existing upsert semantics, including delete-marker records, and produces the expected table state.
+
+Existing DataFrame, Spark SQL, and Structured Streaming tests provide regression coverage.
+
+## Alternatives Considered
+
+### Require timeline-preserving truncate before SDP support
+
+A logical truncate implemented through a replace commit would preserve timeline history and benefit Hudi SQL users as well as SDP. It can be developed and adopted independently; the SDP integration does not depend on it. Retaining the timeline would not make the truncate–alter–append sequence atomic.
+
+### Treat truncate as a no-op and overwrite on append
Review Comment:
**đź’¬** Third option worth listing here
Agree with not adopting the no-op truncate. The alternative the text skips is a timeline-preserving truncate (one replacecommit, `insert_overwrite_table` of an empty dataset) followed by the normal append. It keeps `TRUNCATE` semantics for SQL users, needs no refresh detection, and leaves only the already-documented empty window between the two commits. See the summary comment for why I think this belongs in scope rather than as an independent improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org