You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "moomindani (via GitHub)" <gi...@apache.org> on 2026/04/09 08:07:13 UTC
[PR] [SPARK-55276][DOCS] Document how SDP datasets are stored and refreshed [spark]
moomindani opened a new pull request, #55277:
URL: https://github.com/apache/spark/pull/55277
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
Add a new "How Datasets are Stored and Refreshed" section to the Spark Declarative Pipelines programming guide. This section covers:
- **Table Format**: Default format (`parquet` via `spark.sql.sources.default`) and how to specify a different format with Python and SQL examples
- **How Materialized Views are Refreshed**: Full recomputation (TRUNCATE + append) on every pipeline run, and how this differs from database-native materialized views
- **How Streaming Tables are Refreshed**: Incremental processing with checkpoints and schema evolution support
- **Full Refresh**: Behavior differences between materialized views and streaming tables
### Why are the changes needed?
The current programming guide explains how to define datasets but does not explain how they are stored or refreshed. Users need to understand:
- What format their tables are stored in by default
- That materialized views perform a full recomputation on every run (unlike PostgreSQL-style MVs)
- That streaming tables require checkpoint storage on a Hadoop-compatible file system
- What `--full-refresh` actually does for each dataset type
Without this information, users cannot make informed decisions about table formats, storage configurations, or pipeline performance.
### Does this PR introduce _any_ user-facing change?
No. Documentation only.
### How was this patch tested?
Documentation change only. Verified the content is accurate by reading the SDP implementation (`DatasetManager.scala`, `FlowExecution.scala`).
### Was this patch authored or co-authored using generative AI tooling?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56451][DOCS] Document how SDP datasets are stored and refreshed [spark]
Posted by "jaceklaskowski (via GitHub)" <gi...@apache.org>.
jaceklaskowski commented on PR #55277:
URL: https://github.com/apache/spark/pull/55277#issuecomment-4230052802
Please add `[SDP]` tag to the title 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56451][DOCS][SDP] Document how SDP datasets are stored and refreshed [spark]
Posted by "moomindani (via GitHub)" <gi...@apache.org>.
moomindani commented on code in PR #55277:
URL: https://github.com/apache/spark/pull/55277#discussion_r3070403887
##########
docs/declarative-pipelines-programming-guide.md:
##########
@@ -536,6 +536,76 @@ When working with sinks, keep the following considerations in mind:
- **Python API**: Sink functionality is available only through the Python API, not SQL
- **Append-only**: Only append operations are supported; full refresh updates reset checkpoints but do not clean previously computed results
+## How Datasets are Stored and Refreshed
+
+This section describes how SDP manages the underlying storage for datasets. Understanding these mechanics helps you choose appropriate table formats and storage configurations.
+
+### Table Format
+
+By default, SDP creates tables using Spark's default table format, which is configured by the `spark.sql.sources.default` property (default: `parquet`). You can specify a different format for individual datasets:
+
+<div class="codetabs">
+<div data-lang="python" markdown="1">
+
+```python
+@dp.table(format="delta")
+def my_streaming_table() -> DataFrame:
+ return spark.readStream.table("source")
+
+@dp.materialized_view(format="orc")
+def my_materialized_view() -> DataFrame:
+ return spark.read.table("source")
+```
+
+</div>
+<div data-lang="SQL" markdown="1">
+
+```sql
+CREATE STREAMING TABLE my_streaming_table
+USING DELTA
+AS SELECT * FROM STREAM source;
+
+CREATE MATERIALIZED VIEW my_materialized_view
+USING ORC
+AS SELECT * FROM source;
+```
+
+</div>
+</div>
+
+SDP itself does not restrict which table formats can be used. However, the table format must be supported by the configured catalog. For example, a Delta catalog only supports Delta tables, while the default session catalog supports Parquet, ORC, and other built-in formats.
Review Comment:
Thank you for the feedback. Revised to: "SDP itself does not restrict which table formats can be used. Any table format available in your Spark environment can be specified. By default, tables are created using Spark's default format (`parquet`), which is configured by `spark.sql.sources.default`."
##########
docs/declarative-pipelines-programming-guide.md:
##########
@@ -536,6 +536,76 @@ When working with sinks, keep the following considerations in mind:
- **Python API**: Sink functionality is available only through the Python API, not SQL
- **Append-only**: Only append operations are supported; full refresh updates reset checkpoints but do not clean previously computed results
+## How Datasets are Stored and Refreshed
+
+This section describes how SDP manages the underlying storage for datasets. Understanding these mechanics helps you choose appropriate table formats and storage configurations.
+
+### Table Format
+
+By default, SDP creates tables using Spark's default table format, which is configured by the `spark.sql.sources.default` property (default: `parquet`). You can specify a different format for individual datasets:
+
+<div class="codetabs">
+<div data-lang="python" markdown="1">
+
+```python
+@dp.table(format="delta")
+def my_streaming_table() -> DataFrame:
+ return spark.readStream.table("source")
+
+@dp.materialized_view(format="orc")
+def my_materialized_view() -> DataFrame:
+ return spark.read.table("source")
+```
+
+</div>
+<div data-lang="SQL" markdown="1">
+
+```sql
+CREATE STREAMING TABLE my_streaming_table
+USING DELTA
+AS SELECT * FROM STREAM source;
+
+CREATE MATERIALIZED VIEW my_materialized_view
+USING ORC
+AS SELECT * FROM source;
+```
+
+</div>
+</div>
+
+SDP itself does not restrict which table formats can be used. However, the table format must be supported by the configured catalog. For example, a Delta catalog only supports Delta tables, while the default session catalog supports Parquet, ORC, and other built-in formats.
+
+### How Materialized Views are Refreshed
+
+A materialized view in SDP is **not** the same as a database-native materialized view (e.g., those in PostgreSQL or Oracle). SDP materialized views work as follows:
+
+1. On each pipeline run, the entire query is re-executed.
+2. The existing table data is truncated.
+3. The new query results are appended to the table.
+
+This means that every refresh is a **full recomputation** - there is no incremental or differential update. For tables with large amounts of data, be aware that each pipeline run will reprocess the entire dataset.
+
+Because of this mechanism, the materialized view's underlying table format must support the `TRUNCATE TABLE` operation.
Review Comment:
Thank you for the suggestion. I checked and the built-in formats such as Parquet, ORC, JSON, and CSV also support `TRUNCATE TABLE`, so I kept this without a specific example to avoid implying it is limited to Delta Lake.
##########
docs/declarative-pipelines-programming-guide.md:
##########
@@ -536,6 +536,76 @@ When working with sinks, keep the following considerations in mind:
- **Python API**: Sink functionality is available only through the Python API, not SQL
- **Append-only**: Only append operations are supported; full refresh updates reset checkpoints but do not clean previously computed results
+## How Datasets are Stored and Refreshed
+
+This section describes how SDP manages the underlying storage for datasets. Understanding these mechanics helps you choose appropriate table formats and storage configurations.
+
+### Table Format
+
+By default, SDP creates tables using Spark's default table format, which is configured by the `spark.sql.sources.default` property (default: `parquet`). You can specify a different format for individual datasets:
+
+<div class="codetabs">
+<div data-lang="python" markdown="1">
+
+```python
+@dp.table(format="delta")
+def my_streaming_table() -> DataFrame:
+ return spark.readStream.table("source")
+
+@dp.materialized_view(format="orc")
+def my_materialized_view() -> DataFrame:
+ return spark.read.table("source")
+```
+
+</div>
+<div data-lang="SQL" markdown="1">
+
+```sql
+CREATE STREAMING TABLE my_streaming_table
+USING DELTA
+AS SELECT * FROM STREAM source;
+
+CREATE MATERIALIZED VIEW my_materialized_view
+USING ORC
+AS SELECT * FROM source;
+```
+
+</div>
+</div>
+
+SDP itself does not restrict which table formats can be used. However, the table format must be supported by the configured catalog. For example, a Delta catalog only supports Delta tables, while the default session catalog supports Parquet, ORC, and other built-in formats.
+
+### How Materialized Views are Refreshed
+
+A materialized view in SDP is **not** the same as a database-native materialized view (e.g., those in PostgreSQL or Oracle). SDP materialized views work as follows:
+
+1. On each pipeline run, the entire query is re-executed.
+2. The existing table data is truncated.
+3. The new query results are appended to the table.
+
+This means that every refresh is a **full recomputation** - there is no incremental or differential update. For tables with large amounts of data, be aware that each pipeline run will reprocess the entire dataset.
+
+Because of this mechanism, the materialized view's underlying table format must support the `TRUNCATE TABLE` operation.
+
+### How Streaming Tables are Refreshed
+
+Unlike materialized views, streaming tables support **incremental processing**:
+
+1. On each pipeline run, only new data from the source is processed.
+2. New data is appended to the existing table data.
+3. A checkpoint tracks the processing progress so subsequent runs resume from where the last run left off.
+
+Streaming tables require a checkpoint directory on a Hadoop-compatible file system (e.g., HDFS, Amazon S3, Azure ADLS Gen2, Google Cloud Storage, or local file system). The checkpoint directory is configured via the `storage` field in the pipeline spec file.
Review Comment:
Applied, thank you.
##########
docs/declarative-pipelines-programming-guide.md:
##########
@@ -536,6 +536,76 @@ When working with sinks, keep the following considerations in mind:
- **Python API**: Sink functionality is available only through the Python API, not SQL
- **Append-only**: Only append operations are supported; full refresh updates reset checkpoints but do not clean previously computed results
+## How Datasets are Stored and Refreshed
+
+This section describes how SDP manages the underlying storage for datasets. Understanding these mechanics helps you choose appropriate table formats and storage configurations.
+
+### Table Format
+
+By default, SDP creates tables using Spark's default table format, which is configured by the `spark.sql.sources.default` property (default: `parquet`). You can specify a different format for individual datasets:
+
+<div class="codetabs">
+<div data-lang="python" markdown="1">
+
+```python
+@dp.table(format="delta")
+def my_streaming_table() -> DataFrame:
+ return spark.readStream.table("source")
+
+@dp.materialized_view(format="orc")
+def my_materialized_view() -> DataFrame:
+ return spark.read.table("source")
+```
+
+</div>
+<div data-lang="SQL" markdown="1">
+
+```sql
+CREATE STREAMING TABLE my_streaming_table
+USING DELTA
+AS SELECT * FROM STREAM source;
+
+CREATE MATERIALIZED VIEW my_materialized_view
+USING ORC
+AS SELECT * FROM source;
+```
+
+</div>
+</div>
+
+SDP itself does not restrict which table formats can be used. However, the table format must be supported by the configured catalog. For example, a Delta catalog only supports Delta tables, while the default session catalog supports Parquet, ORC, and other built-in formats.
+
+### How Materialized Views are Refreshed
+
+A materialized view in SDP is **not** the same as a database-native materialized view (e.g., those in PostgreSQL or Oracle). SDP materialized views work as follows:
+
+1. On each pipeline run, the entire query is re-executed.
+2. The existing table data is truncated.
+3. The new query results are appended to the table.
+
+This means that every refresh is a **full recomputation** - there is no incremental or differential update. For tables with large amounts of data, be aware that each pipeline run will reprocess the entire dataset.
+
+Because of this mechanism, the materialized view's underlying table format must support the `TRUNCATE TABLE` operation.
+
+### How Streaming Tables are Refreshed
+
+Unlike materialized views, streaming tables support **incremental processing**:
+
+1. On each pipeline run, only new data from the source is processed.
+2. New data is appended to the existing table data.
+3. A checkpoint tracks the processing progress so subsequent runs resume from where the last run left off.
+
+Streaming tables require a checkpoint directory on a Hadoop-compatible file system (e.g., HDFS, Amazon S3, Azure ADLS Gen2, Google Cloud Storage, or local file system). The checkpoint directory is configured via the `storage` field in the pipeline spec file.
+
+Streaming tables also support **schema evolution**: when the schema of incoming data changes, SDP merges the new schema with the existing table schema automatically.
+
+### Full Refresh
+
+You can force a full refresh of specific datasets or the entire pipeline using the `--full-refresh` or `--full-refresh-all` CLI options. A full refresh:
Review Comment:
Applied, thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56451][DOCS] Document how SDP datasets are stored and refreshed [spark]
Posted by "moomindani (via GitHub)" <gi...@apache.org>.
moomindani commented on PR #55277:
URL: https://github.com/apache/spark/pull/55277#issuecomment-4227509879
Thank you for pointing that out, @dongjoon-hyun. I've updated the PR title and commit message to use the correct JIRA ID: [SPARK-56451](https://issues.apache.org/jira/browse/SPARK-56451). The GitHub issue has been closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56451][DOCS][SDP] Document how SDP datasets are stored and refreshed [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #55277:
URL: https://github.com/apache/spark/pull/55277#issuecomment-5124921928
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56451][DOCS][SDP] Document how SDP datasets are stored and refreshed [spark]
Posted by "moomindani (via GitHub)" <gi...@apache.org>.
moomindani commented on PR #55277:
URL: https://github.com/apache/spark/pull/55277#issuecomment-4233098630
Added `[SDP]` tag to the title. Thank you, @jaceklaskowski.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56451][DOCS][SDP] Document how SDP datasets are stored and refreshed [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #55277: [SPARK-56451][DOCS][SDP] Document how SDP datasets are stored and refreshed
URL: https://github.com/apache/spark/pull/55277
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56451][DOCS] Document how SDP datasets are stored and refreshed [spark]
Posted by "jaceklaskowski (via GitHub)" <gi...@apache.org>.
jaceklaskowski commented on code in PR #55277:
URL: https://github.com/apache/spark/pull/55277#discussion_r3068468865
##########
docs/declarative-pipelines-programming-guide.md:
##########
@@ -536,6 +536,76 @@ When working with sinks, keep the following considerations in mind:
- **Python API**: Sink functionality is available only through the Python API, not SQL
- **Append-only**: Only append operations are supported; full refresh updates reset checkpoints but do not clean previously computed results
+## How Datasets are Stored and Refreshed
+
+This section describes how SDP manages the underlying storage for datasets. Understanding these mechanics helps you choose appropriate table formats and storage configurations.
+
+### Table Format
+
+By default, SDP creates tables using Spark's default table format, which is configured by the `spark.sql.sources.default` property (default: `parquet`). You can specify a different format for individual datasets:
+
+<div class="codetabs">
+<div data-lang="python" markdown="1">
+
+```python
+@dp.table(format="delta")
+def my_streaming_table() -> DataFrame:
+ return spark.readStream.table("source")
+
+@dp.materialized_view(format="orc")
+def my_materialized_view() -> DataFrame:
+ return spark.read.table("source")
+```
+
+</div>
+<div data-lang="SQL" markdown="1">
+
+```sql
+CREATE STREAMING TABLE my_streaming_table
+USING DELTA
+AS SELECT * FROM STREAM source;
+
+CREATE MATERIALIZED VIEW my_materialized_view
+USING ORC
+AS SELECT * FROM source;
+```
+
+</div>
+</div>
+
+SDP itself does not restrict which table formats can be used. However, the table format must be supported by the configured catalog. For example, a Delta catalog only supports Delta tables, while the default session catalog supports Parquet, ORC, and other built-in formats.
Review Comment:
What's a "catalog" here? Table formats are set up via packages on command line when Spark Connect server's started.
##########
docs/declarative-pipelines-programming-guide.md:
##########
@@ -536,6 +536,76 @@ When working with sinks, keep the following considerations in mind:
- **Python API**: Sink functionality is available only through the Python API, not SQL
- **Append-only**: Only append operations are supported; full refresh updates reset checkpoints but do not clean previously computed results
+## How Datasets are Stored and Refreshed
+
+This section describes how SDP manages the underlying storage for datasets. Understanding these mechanics helps you choose appropriate table formats and storage configurations.
+
+### Table Format
+
+By default, SDP creates tables using Spark's default table format, which is configured by the `spark.sql.sources.default` property (default: `parquet`). You can specify a different format for individual datasets:
+
+<div class="codetabs">
+<div data-lang="python" markdown="1">
+
+```python
+@dp.table(format="delta")
+def my_streaming_table() -> DataFrame:
+ return spark.readStream.table("source")
+
+@dp.materialized_view(format="orc")
+def my_materialized_view() -> DataFrame:
+ return spark.read.table("source")
+```
+
+</div>
+<div data-lang="SQL" markdown="1">
+
+```sql
+CREATE STREAMING TABLE my_streaming_table
+USING DELTA
+AS SELECT * FROM STREAM source;
+
+CREATE MATERIALIZED VIEW my_materialized_view
+USING ORC
+AS SELECT * FROM source;
+```
+
+</div>
+</div>
+
+SDP itself does not restrict which table formats can be used. However, the table format must be supported by the configured catalog. For example, a Delta catalog only supports Delta tables, while the default session catalog supports Parquet, ORC, and other built-in formats.
+
+### How Materialized Views are Refreshed
+
+A materialized view in SDP is **not** the same as a database-native materialized view (e.g., those in PostgreSQL or Oracle). SDP materialized views work as follows:
+
+1. On each pipeline run, the entire query is re-executed.
+2. The existing table data is truncated.
+3. The new query results are appended to the table.
+
+This means that every refresh is a **full recomputation** - there is no incremental or differential update. For tables with large amounts of data, be aware that each pipeline run will reprocess the entire dataset.
+
+Because of this mechanism, the materialized view's underlying table format must support the `TRUNCATE TABLE` operation.
+
+### How Streaming Tables are Refreshed
+
+Unlike materialized views, streaming tables support **incremental processing**:
+
+1. On each pipeline run, only new data from the source is processed.
+2. New data is appended to the existing table data.
+3. A checkpoint tracks the processing progress so subsequent runs resume from where the last run left off.
+
+Streaming tables require a checkpoint directory on a Hadoop-compatible file system (e.g., HDFS, Amazon S3, Azure ADLS Gen2, Google Cloud Storage, or local file system). The checkpoint directory is configured via the `storage` field in the pipeline spec file.
Review Comment:
```suggestion
Streaming tables require a checkpoint directory on a Hadoop-compatible file system (e.g., local file system, HDFS, Amazon S3, Azure ADLS Gen2, Google Cloud Storage). The checkpoint directory is configured via the `storage` field in the pipeline spec file.
```
##########
docs/declarative-pipelines-programming-guide.md:
##########
@@ -536,6 +536,76 @@ When working with sinks, keep the following considerations in mind:
- **Python API**: Sink functionality is available only through the Python API, not SQL
- **Append-only**: Only append operations are supported; full refresh updates reset checkpoints but do not clean previously computed results
+## How Datasets are Stored and Refreshed
+
+This section describes how SDP manages the underlying storage for datasets. Understanding these mechanics helps you choose appropriate table formats and storage configurations.
+
+### Table Format
+
+By default, SDP creates tables using Spark's default table format, which is configured by the `spark.sql.sources.default` property (default: `parquet`). You can specify a different format for individual datasets:
+
+<div class="codetabs">
+<div data-lang="python" markdown="1">
+
+```python
+@dp.table(format="delta")
+def my_streaming_table() -> DataFrame:
+ return spark.readStream.table("source")
+
+@dp.materialized_view(format="orc")
+def my_materialized_view() -> DataFrame:
+ return spark.read.table("source")
+```
+
+</div>
+<div data-lang="SQL" markdown="1">
+
+```sql
+CREATE STREAMING TABLE my_streaming_table
+USING DELTA
+AS SELECT * FROM STREAM source;
+
+CREATE MATERIALIZED VIEW my_materialized_view
+USING ORC
+AS SELECT * FROM source;
+```
+
+</div>
+</div>
+
+SDP itself does not restrict which table formats can be used. However, the table format must be supported by the configured catalog. For example, a Delta catalog only supports Delta tables, while the default session catalog supports Parquet, ORC, and other built-in formats.
+
+### How Materialized Views are Refreshed
+
+A materialized view in SDP is **not** the same as a database-native materialized view (e.g., those in PostgreSQL or Oracle). SDP materialized views work as follows:
+
+1. On each pipeline run, the entire query is re-executed.
+2. The existing table data is truncated.
+3. The new query results are appended to the table.
+
+This means that every refresh is a **full recomputation** - there is no incremental or differential update. For tables with large amounts of data, be aware that each pipeline run will reprocess the entire dataset.
+
+Because of this mechanism, the materialized view's underlying table format must support the `TRUNCATE TABLE` operation.
Review Comment:
```suggestion
Because of this mechanism, the materialized view's underlying table format must support the `TRUNCATE TABLE` operation (e.g., Delta Lake).
```
##########
docs/declarative-pipelines-programming-guide.md:
##########
@@ -536,6 +536,76 @@ When working with sinks, keep the following considerations in mind:
- **Python API**: Sink functionality is available only through the Python API, not SQL
- **Append-only**: Only append operations are supported; full refresh updates reset checkpoints but do not clean previously computed results
+## How Datasets are Stored and Refreshed
+
+This section describes how SDP manages the underlying storage for datasets. Understanding these mechanics helps you choose appropriate table formats and storage configurations.
+
+### Table Format
+
+By default, SDP creates tables using Spark's default table format, which is configured by the `spark.sql.sources.default` property (default: `parquet`). You can specify a different format for individual datasets:
+
+<div class="codetabs">
+<div data-lang="python" markdown="1">
+
+```python
+@dp.table(format="delta")
+def my_streaming_table() -> DataFrame:
+ return spark.readStream.table("source")
+
+@dp.materialized_view(format="orc")
+def my_materialized_view() -> DataFrame:
+ return spark.read.table("source")
+```
+
+</div>
+<div data-lang="SQL" markdown="1">
+
+```sql
+CREATE STREAMING TABLE my_streaming_table
+USING DELTA
+AS SELECT * FROM STREAM source;
+
+CREATE MATERIALIZED VIEW my_materialized_view
+USING ORC
+AS SELECT * FROM source;
+```
+
+</div>
+</div>
+
+SDP itself does not restrict which table formats can be used. However, the table format must be supported by the configured catalog. For example, a Delta catalog only supports Delta tables, while the default session catalog supports Parquet, ORC, and other built-in formats.
+
+### How Materialized Views are Refreshed
+
+A materialized view in SDP is **not** the same as a database-native materialized view (e.g., those in PostgreSQL or Oracle). SDP materialized views work as follows:
+
+1. On each pipeline run, the entire query is re-executed.
+2. The existing table data is truncated.
+3. The new query results are appended to the table.
+
+This means that every refresh is a **full recomputation** - there is no incremental or differential update. For tables with large amounts of data, be aware that each pipeline run will reprocess the entire dataset.
+
+Because of this mechanism, the materialized view's underlying table format must support the `TRUNCATE TABLE` operation.
+
+### How Streaming Tables are Refreshed
+
+Unlike materialized views, streaming tables support **incremental processing**:
+
+1. On each pipeline run, only new data from the source is processed.
+2. New data is appended to the existing table data.
+3. A checkpoint tracks the processing progress so subsequent runs resume from where the last run left off.
+
+Streaming tables require a checkpoint directory on a Hadoop-compatible file system (e.g., HDFS, Amazon S3, Azure ADLS Gen2, Google Cloud Storage, or local file system). The checkpoint directory is configured via the `storage` field in the pipeline spec file.
+
+Streaming tables also support **schema evolution**: when the schema of incoming data changes, SDP merges the new schema with the existing table schema automatically.
+
+### Full Refresh
+
+You can force a full refresh of specific datasets or the entire pipeline using the `--full-refresh` or `--full-refresh-all` CLI options. A full refresh:
Review Comment:
```suggestion
You can force a full refresh of specific datasets or the entire pipeline using the `--full-refresh` or `--full-refresh-all` CLI options, respectively. A full refresh:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org