You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/08/11 11:33:54 UTC
(datafusion) branch gh-readonly-queue/main/pr-24155-4978706c64d9785ad9908dba26edff98d327bf14 created (now d5868e6834)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-24155-4978706c64d9785ad9908dba26edff98d327bf14
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at d5868e6834 docs: explain Parquet content-defined chunking (#24155)
This branch includes the following new commits:
new d5868e6834 docs: explain Parquet content-defined chunking (#24155)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: docs: explain Parquet content-defined chunking (#24155)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-24155-4978706c64d9785ad9908dba26edff98d327bf14
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit d5868e683435042aa5fa72465a8a2d667d8cc468
Author: Goutam Adwant <86...@users.noreply.github.com>
AuthorDate: Tue Aug 11 04:33:30 2026 -0700
docs: explain Parquet content-defined chunking (#24155)
## Which issue does this PR close?
- Closes #21404.
## Rationale for this change
Parquet content-defined chunking is available as an experimental writer
feature, but the user documentation does not explain when it is useful
or how to configure it.
Users need to understand that CDC benefits storage or transfer systems
that reuse duplicate byte ranges, does not perform deduplication itself,
and requires the sequential writer path for each output file.
## What changes are included in this PR?
- Add a user guide explaining Parquet content-defined chunking from a
user perspective.
- Describe appropriate use cases, storage requirements, limitations, and
the sequential-writer tradeoff.
- Explain how directory output distributes batches across files and why
stable input ordering and file layout improve deduplication.
- Provide single-file SQL and Rust examples, session-level
configuration, defaults, and tuning guidance.
- Add the CDC writer options to the Parquet format-options reference.
- Add the guide to the documentation index near the advanced user
guides.
## Are these changes tested?
No test code is changed because this is a documentation-only update.
The documented behavior and examples were verified with:
- `cargo test --profile=ci --test sqllogictests -- parquet_cdc.slt`
- `cargo fmt --all -- --check`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `./ci/scripts/doc_prettier_check.sh`
- `git diff --check`
## Are there any user-facing changes?
Yes. This adds user-facing documentation for the existing experimental
Parquet CDC feature. It does not change runtime behavior or public APIs.
---
docs/source/index.rst | 1 +
.../user-guide/parquet-content-defined-chunking.md | 163 +++++++++++++++++++++
docs/source/user-guide/sql/format_options.md | 4 +
3 files changed, 168 insertions(+)
diff --git a/docs/source/index.rst b/docs/source/index.rst
index b939be86a0..0e3f56a7e1 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -126,6 +126,7 @@ To get started, see
user-guide/sql/index
user-guide/configs
user-guide/explain-usage
+ user-guide/parquet-content-defined-chunking
user-guide/metrics
user-guide/faq
diff --git a/docs/source/user-guide/parquet-content-defined-chunking.md b/docs/source/user-guide/parquet-content-defined-chunking.md
new file mode 100644
index 0000000000..d456200e69
--- /dev/null
+++ b/docs/source/user-guide/parquet-content-defined-chunking.md
@@ -0,0 +1,163 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Parquet Content-Defined Chunking
+
+Content-defined chunking (CDC) is an experimental Parquet writer feature that
+makes data page boundaries depend on column values rather than fixed row or byte
+counts. This makes unchanged regions more likely to produce identical pages when
+closely related versions of a dataset are written with the same settings.
+
+CDC is useful when the resulting files are stored or transferred through a
+content-addressable or block-deduplicating system. Such a system can reuse the
+identical pages instead of storing or transferring them again. For example, a
+small insertion near the beginning of a dataset can change one page while later
+page boundaries converge back to those of the previous version.
+
+CDC does not itself deduplicate data or provide a page store. On a conventional
+filesystem or object store, each Parquet file is still stored in full. The output
+is a normal Parquet file and requires no CDC-specific reader support.
+
+## When to enable CDC
+
+Consider CDC when all of the following apply:
+
+- You regularly write similar versions of the same dataset.
+- Your storage or transfer layer detects and reuses duplicate byte ranges.
+- Reducing storage or network transfer is more important than maximizing write
+ parallelism for an individual file.
+
+Leave CDC disabled for ordinary Parquet output unless you have measured a benefit
+in the system that stores or transfers the files. CDC is disabled by default.
+
+When CDC is enabled, DataFusion uses the sequential Arrow writer for each output
+file because the chunker's state must persist across row groups. This can reduce
+write throughput compared with DataFusion's parallel writer path. Writing
+different output files can still proceed concurrently.
+
+CDC operates independently for each output file. When `COPY` targets a
+directory, DataFusion distributes input RecordBatches in round-robin order across
+parallel output files; `datafusion.execution.minimum_parallel_output_files`
+defaults to four. If batching or file assignment changes between dataset
+versions, unchanged rows can move between files and reduce deduplication. For
+the best results, keep the input order and output file layout stable. Use a
+filename target for single-file output, or partition by stable keys when
+multiple files are required. See
+[Configuration Settings](configs.md#setting-configuration-options).
+
+## Enable CDC with SQL
+
+Set CDC for one [`COPY`](sql/dml.md#copy) operation with Parquet format options.
+The filename target in this example selects single-file output:
+
+```sql
+COPY (
+ SELECT
+ value AS id,
+ CONCAT('event-', CAST(value AS VARCHAR)) AS event
+ FROM generate_series(1, 100000)
+) TO 'cdc-output.parquet'
+STORED AS PARQUET
+OPTIONS (
+ 'format.content_defined_chunking.enabled' 'true'
+);
+```
+
+The default chunking parameters are a good starting point. The next example
+specifies those defaults explicitly for one write; it does not change their
+values:
+
+```sql
+COPY source_table TO 'cdc-output.parquet'
+STORED AS PARQUET
+OPTIONS (
+ 'format.content_defined_chunking.enabled' 'true',
+ 'format.content_defined_chunking.min_chunk_size' '262144',
+ 'format.content_defined_chunking.max_chunk_size' '1048576',
+ 'format.content_defined_chunking.norm_level' '0'
+);
+```
+
+Change these values only after measuring with representative data.
+
+You can instead enable CDC for subsequent Parquet writes in the session:
+
+```sql
+SET datafusion.execution.parquet.content_defined_chunking.enabled = true;
+```
+
+The corresponding environment variable is
+`DATAFUSION_EXECUTION_PARQUET_CONTENT_DEFINED_CHUNKING_ENABLED`. See
+[Configuration Settings](configs.md#setting-configuration-options) for all ways
+to set session options.
+
+## Enable CDC with the Rust API
+
+Pass [`TableParquetOptions`] to [`DataFrame::write_parquet`]:
+
+```rust
+use datafusion::config::{ParquetCdcOptions, TableParquetOptions};
+use datafusion::dataframe::DataFrameWriteOptions;
+use datafusion::error::Result;
+use datafusion::prelude::SessionContext;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+ let ctx = SessionContext::new();
+ let df = ctx
+ .sql("SELECT value AS id FROM generate_series(1, 100000)")
+ .await?;
+
+ let mut parquet_options = TableParquetOptions::default();
+ parquet_options.global.content_defined_chunking = ParquetCdcOptions::enabled();
+
+ df.write_parquet(
+ "cdc-output.parquet",
+ DataFrameWriteOptions::new().with_single_file_output(true),
+ Some(parquet_options),
+ )
+ .await?;
+
+ Ok(())
+}
+```
+
+Set the fields of `ParquetCdcOptions` directly to use non-default chunk sizes or
+normalization.
+
+## Tuning
+
+| Option | Default | Effect |
+| ---------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `min_chunk_size` | 256 KiB | Minimum logical size before the rolling hash can select a boundary. |
+| `max_chunk_size` | 1 MiB | Maximum logical size before the writer forces a boundary. It must be greater than `min_chunk_size`. |
+| `norm_level` | `0` | Controls how aggressively boundaries are selected. Higher values can improve deduplication but create more small pages; recommended range is `-3` through `3`. |
+
+Chunk sizes are measured from logical column data before encoding and
+compression. Definition and repetition levels for nested data also count toward
+the size.
+
+Use the same CDC, encoding, compression, and schema settings when comparing
+dataset versions. Changing writer settings can change the page bytes and reduce
+deduplication even when the logical data is unchanged. Measure the deduplication
+ratio, output size, network transfer, and write time with representative data
+before changing the defaults.
+
+[`dataframe::write_parquet`]: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html#method.write_parquet
+[`tableparquetoptions`]: https://docs.rs/datafusion/latest/datafusion/common/config/struct.TableParquetOptions.html
diff --git a/docs/source/user-guide/sql/format_options.md b/docs/source/user-guide/sql/format_options.md
index ca79858dae..719fd5bd6b 100644
--- a/docs/source/user-guide/sql/format_options.md
+++ b/docs/source/user-guide/sql/format_options.md
@@ -164,6 +164,10 @@ The following options are available when reading or writing Parquet files. If an
| ALLOW_SINGLE_FILE_PARALLELISM | No | Enables parallel serialization of columns in a single file. | `'allow_single_file_parallelism'` | true |
| MAXIMUM_PARALLEL_ROW_GROUP_WRITERS | No | Maximum number of parallel row group writers. | `'maximum_parallel_row_group_writers'` | 1 |
| MAXIMUM_BUFFERED_RECORD_BATCHES_PER_STREAM | No | Maximum number of buffered record batches per stream. | `'maximum_buffered_record_batches_per_stream'` | 2 |
+| CONTENT_DEFINED_CHUNKING_ENABLED | No | Enables experimental content-defined chunking when writing Parquet files. Enabling it uses the sequential writer for each output file so chunker state persists across row groups. See [Parquet Content-Defined Chunking](../parquet-content-defined-chunking.md). | `'content_defined_chunking.enabled'` | false |
+| CONTENT_DEFINED_CHUNKING_MIN_CHUNK_SIZE | No | Minimum logical size in bytes before the rolling hash can select a chunk boundary. | `'content_defined_chunking.min_chunk_size'` | 262144 (256 KiB) |
+| CONTENT_DEFINED_CHUNKING_MAX_CHUNK_SIZE | No | Maximum logical size in bytes before the writer forces a chunk boundary. Must be greater than `content_defined_chunking.min_chunk_size`. | `'content_defined_chunking.max_chunk_size'` | 1048576 (1 MiB) |
+| CONTENT_DEFINED_CHUNKING_NORM_LEVEL | No | Controls how aggressively chunk boundaries are selected. Higher values can improve deduplication but increase fragmentation. The recommended range is `-3` through `3`. | `'content_defined_chunking.norm_level'` | 0 |
| KEY_VALUE_METADATA | No (Key is specific) | Adds custom key-value pairs to the file metadata. Use the format `'metadata::your_key_name' 'your_value'`. Multiple entries allowed. | `'metadata::key_name'` | None |
**Example:**
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org