You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/05 17:12:36 UTC

(datafusion-comet) branch gh-readonly-queue/main/pr-6241-83285bbe1b604e3958bffceca2e4dc9137d98fec created (now 74257803f1)

This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-6241-83285bbe1b604e3958bffceca2e4dc9137d98fec
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git


      at 74257803f1 perf: choose dictionary encoding like iceberg-java in the native Iceberg writer (#6241)

This branch includes the following new commits:

     new 74257803f1 perf: choose dictionary encoding like iceberg-java in the native Iceberg writer (#6241)

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org


(datafusion-comet) 01/01: perf: choose dictionary encoding like iceberg-java in the native Iceberg writer (#6241)

Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6241-83285bbe1b604e3958bffceca2e4dc9137d98fec
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git

commit 74257803f1a364a4e0b177f526b5ba30f637df80
Author: Andy Grove <ag...@apache.org>
AuthorDate: Mon Oct 5 14:04:50 2026 +0000

    perf: choose dictionary encoding like iceberg-java in the native Iceberg writer (#6241)
    
    * perf: choose dictionary encoding like iceberg-java in the native Iceberg writer
    
    parquet-mr, which iceberg-java writes through, drops dictionary encoding for a
    column chunk when the chunk's first data page shows the dictionary saving no
    space, and then writes no dictionary page. parquet-rs keeps the dictionary until
    it fills and still writes the page, so a high-cardinality column in a
    natively written file carried a dictionary page of up to
    write.parquet.dict-size-bytes that every selective read had to fetch.
    
    The native writer now makes parquet-mr's decision before it opens a partition's
    first file. Each partition's first page of rows is held back, DictionaryChooser
    replays parquet-mr's first-page size accounting over those rows, and the
    partition's files are opened with dictionary encoding off for the columns
    parquet-mr would write plain. Across all of a task's partitions, the held rows
    are bounded by the row group size. The roll grid does not move.
    
    Closes #6114.
    
    * Read only the columns with a choice to make, lazily, when choosing dictionaries
    
    The chooser expanded every nested column's entries into vectors of child
    indices before it looked at the leaf, even for leaves with no choice to make:
    1000 rows of 8192-element boolean lists, about 1 MiB of input, cost 128 MiB of
    temporary allocation. It also kept every column's page state, including one id
    per value, until the last column was judged.
    
    It now reads only the leaf columns there is a choice for, one at a time, and
    walks each through a lazy iterator that stops where the column's first page
    ends. The RLE-hybrid length is counted from whether each id repeats the one
    before it, so no per-value ids are kept, and what the chooser holds is one
    column's dictionary. Decisions are unchanged; the parquet-mr ground-truth tests
    pass as before.
    
    * Fix the Iceberg write suite's build after merging main
    
    Main's suite now imports java.nio.file.Path, which made this PR's
    org.apache.hadoop.fs.Path import ambiguous. Import the Hadoop class as
    HadoopPath instead, and put the Parquet imports in the org.apache group
    that spotless expects.
    
    * Give the mid-write retry test a first page that ends before its failure
    
    The test from #6111 needs the native writer to have finalized a data
    file by row 7000 of 10000. This PR holds each partition's first data
    page of rows back for the dictionary choice, and the default page is
    20000 rows, so nothing reached disk before the injected failure. A
    1000-row page keeps the test on its original timeline.
---
 docs/source/contributor-guide/iceberg-writes.md    |   24 +-
 docs/source/user-guide/latest/iceberg-writes.md    |   27 +-
 .../src/execution/operators/iceberg_dictionary.rs  | 1277 ++++++++++++++++++++
 .../core/src/execution/operators/iceberg_write.rs  |  565 +++++++--
 native/core/src/execution/operators/mod.rs         |    1 +
 .../comet/CometIcebergWriteActionSuite.scala       |  173 ++-
 6 files changed, 1975 insertions(+), 92 deletions(-)

diff --git a/docs/source/contributor-guide/iceberg-writes.md b/docs/source/contributor-guide/iceberg-writes.md
index 98901b474e..763cb88565 100644
--- a/docs/source/contributor-guide/iceberg-writes.md
+++ b/docs/source/contributor-guide/iceberg-writes.md
@@ -205,12 +205,14 @@ columns.
 ### Native execution
 
 The planner builds `IcebergWriteExec` (`native/core/src/execution/operators/iceberg_write.rs`) over
-the FFI scan of the child's batches. `run_write_task` builds iceberg-rust's writer stack once per
-task:
+the FFI scan of the child's batches. `run_write_task` builds iceberg-rust's partitioning writer once
+per task, and `PartitionWriterBuilder` builds the rest of the stack once per partition, with that
+partition's Parquet writer properties:
 
 ```text
-ParquetWriterBuilder -> RollingFileWriterBuilder -> DataFileWriterBuilder
-  -> UnpartitionedWriter | FanoutWriter | ClusteredWriter
+UnpartitionedWriter | FanoutWriter | ClusteredWriter
+  -> PartitionWriterBuilder, per partition:
+       ParquetWriterBuilder -> RollingFileWriterBuilder -> DataFileWriterBuilder
 ```
 
 Points where Comet adapts iceberg-rust to match iceberg-java:
@@ -239,6 +241,19 @@ Points where Comet adapts iceberg-rust to match iceberg-java:
 - **Row pacing.** iceberg-java's rolling writer checks the target file size every 1000 rows of the
   current file. iceberg-rust checks once per `write` call. `RowPacer` hands the writer rows in
   `ROWS_DIVISOR` (1000) row units per file, so both writers roll on the same grid.
+- **Dictionary choice.** parquet-mr writes a column chunk plain, with no dictionary page, when its
+  first data page shows the dictionary saving no space. parquet-rs has no such check, and fixes a
+  column's encoding when the file opens
+  ([#6114](https://github.com/apache/datafusion-comet/issues/6114)). So each partition's
+  `PartitionFeed` holds back the partition's first page of rows (`write.parquet.page-row-limit`,
+  capped by `write.parquet.row-group-size-bytes`, which the fanout feeds share) before any of them
+  reaches the writer. `DictionaryChooser` (`iceberg_dictionary.rs`) replays parquet-mr's accounting
+  over those rows and turns dictionary encoding off for the columns parquet-mr would write plain.
+  It reads only the columns there is a choice for, one at a time, and walks each lazily only as far
+  as its first page, so beyond the held rows it keeps no more than one column's dictionary. The
+  accounting follows `FallbackValuesWriter`, `DictionaryValuesWriter` and
+  `RunLengthBitPackingHybridEncoder`, and its tests pin answers recorded from parquet-mr itself. The
+  held rows then go through `RowPacer`, so the roll grid does not move.
 - **Field ids and casting.** `decorate_batch_with_field_ids` casts each batch to the
   field-id-annotated Arrow schema derived from the Iceberg schema, with `safe: false`, so a type
   mismatch fails the task instead of writing NULLs.
@@ -351,6 +366,7 @@ the writer: run the write suites and the Iceberg Spark tests. The pin policy is
 | `CometIcebergRewriteActionSuite`                                 | Iceberg's `rewrite_data_files` with the split plan and the native writer.                                                                                                                   |
 | `IcebergWriteProtoTranslationSuite`                              | Translation of properties into `IcebergParquetWriteSettings` and the writer mode.                                                                                                           |
 | Rust tests in `iceberg_write.rs` and in `iceberg_partition_*.rs` | File rolling on the 1000-row grid, fanout order, clustered input checks, cleanup guard, manifest round trip, partition path rendering, partition values past `chrono`'s calendar.           |
+| Rust tests in `iceberg_dictionary.rs`                            | The per-column dictionary choice against answers recorded from parquet-mr, including columns either side of its cut-off.                                                                    |
 | `CometIcebergWriteBenchmark`                                     | Native versus iceberg-java for unpartitioned, clustered, fanout and copy-on-write delete writes. It checks each arm's plan before timing it.                                                |
 
 The Comet suites run against the Iceberg version each Spark profile pins in `spark/pom.xml`: 1.5.2
diff --git a/docs/source/user-guide/latest/iceberg-writes.md b/docs/source/user-guide/latest/iceberg-writes.md
index 0b340dbbe2..21da79869f 100644
--- a/docs/source/user-guide/latest/iceberg-writes.md
+++ b/docs/source/user-guide/latest/iceberg-writes.md
@@ -67,6 +67,12 @@ the same `TaskCommit` message the JVM writer would have produced. Everything ice
 post-write — snapshot assignment, manifest-list aggregation, commit validation and retries —
 is untouched: `IcebergCommit` performs the normal `BatchWrite.commit`.
 
+Before a task opens a partition's first file, it holds that partition's first
+`write.parquet.page-row-limit` rows in memory, so that it can choose which columns to
+dictionary-encode the way iceberg-java would (see the accepted divergences below). The rows a
+task holds back this way, across all of its partitions, stay within about
+`write.parquet.row-group-size-bytes`.
+
 ## Configuration
 
 Standard Comet + Iceberg setup (see [`iceberg.md`](iceberg.md)) plus the write-side toggle:
@@ -299,14 +305,19 @@ a data file but not what any reader computes from it:
 - Dictionary-encoded pages are labeled `RLE_DICTIONARY` (parquet-mr v1 files: `PLAIN_DICTIONARY`).
 - Fixed-length binary columns (`uuid`, `fixed`, decimals with precision > 18) are not
   dictionary-encoded (parquet-mr dictionary-encodes them).
-- High-cardinality columns keep a dictionary page. parquet-mr abandons dictionary encoding for a
-  column chunk, and writes no dictionary page, when the first check shows the dictionary is not
-  saving space. parquet-rs keeps dictionary encoding until the dictionary reaches
-  `write.parquet.dict-size-bytes` (2 MB by default), then switches to plain encoding for the rest
-  of the chunk and still writes the dictionary page. Results are the same, but a selective read of
-  a native-written file fetches that dictionary page for every column chunk it touches, so it
-  reads more bytes than it would from an iceberg-java file
-  ([#6114](https://github.com/apache/datafusion-comet/issues/6114)).
+- Which columns are dictionary-encoded is decided as parquet-mr decides it: a column whose first
+  data page shows the dictionary saving no space is written plain, with no dictionary page,
+  instead of carrying a dictionary page that every selective read of it would have to fetch
+  ([#6114](https://github.com/apache/datafusion-comet/issues/6114)). The native writer decides
+  once per partition, from that partition's first page of rows in the task, and keeps the
+  decision for every file and row group it writes for the partition; parquet-mr decides again
+  for every row group. Where the page size rather than `write.parquet.page-row-limit` ends a
+  column's first page, the native page ends at the first row past parquet-mr's size threshold,
+  while parquet-mr only ends it at its next periodic size check, so a column close to the
+  cut-off can be decided the other way. Close to the cut-off both encodings take about the same
+  space. A column that keeps its dictionary and later fills it falls back to plain on both
+  writers, but parquet-rs's dictionary page then holds every entry, where parquet-mr's holds
+  only the entries earlier pages used.
 - Row-group boundaries: parquet-mr flushes by byte size at a record-count check cadence,
   parquet-rs buffers by row count. File naming follows the same cadence-style difference
   (iceberg-java names files `<partition>-<task>-<operation>-<count>`; iceberg-rust uses a
diff --git a/native/core/src/execution/operators/iceberg_dictionary.rs b/native/core/src/execution/operators/iceberg_dictionary.rs
new file mode 100644
index 0000000000..0d6dc8165e
--- /dev/null
+++ b/native/core/src/execution/operators/iceberg_dictionary.rs
@@ -0,0 +1,1277 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Which columns of a native Iceberg data file are dictionary-encoded.
+//!
+//! iceberg-java writes Parquet through parquet-mr, which judges dictionary encoding on the first
+//! data page of every column chunk (`FallbackValuesWriter`): unless the page's dictionary-encoded
+//! values plus its dictionary come out smaller than the page's plain encoding, the chunk is written
+//! plain from its first page on and has no dictionary page. parquet-rs never makes that
+//! comparison. It keeps a column dictionary-encoded until the dictionary reaches
+//! `dictionary_page_size_limit`, then falls back to plain for the rest of the chunk but still
+//! writes the dictionary page. A high-cardinality column therefore carries a dictionary page of up
+//! to `write.parquet.dict-size-bytes` that every selective read of the chunk has to fetch, where
+//! iceberg-java writes none (apache/datafusion-comet#6114).
+//!
+//! parquet-rs cannot change a column's encoding once a file is open, so the native writer takes
+//! parquet-mr's decision before opening one: [`DictionaryChooser`] replays parquet-mr's size
+//! accounting over the rows each column's first data page would hold, and turns dictionary encoding
+//! off for the columns parquet-mr would write plain. The accounting follows `FallbackValuesWriter`,
+//! `DictionaryValuesWriter` and `RunLengthBitPackingHybridEncoder` in parquet-column, and the page
+//! boundary follows `ColumnWriteStoreBase.sizeCheck`.
+
+use arrow::array::{downcast_primitive_array, Array, AsArray, OffsetSizeTrait, RecordBatch};
+use arrow::datatypes::{DataType, Fields, Schema as ArrowSchema, ToByteSlice};
+use datafusion::common::HashSet;
+use datafusion::error::{DataFusionError, Result as DFResult};
+use parquet::arrow::ArrowSchemaConverter;
+use parquet::basic::Type as PhysicalType;
+use parquet::file::properties::WriterProperties;
+use parquet::schema::types::ColumnPath;
+
+/// The row count of parquet-mr's first page size check. Iceberg passes
+/// `write.parquet.row-group-check-min-record-count` to parquet-mr as the minimum row count between
+/// page size checks, and the native writer declines tables that change it from this default, so no
+/// first page ends before this row even when `write.parquet.page-row-limit` is lower.
+const FIRST_PAGE_SIZE_CHECK_ROWS: usize = 100;
+
+/// parquet-mr ends a page once less than this share of the page size is left
+/// (`ColumnWriteStoreBase.THRESHOLD_TOLERANCE_RATIO`).
+const PAGE_SIZE_TOLERANCE_RATIO: f32 = 0.1;
+
+/// Decides, from the rows a partition's data starts with, which of its columns keep dictionary
+/// encoding.
+///
+/// The choice is made once per partition and applies to every file and row group the task writes
+/// for it, where parquet-mr decides again for every column chunk. A column's first data page is
+/// the table's `write.parquet.page-row-limit` rows unless the page size ends it sooner. parquet-mr
+/// only notices a full page at its periodic size checks, whose timing depends on every column, so
+/// where the page size decides, this ends the page at the first row past parquet-mr's threshold and
+/// can judge a column on somewhat fewer rows than parquet-mr would.
+pub(super) struct DictionaryChooser {
+    /// The table's writer properties, which every choice starts from.
+    base: WriterProperties,
+    /// The fields every batch is written with.
+    fields: Fields,
+    /// The leaf columns there is a choice to make for, in Parquet schema order. parquet-rs never
+    /// dictionary-encodes booleans, nor fixed-length byte arrays under Parquet format v1, and a
+    /// column whose dictionary is already off stays off, so none of those is here and their values
+    /// are never read.
+    candidates: Vec<Candidate>,
+    /// Rows the first data page of every column holds at most.
+    page_rows: usize,
+    /// Plain-encoded bytes at which parquet-mr's size check ends a page.
+    page_bytes: u64,
+    /// Memory at which a partition stops holding rows back, however few it has: the row group
+    /// size, since parquet-mr's first page never outlasts its first row group.
+    max_held_bytes: Option<usize>,
+}
+
+/// A leaf column whose dictionary encoding is up for choosing.
+struct Candidate {
+    path: ColumnPath,
+    /// Where the column's values are in a batch.
+    leaf: Leaf,
+    /// Plain-encoded width of one value, or `None` for a byte array, which parquet-mr accounts as
+    /// a 4-byte length plus the bytes.
+    width: Option<u64>,
+    /// `write.parquet.dict-size-bytes`. parquet-mr abandons a dictionary as soon as it grows past
+    /// this, which on a first page leaves no dictionary page at all.
+    dictionary_limit: u64,
+}
+
+/// The way from a batch down to one leaf column: a top-level column, then a step into each nested
+/// level.
+#[derive(Clone)]
+struct Leaf {
+    column: usize,
+    steps: Vec<Step>,
+}
+
+#[derive(Clone, Copy)]
+enum Step {
+    /// Into a struct's field.
+    Field(usize),
+    /// Into a list's elements.
+    Element,
+    /// Into a map's keys.
+    Key,
+    /// Into a map's values.
+    Value,
+}
+
+impl DictionaryChooser {
+    /// `schema` is the Arrow schema every batch is written with, which fixes the Parquet leaf
+    /// columns the same way the parquet writer derives them.
+    pub(super) fn try_new(base: WriterProperties, schema: &ArrowSchema) -> DFResult<Self> {
+        let parquet_schema = ArrowSchemaConverter::new()
+            .with_coerce_types(base.coerce_types())
+            .convert(schema)
+            .map_err(DataFusionError::from)?;
+        let mut leaves = Vec::new();
+        for (column, field) in schema.fields().iter().enumerate() {
+            let mut leaf = Leaf {
+                column,
+                steps: Vec::new(),
+            };
+            collect_leaves(field.data_type(), &mut leaf, &mut leaves);
+        }
+        // Both lists come from the same depth-first walk of the schema, so they line up one to one.
+        // Should they ever not, there is no telling which values belong to which column, and every
+        // column is left as configured.
+        let candidates = if leaves.len() == parquet_schema.num_columns() {
+            parquet_schema
+                .columns()
+                .iter()
+                .zip(leaves)
+                .filter_map(|(descr, leaf)| {
+                    let path = descr.path();
+                    let width = match descr.physical_type() {
+                        PhysicalType::INT32 | PhysicalType::FLOAT => Some(4),
+                        PhysicalType::INT64 | PhysicalType::DOUBLE => Some(8),
+                        PhysicalType::BYTE_ARRAY => None,
+                        // INT96 is not an Iceberg type.
+                        PhysicalType::BOOLEAN
+                        | PhysicalType::FIXED_LEN_BYTE_ARRAY
+                        | PhysicalType::INT96 => return None,
+                    };
+                    base.dictionary_enabled(path).then(|| Candidate {
+                        path: path.clone(),
+                        leaf,
+                        width,
+                        dictionary_limit: base.column_dictionary_page_size_limit(path) as u64,
+                    })
+                })
+                .collect()
+        } else {
+            Vec::new()
+        };
+        let page_size = base.data_page_size_limit();
+        let tolerance = (page_size as f32 * PAGE_SIZE_TOLERANCE_RATIO) as usize;
+        Ok(Self {
+            fields: schema.fields().clone(),
+            candidates,
+            page_rows: base
+                .data_page_row_count_limit()
+                .max(FIRST_PAGE_SIZE_CHECK_ROWS),
+            page_bytes: page_size.saturating_sub(tolerance) as u64,
+            max_held_bytes: base.max_row_group_bytes(),
+            base,
+        })
+    }
+
+    /// The table's writer properties, before any choice.
+    pub(super) fn base(&self) -> &WriterProperties {
+        &self.base
+    }
+
+    /// Whether a partition that has shown `rows` rows, held in `bytes` of memory, should hold back
+    /// more before choosing.
+    pub(super) fn wants_more(&self, rows: usize, bytes: usize) -> bool {
+        !self.candidates.is_empty() && rows < self.page_rows && self.may_hold(bytes)
+    }
+
+    /// Whether rows taking `bytes` of memory may stay held back.
+    pub(super) fn may_hold(&self, bytes: usize) -> bool {
+        self.max_held_bytes.is_none_or(|max| bytes < max)
+    }
+
+    /// The writer properties for a partition whose rows start with `sample`: the table's, with
+    /// dictionary encoding turned off for every column parquet-mr would write plain.
+    ///
+    /// Columns are judged one at a time, and each is read only as far as its first page reaches,
+    /// so what this keeps at any moment is one column's dictionary, which cannot outgrow
+    /// `write.parquet.dict-size-bytes` by more than a value.
+    pub(super) fn choose(&self, sample: &[RecordBatch]) -> WriterProperties {
+        if sample
+            .iter()
+            .any(|batch| batch.schema_ref().fields() != &self.fields)
+        {
+            // There is no telling which values belong to which column.
+            return self.base.clone();
+        }
+        let mut plain = Vec::new();
+        for candidate in &self.candidates {
+            let mut page = FirstPage::new(candidate, self.page_bytes);
+            for batch in sample {
+                if page.is_closed() || page.rows == self.page_rows {
+                    break;
+                }
+                let rows = batch.num_rows().min(self.page_rows - page.rows);
+                let top: Entries = Box::new((0..rows).map(|row| (row as u32, row)));
+                let Some((array, entries)) = candidate.leaf.entries(batch, top) else {
+                    // Cannot happen once the fields match, but a guess is worse than no choice.
+                    return self.base.clone();
+                };
+                page.extend(array, entries, rows);
+            }
+            if page.rows > 0 && !page.keeps_dictionary() {
+                plain.push(&candidate.path);
+            }
+        }
+        if plain.is_empty() {
+            return self.base.clone();
+        }
+        plain
+            .into_iter()
+            .fold(self.base.clone().into_builder(), |builder, path| {
+                builder.set_column_dictionary_enabled(path.clone(), false)
+            })
+            .build()
+    }
+}
+
+/// Appends the way to every leaf under a column of `data_type` that `leaf` leads to, in the
+/// depth-first order Parquet numbers leaf columns in.
+fn collect_leaves(data_type: &DataType, leaf: &mut Leaf, leaves: &mut Vec<Leaf>) {
+    match data_type {
+        DataType::Struct(fields) => {
+            for (field, child) in fields.iter().enumerate() {
+                descend(Step::Field(field), child.data_type(), leaf, leaves);
+            }
+        }
+        DataType::List(element)
+        | DataType::LargeList(element)
+        | DataType::FixedSizeList(element, _) => {
+            descend(Step::Element, element.data_type(), leaf, leaves)
+        }
+        DataType::Map(entries, _) => match entries.data_type() {
+            DataType::Struct(fields) if fields.len() == 2 => {
+                descend(Step::Key, fields[0].data_type(), leaf, leaves);
+                descend(Step::Value, fields[1].data_type(), leaf, leaves);
+            }
+            _ => leaves.push(leaf.clone()),
+        },
+        _ => leaves.push(leaf.clone()),
+    }
+}
+
+fn descend(step: Step, data_type: &DataType, leaf: &mut Leaf, leaves: &mut Vec<Leaf>) {
+    leaf.steps.push(step);
+    collect_leaves(data_type, leaf, leaves);
+    leaf.steps.pop();
+}
+
+/// `(row, index)` pairs, one per entry of an array that the column chunk stores: its row of the
+/// batch, and its index into the array.
+type Entries<'a> = Box<dyn Iterator<Item = (u32, usize)> + 'a>;
+
+impl Leaf {
+    /// The leaf's array in `batch`, with the entries of it that lie under the rows `top` yields and
+    /// whose ancestors are all present, in the order the column chunk stores them. Nothing is
+    /// expanded ahead: each entry is produced when the page asks for it. `None` when `batch` does
+    /// not have the shape the schema promised.
+    fn entries<'a>(
+        &self,
+        batch: &'a RecordBatch,
+        top: Entries<'a>,
+    ) -> Option<(&'a dyn Array, Entries<'a>)> {
+        let mut array = batch.columns().get(self.column)?.as_ref();
+        let mut entries = top;
+        for step in &self.steps {
+            let present = present(array, entries);
+            (array, entries) = match (step, array.data_type()) {
+                (Step::Field(field), DataType::Struct(_)) => {
+                    (array.as_struct().columns().get(*field)?.as_ref(), present)
+                }
+                (Step::Element, DataType::List(_)) => {
+                    let list = array.as_list::<i32>();
+                    (
+                        list.values().as_ref(),
+                        children(present, list.value_offsets()),
+                    )
+                }
+                (Step::Element, DataType::LargeList(_)) => {
+                    let list = array.as_list::<i64>();
+                    (
+                        list.values().as_ref(),
+                        children(present, list.value_offsets()),
+                    )
+                }
+                (Step::Element, DataType::FixedSizeList(_, size)) => {
+                    let size = *size as usize;
+                    let elements: Entries<'a> = Box::new(present.flat_map(move |(row, index)| {
+                        (index * size..(index + 1) * size).map(move |element| (row, element))
+                    }));
+                    (array.as_fixed_size_list().values().as_ref(), elements)
+                }
+                (Step::Key, DataType::Map(_, _)) => {
+                    let map = array.as_map();
+                    (map.keys().as_ref(), children(present, map.value_offsets()))
+                }
+                (Step::Value, DataType::Map(_, _)) => {
+                    let map = array.as_map();
+                    (
+                        map.values().as_ref(),
+                        children(present, map.value_offsets()),
+                    )
+                }
+                _ => return None,
+            };
+        }
+        Some((array, entries))
+    }
+}
+
+/// The entries whose value in `array` is not null.
+fn present<'a>(array: &'a dyn Array, entries: Entries<'a>) -> Entries<'a> {
+    match array.nulls() {
+        Some(nulls) => Box::new(entries.filter(move |&(_, index)| nulls.is_valid(index))),
+        None => entries,
+    }
+}
+
+/// The entries of the children of every list or map entry in `entries`.
+fn children<'a, O: OffsetSizeTrait>(entries: Entries<'a>, offsets: &'a [O]) -> Entries<'a> {
+    Box::new(entries.flat_map(move |(row, index)| {
+        (offsets[index].as_usize()..offsets[index + 1].as_usize()).map(move |child| (row, child))
+    }))
+}
+
+type ValueBytes<'a> = Box<dyn Fn(usize) -> &'a [u8] + 'a>;
+
+/// The bytes that tell `array`'s values apart: a fixed-width value's native bytes, or a string's or
+/// binary's content. `None` for a type an Iceberg schema cannot produce.
+fn value_bytes<'a>(array: &'a dyn Array) -> Option<ValueBytes<'a>> {
+    let bytes: ValueBytes<'a> = downcast_primitive_array!(
+        array => Box::new(move |i| array.values()[i].to_byte_slice()),
+        DataType::Utf8 => {
+            let array = array.as_string::<i32>();
+            Box::new(move |i| array.value(i).as_bytes())
+        }
+        DataType::LargeUtf8 => {
+            let array = array.as_string::<i64>();
+            Box::new(move |i| array.value(i).as_bytes())
+        }
+        DataType::Utf8View => {
+            let array = array.as_string_view();
+            Box::new(move |i| array.value(i).as_bytes())
+        }
+        DataType::Binary => {
+            let array = array.as_binary::<i32>();
+            Box::new(move |i| array.value(i))
+        }
+        DataType::LargeBinary => {
+            let array = array.as_binary::<i64>();
+            Box::new(move |i| array.value(i))
+        }
+        DataType::BinaryView => {
+            let array = array.as_binary_view();
+            Box::new(move |i| array.value(i))
+        }
+        _ => return None,
+    );
+    Some(bytes)
+}
+
+/// What parquet-mr's dictionary writer has seen of one column chunk's first data page.
+struct FirstPage<'a> {
+    width: Option<u64>,
+    dictionary_limit: u64,
+    page_bytes: u64,
+    /// Every value seen so far, once each: the dictionary.
+    dictionary: HashSet<&'a [u8]>,
+    /// The value before the one being added.
+    previous: Option<&'a [u8]>,
+    /// The runs the page's dictionary ids would be encoded in.
+    ids: HybridRuns,
+    plain_bytes: u64,
+    dictionary_bytes: u64,
+    /// Rows the page has taken so far.
+    rows: usize,
+    /// The page ended at a size check.
+    full: bool,
+    /// The dictionary outgrew `dictionary_limit` before the page ended.
+    overflowed: bool,
+    /// The column's arrays are of a type there is no reading, so it is left as configured.
+    unreadable: bool,
+}
+
+impl<'a> FirstPage<'a> {
+    fn new(candidate: &Candidate, page_bytes: u64) -> Self {
+        Self {
+            width: candidate.width,
+            dictionary_limit: candidate.dictionary_limit,
+            page_bytes,
+            dictionary: HashSet::default(),
+            previous: None,
+            ids: HybridRuns::default(),
+            plain_bytes: 0,
+            dictionary_bytes: 0,
+            rows: 0,
+            full: false,
+            overflowed: false,
+            unreadable: false,
+        }
+    }
+
+    fn is_closed(&self) -> bool {
+        self.full || self.overflowed || self.unreadable
+    }
+
+    /// Adds `rows` rows of one batch, given as the entries of the leaf's `array` they hold. Stops
+    /// taking entries as soon as the page ends.
+    fn extend(
+        &mut self,
+        array: &'a dyn Array,
+        entries: impl Iterator<Item = (u32, usize)>,
+        rows: usize,
+    ) {
+        if self.is_closed() {
+            return;
+        }
+        let Some(value) = value_bytes(array) else {
+            self.unreadable = true;
+            return;
+        };
+        let nulls = array.nulls();
+        let mut entries = entries.peekable();
+        for row in 0..rows as u32 {
+            while let Some((_, index)) = entries.next_if(|&(entry_row, _)| entry_row == row) {
+                if nulls.is_none_or(|nulls| nulls.is_valid(index)) {
+                    self.add(value(index));
+                    if self.overflowed {
+                        return;
+                    }
+                }
+            }
+            self.rows += 1;
+            if self.rows >= FIRST_PAGE_SIZE_CHECK_ROWS && self.plain_bytes >= self.page_bytes {
+                self.full = true;
+                return;
+            }
+        }
+    }
+
+    /// `DictionaryValuesWriter.write*` and `FallbackValuesWriter`'s raw size, for one value.
+    fn add(&mut self, value: &'a [u8]) {
+        let size = self.width.unwrap_or(4 + value.len() as u64);
+        self.plain_bytes += size;
+        if self.dictionary.insert(value) {
+            self.dictionary_bytes += size;
+        }
+        // Ids are handed out in order of first appearance, so an id repeats the one before it
+        // exactly when its value does.
+        self.ids.write(self.previous == Some(value));
+        self.previous = Some(value);
+        // `DictionaryValuesWriter.shouldFallBack`, checked after every value.
+        if self.dictionary_bytes > self.dictionary_limit {
+            self.overflowed = true;
+        }
+    }
+
+    /// `FallbackValuesWriter.getBytes` on the first page: the dictionary stays only if the page's
+    /// dictionary-encoded bytes (a bit-width byte, then the ids) plus the dictionary are fewer than
+    /// the page's plain bytes (`DictionaryValuesWriter.isCompressionSatisfying`). A page of nulls
+    /// only is plain, since nothing is fewer than zero bytes.
+    fn keeps_dictionary(&self) -> bool {
+        if self.unreadable {
+            return true;
+        }
+        if self.overflowed {
+            return false;
+        }
+        let max_id = self.dictionary.len().saturating_sub(1) as u32;
+        let bit_width = u32::BITS - max_id.leading_zeros();
+        1 + self.ids.len(bit_width) + self.dictionary_bytes < self.plain_bytes
+    }
+}
+
+/// `RunLengthBitPackingHybridEncoder`'s run-splitting state machine, counting what it would write
+/// instead of writing it. A value repeated 8 or more times from the start of a group of 8 becomes
+/// an RLE run; everything else is bit-packed 8 values at a time, up to 63 groups per run header.
+///
+/// Where the encoder cuts runs depends only on whether each id repeats the one before it, so that
+/// is all this is told. The bit width, which the dictionary's final size decides, only comes in
+/// when the length is read off.
+#[derive(Clone, Default)]
+struct HybridRuns {
+    repeat_count: u64,
+    buffered: u32,
+    /// Groups in the bit-packed run that is open.
+    groups: u32,
+    bit_packed_run_open: bool,
+    /// One header byte each.
+    bit_packed_runs: u64,
+    /// `bit_width` bytes each: eight values of `bit_width` bits.
+    bit_packed_groups: u64,
+    /// The value, padded to whole bytes, each.
+    rle_runs: u64,
+    /// The varint headers of the RLE runs, which hold the run lengths.
+    rle_header_bytes: u64,
+}
+
+impl HybridRuns {
+    /// `writeInt` for an id that does or does not repeat the one before it. The encoder starts from
+    /// a previous value of 0, which is always the first id, and either way a first value starts a
+    /// run of one.
+    fn write(&mut self, repeat: bool) {
+        if repeat {
+            self.repeat_count += 1;
+            if self.repeat_count >= 8 {
+                return;
+            }
+        } else {
+            if self.repeat_count >= 8 {
+                self.write_rle_run();
+            }
+            self.repeat_count = 1;
+        }
+        self.buffered += 1;
+        if self.buffered == 8 {
+            self.write_bit_packed_group();
+        }
+    }
+
+    fn write_bit_packed_group(&mut self) {
+        if self.groups >= 63 {
+            self.end_bit_packed_run();
+        }
+        if !self.bit_packed_run_open {
+            self.bit_packed_runs += 1;
+            self.bit_packed_run_open = true;
+        }
+        self.bit_packed_groups += 1;
+        self.buffered = 0;
+        self.repeat_count = 0;
+        self.groups += 1;
+    }
+
+    fn end_bit_packed_run(&mut self) {
+        self.bit_packed_run_open = false;
+        self.groups = 0;
+    }
+
+    fn write_rle_run(&mut self) {
+        self.end_bit_packed_run();
+        self.rle_runs += 1;
+        self.rle_header_bytes += varint_len(self.repeat_count << 1);
+        self.repeat_count = 0;
+        self.buffered = 0;
+    }
+
+    /// The bytes `toBytes()` would return at `bit_width`, including its flush of what is still
+    /// buffered, whose last group is padded to 8 values.
+    fn len(&self, bit_width: u32) -> u64 {
+        let mut finished = self.clone();
+        if finished.repeat_count >= 8 {
+            finished.write_rle_run();
+        } else if finished.buffered > 0 {
+            finished.write_bit_packed_group();
+        }
+        let bit_width = bit_width as u64;
+        finished.bit_packed_runs
+            + finished.bit_packed_groups * bit_width
+            + finished.rle_header_bytes
+            + finished.rle_runs * bit_width.div_ceil(8)
+    }
+}
+
+fn varint_len(value: u64) -> u64 {
+    (u64::BITS - value.leading_zeros()).div_ceil(7).max(1) as u64
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::cell::Cell;
+    use std::sync::Arc;
+
+    use arrow::array::builder::{
+        Int32Builder, Int64Builder, ListBuilder, MapBuilder, MapFieldNames, StringBuilder,
+    };
+    use arrow::array::{
+        ArrayRef, BooleanArray, Date32Array, Decimal128Array, FixedSizeBinaryArray, Float32Array,
+        Float64Array, Int32Array, Int64Array, ListArray, StringArray, StructArray,
+    };
+    use arrow::buffer::{NullBuffer, OffsetBuffer};
+    use arrow::datatypes::{Field, Fields};
+
+    /// splitmix64, the generator the parquet-mr ground truth below was recorded with.
+    fn mix(x: u64) -> u64 {
+        let mut x = x.wrapping_add(0x9E37_79B9_7F4A_7C15);
+        x = (x ^ (x >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
+        x = (x ^ (x >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
+        x ^ (x >> 31)
+    }
+
+    const CARDINALITIES: [u64; 14] = [
+        1000, 2780, 5000, 12000, 13000, 14000, 14500, 15000, 15500, 16000, 17000, 20000, 30000,
+        38000,
+    ];
+    const CYCLE_UNIQUE: [u64; 9] = [60, 65, 68, 69, 70, 71, 72, 75, 80];
+    const WIDE_CARDINALITIES: [u64; 4] = [3000, 5000, 7000, 9000];
+
+    /// Rows `rows` of a table built to sit on both sides of parquet-mr's cut-off: random columns
+    /// of rising cardinality in each physical type, runs, cycles that mix repeats with unique
+    /// values, nulls, wide strings whose first page the page size ends, and nested leaves.
+    fn corpus(rows: std::ops::Range<u64>) -> RecordBatch {
+        let ints = |f: &dyn Fn(u64) -> i32| -> ArrayRef {
+            Arc::new(Int32Array::from_iter_values(rows.clone().map(f)))
+        };
+        let longs = |f: &dyn Fn(u64) -> i64| -> ArrayRef {
+            Arc::new(Int64Array::from_iter_values(rows.clone().map(f)))
+        };
+        let strings = |f: &dyn Fn(u64) -> String| -> ArrayRef {
+            Arc::new(StringArray::from_iter_values(rows.clone().map(f)))
+        };
+        let mut columns: Vec<(String, ArrayRef)> = vec![
+            ("unique_long".into(), longs(&|i| i as i64)),
+            ("mod10_int".into(), ints(&|i| (i % 10) as i32)),
+        ];
+        for c in CARDINALITIES {
+            columns.push((format!("rand_int_{c}"), ints(&|i| (mix(i) % c) as i32)));
+        }
+        for c in CARDINALITIES {
+            columns.push((
+                format!("rand_long_{c}"),
+                longs(&|i| (mix(i + 7) % c) as i64),
+            ));
+        }
+        for c in CARDINALITIES {
+            columns.push((
+                format!("rand_str_{c}"),
+                strings(&|i| format!("s{}", mix(i + 13) % c)),
+            ));
+        }
+        columns.push((
+            "unique_str".into(),
+            strings(&|i| format!("{:016x}", mix(i))),
+        ));
+        columns.push((
+            "all_null".into(),
+            Arc::new(Int32Array::from_iter(rows.clone().map(|_| None::<i32>))),
+        ));
+        columns.push((
+            "half_null_unique".into(),
+            Arc::new(Int64Array::from_iter(
+                rows.clone().map(|i| (i % 2 == 0).then_some(i as i64)),
+            )),
+        ));
+        columns.push(("runs_of_3".into(), longs(&|i| (i / 3) as i64)));
+        columns.push(("runs_of_10".into(), ints(&|i| (i / 10) as i32)));
+        for u in CYCLE_UNIQUE {
+            columns.push((
+                format!("cycle_unique_{u}"),
+                ints(&|i| if i % 100 < u { (1 + i) as i32 } else { 0 }),
+            ));
+        }
+        columns.push((
+            "unique_double".into(),
+            Arc::new(Float64Array::from_iter_values(
+                rows.clone().map(|i| f64::from_bits(mix(i) >> 2)),
+            )),
+        ));
+        columns.push((
+            "low_float".into(),
+            Arc::new(Float32Array::from_iter_values(
+                rows.clone().map(|i| (i % 100) as f32),
+            )),
+        ));
+        columns.push((
+            "wide_unique_str".into(),
+            strings(&|i| format!("{:016x}", mix(i + 99)).repeat(12)),
+        ));
+        columns.push((
+            "wide_low_str".into(),
+            strings(&|i| format!("{}{}", "x".repeat(200), i % 20)),
+        ));
+        for c in WIDE_CARDINALITIES {
+            columns.push((
+                format!("wide_rand_str_{c}"),
+                strings(&|i| format!("{}{}", "y".repeat(120), mix(i + 31) % c)),
+            ));
+        }
+        columns.push((
+            "bool".into(),
+            Arc::new(BooleanArray::from_iter(
+                rows.clone().map(|i| Some(i % 3 == 0)),
+            )),
+        ));
+        columns.push((
+            "dec_low".into(),
+            Arc::new(
+                Decimal128Array::from_iter_values(rows.clone().map(|i| (i % 1000) as i128))
+                    .with_precision_and_scale(10, 2)
+                    .unwrap(),
+            ),
+        ));
+        columns.push((
+            "date_low".into(),
+            Arc::new(Date32Array::from_iter_values(
+                rows.clone().map(|i| 18000 + (i % 365) as i32),
+            )),
+        ));
+        let struct_fields = Fields::from(vec![
+            Field::new("a", DataType::Int64, true),
+            Field::new("b", DataType::Int32, true),
+        ]);
+        columns.push((
+            "st".into(),
+            Arc::new(StructArray::new(
+                struct_fields,
+                vec![longs(&|i| mix(i + 5) as i64), ints(&|i| (i % 7) as i32)],
+                None,
+            )),
+        ));
+        let element = |data_type| Arc::new(Field::new("element", data_type, true));
+        let mut list_low =
+            ListBuilder::new(Int32Builder::new()).with_field(element(DataType::Int32));
+        let mut list_unique =
+            ListBuilder::new(Int64Builder::new()).with_field(element(DataType::Int64));
+        let mut map = MapBuilder::new(
+            Some(MapFieldNames {
+                entry: "key_value".into(),
+                key: "key".into(),
+                value: "value".into(),
+            }),
+            StringBuilder::new(),
+            Int64Builder::new(),
+        );
+        for i in rows.clone() {
+            list_low.values().append_value((i % 5) as i32);
+            list_low.values().append_value((i % 3) as i32);
+            list_low.values().append_null();
+            list_low.append(true);
+            for k in 0..3 {
+                list_unique.values().append_value((3 * i + k) as i64);
+            }
+            list_unique.append(true);
+            map.keys().append_value(format!("k{}", i % 4));
+            map.values().append_value(mix(i + 11) as i64);
+            map.keys().append_value(format!("z{}", i % 2));
+            map.values().append_value((i % 9) as i64);
+            map.append(true).unwrap();
+        }
+        columns.push(("list_low".into(), Arc::new(list_low.finish())));
+        columns.push(("list_unique".into(), Arc::new(list_unique.finish())));
+        columns.push(("m".into(), Arc::new(map.finish())));
+        RecordBatch::try_from_iter(columns).unwrap()
+    }
+
+    /// `rows` rows of [`corpus`] in batches of `batch_rows`.
+    fn corpus_batches(rows: u64, batch_rows: u64) -> Vec<RecordBatch> {
+        (0..rows)
+            .step_by(batch_rows as usize)
+            .map(|start| corpus(start..(start + batch_rows).min(rows)))
+            .collect()
+    }
+
+    fn properties(page_rows: usize, page_size: usize, dictionary_limit: usize) -> WriterProperties {
+        WriterProperties::builder()
+            .set_data_page_row_count_limit(page_rows)
+            .set_data_page_size_limit(page_size)
+            .set_dictionary_page_size_limit(dictionary_limit)
+            .set_max_row_group_bytes(Some(128 * 1024 * 1024))
+            .set_max_row_group_row_count(None)
+            .build()
+    }
+
+    /// The leaf columns that keep dictionary encoding under the properties chosen for `sample`,
+    /// leaving out the types parquet-rs never dictionary-encodes.
+    fn dictionary_columns(properties: WriterProperties, sample: &[RecordBatch]) -> Vec<String> {
+        let schema = sample[0].schema();
+        let chosen = DictionaryChooser::try_new(properties, &schema)
+            .unwrap()
+            .choose(sample);
+        ArrowSchemaConverter::new()
+            .convert(&schema)
+            .unwrap()
+            .columns()
+            .iter()
+            .filter(|descr| {
+                !matches!(
+                    descr.physical_type(),
+                    PhysicalType::BOOLEAN | PhysicalType::FIXED_LEN_BYTE_ARRAY
+                ) && chosen.dictionary_enabled(descr.path())
+            })
+            .map(|descr| descr.path().string())
+            .collect()
+    }
+
+    // The expected columns below are parquet-mr's own answers. They were recorded by writing the
+    // same rows through iceberg-java's `Parquet.write` with `GenericParquetWriter`, with the same
+    // `write.parquet.page-row-limit`, `page-size-bytes` and `dict-size-bytes`, and listing the
+    // column chunks of the first row group whose encoding stats include a dictionary page. Iceberg
+    // 1.5.2, 1.8.1, 1.10.0 and 1.11.0 (parquet-mr 1.13.1, 1.15.0, 1.16.0 and 1.17.1) all agree.
+
+    #[test]
+    fn matches_parquet_mr_at_iceberg_defaults() {
+        let sample = corpus_batches(24_576, 8192);
+        assert_eq!(
+            dictionary_columns(properties(20_000, 1024 * 1024, 2 * 1024 * 1024), &sample),
+            [
+                "mod10_int",
+                "rand_int_1000",
+                "rand_int_2780",
+                "rand_int_5000",
+                "rand_int_12000",
+                "rand_int_13000",
+                "rand_int_14000",
+                "rand_int_14500",
+                "rand_int_15000",
+                "rand_long_1000",
+                "rand_long_2780",
+                "rand_long_5000",
+                "rand_long_12000",
+                "rand_long_13000",
+                "rand_long_14000",
+                "rand_long_14500",
+                "rand_long_15000",
+                "rand_long_15500",
+                "rand_long_16000",
+                "rand_long_17000",
+                "rand_long_20000",
+                "rand_long_30000",
+                "rand_str_1000",
+                "rand_str_2780",
+                "rand_str_5000",
+                "rand_str_12000",
+                "rand_str_13000",
+                "rand_str_14000",
+                "rand_str_14500",
+                "rand_str_15000",
+                "rand_str_15500",
+                "rand_str_16000",
+                "rand_str_17000",
+                "rand_str_20000",
+                "rand_str_30000",
+                "rand_str_38000",
+                "runs_of_3",
+                "runs_of_10",
+                "cycle_unique_60",
+                "cycle_unique_65",
+                "low_float",
+                "wide_low_str",
+                "wide_rand_str_3000",
+                "wide_rand_str_5000",
+                "wide_rand_str_7000",
+                "wide_rand_str_9000",
+                "dec_low",
+                "date_low",
+                "st.b",
+                "list_low.list.element",
+                "m.key_value.key",
+                "m.key_value.value",
+            ]
+        );
+    }
+
+    /// Pages that end at 16 KiB, before the 2000-row limit, for all but the narrowest columns: the
+    /// settings of `CometIcebergNativeSuite`'s page skipping test.
+    #[test]
+    fn matches_parquet_mr_when_the_page_size_ends_the_first_page() {
+        let sample = corpus_batches(8192, 8192);
+        assert_eq!(
+            dictionary_columns(properties(2000, 16 * 1024, 2 * 1024 * 1024), &sample),
+            [
+                "mod10_int",
+                "rand_int_1000",
+                "rand_long_1000",
+                "rand_long_2780",
+                "rand_str_1000",
+                "rand_str_2780",
+                "rand_str_5000",
+                "runs_of_3",
+                "runs_of_10",
+                "cycle_unique_60",
+                "cycle_unique_65",
+                "cycle_unique_68",
+                "cycle_unique_69",
+                "cycle_unique_70",
+                "cycle_unique_71",
+                "cycle_unique_72",
+                "low_float",
+                "wide_low_str",
+                "wide_rand_str_3000",
+                "wide_rand_str_5000",
+                "wide_rand_str_7000",
+                "wide_rand_str_9000",
+                "dec_low",
+                "date_low",
+                "st.b",
+                "list_low.list.element",
+                "m.key_value.key",
+                "m.key_value.value",
+            ]
+        );
+    }
+
+    /// parquet-mr does not check page sizes before row 100, so a lower row limit still gives
+    /// 100-row first pages.
+    #[test]
+    fn matches_parquet_mr_when_the_row_limit_is_below_the_first_size_check() {
+        let sample = corpus_batches(1000, 1000);
+        assert_eq!(
+            dictionary_columns(properties(50, 1024 * 1024, 2 * 1024 * 1024), &sample),
+            [
+                "mod10_int",
+                "runs_of_3",
+                "runs_of_10",
+                "cycle_unique_60",
+                "cycle_unique_65",
+                "cycle_unique_68",
+                "cycle_unique_69",
+                "cycle_unique_70",
+                "cycle_unique_71",
+                "cycle_unique_72",
+                "cycle_unique_75",
+                "cycle_unique_80",
+                "wide_low_str",
+                "wide_rand_str_3000",
+                "wide_rand_str_5000",
+                "wide_rand_str_7000",
+                "wide_rand_str_9000",
+                "st.b",
+                "list_low.list.element",
+                "m.key_value.key",
+                "m.key_value.value",
+            ]
+        );
+    }
+
+    /// A partition with fewer rows than a page is judged on all of them, as parquet-mr judges a
+    /// column chunk that ends within its first page.
+    #[test]
+    fn matches_parquet_mr_when_the_rows_end_within_the_first_page() {
+        let sample = corpus_batches(5000, 2048);
+        assert_eq!(
+            dictionary_columns(properties(20_000, 1024 * 1024, 2 * 1024 * 1024), &sample),
+            [
+                "mod10_int",
+                "rand_int_1000",
+                "rand_int_2780",
+                "rand_long_1000",
+                "rand_long_2780",
+                "rand_long_5000",
+                "rand_str_1000",
+                "rand_str_2780",
+                "rand_str_5000",
+                "rand_str_13000",
+                "runs_of_3",
+                "runs_of_10",
+                "cycle_unique_60",
+                "cycle_unique_65",
+                "cycle_unique_68",
+                "cycle_unique_69",
+                "cycle_unique_70",
+                "cycle_unique_71",
+                "low_float",
+                "wide_low_str",
+                "wide_rand_str_3000",
+                "wide_rand_str_5000",
+                "wide_rand_str_7000",
+                "wide_rand_str_9000",
+                "dec_low",
+                "date_low",
+                "st.b",
+                "list_low.list.element",
+                "m.key_value.key",
+                "m.key_value.value",
+            ]
+        );
+    }
+
+    /// A dictionary that outgrows `write.parquet.dict-size-bytes` on the first page is dropped,
+    /// however much it would have saved.
+    #[test]
+    fn matches_parquet_mr_when_the_dictionary_outgrows_its_limit() {
+        let sample = corpus_batches(24_576, 8192);
+        assert_eq!(
+            dictionary_columns(properties(20_000, 1024 * 1024, 4096), &sample),
+            [
+                "mod10_int",
+                "rand_int_1000",
+                "low_float",
+                "date_low",
+                "st.b",
+                "list_low.list.element",
+                "m.key_value.key",
+            ]
+        );
+    }
+
+    /// Where a partition's rows were cut into batches does not change its choice.
+    #[test]
+    fn batch_boundaries_do_not_change_the_choice() {
+        let properties = || properties(20_000, 1024 * 1024, 2 * 1024 * 1024);
+        let whole = dictionary_columns(properties(), &corpus_batches(20_000, 20_000));
+        for batch_rows in [777, 1000, 8192] {
+            assert_eq!(
+                dictionary_columns(properties(), &corpus_batches(20_000, batch_rows)),
+                whole,
+                "batches of {batch_rows} rows"
+            );
+        }
+    }
+
+    #[test]
+    fn hybrid_len_matches_parquet_mr() {
+        fn sequence(len: usize, value: impl Fn(usize) -> u32) -> Vec<u32> {
+            (0..len).map(value).collect()
+        }
+        // What `RunLengthBitPackingHybridEncoder.toBytes()` returned for the same values, at the
+        // bit width of their maximum.
+        let cases: Vec<(&str, Vec<u32>, u64)> = vec![
+            ("empty", vec![], 0),
+            ("one zero", vec![0], 1),
+            ("20000 zeros", vec![0; 20_000], 3),
+            ("ascending", sequence(20_000, |i| i as u32), 37_540),
+            ("seven ones", vec![1; 7], 2),
+            ("eight ones", vec![1; 8], 2),
+            ("nine ones", vec![1; 9], 2),
+            ("runs of 3", sequence(20_000, |i| (i / 3) as u32), 32_540),
+            ("runs of 8", sequence(20_000, |i| (i / 8) as u32), 7_500),
+            ("runs of 10", sequence(20_000, |i| (i / 10) as u32), 6_000),
+            (
+                "runs of 11 starting mid-group",
+                sequence(20_000, |i| ((i + 3) / 11) as u32),
+                5_466,
+            ),
+            (
+                "random under 5000",
+                sequence(20_000, |i| (mix(i as u64) % 5000) as u32),
+                32_540,
+            ),
+            (
+                "cycles of 70 unique and 30 repeats",
+                sequence(20_000, |i| if i % 100 < 70 { 1 + i as u32 } else { 0 }),
+                27_800,
+            ),
+            (
+                "runs of 9 broken by one value",
+                sequence(1000, |i| if i % 10 < 9 { 5 } else { i as u32 }),
+                1_100,
+            ),
+            (
+                "scattered repeats",
+                sequence(5000, |i| {
+                    if mix(i as u64).is_multiple_of(3) {
+                        i as u32
+                    } else {
+                        7
+                    }
+                }),
+                7_794,
+            ),
+            (
+                "504 ascending, 63 groups in one run",
+                sequence(504, |i| i as u32),
+                568,
+            ),
+            ("505 ascending", sequence(505, |i| i as u32), 578),
+            (
+                "512 ascending, 64 groups in two runs",
+                sequence(512, |i| i as u32),
+                578,
+            ),
+            ("513 ascending", sequence(513, |i| i as u32), 652),
+            ("600 ascending", sequence(600, |i| i as u32), 752),
+        ];
+        for (name, values, expected) in cases {
+            let mut runs = HybridRuns::default();
+            for (i, value) in values.iter().enumerate() {
+                runs.write(i > 0 && values[i - 1] == *value);
+            }
+            let max = values.iter().copied().max().unwrap_or(0);
+            assert_eq!(
+                runs.len(u32::BITS - max.leading_zeros()),
+                expected,
+                "{name}"
+            );
+        }
+    }
+
+    /// The dictionary has to come out strictly smaller. Here the ids (a bit-width byte and one
+    /// bit-packed group, 5 bytes) plus the dictionary (7 values, 35 bytes) exactly match the plain
+    /// page (8 values, 40 bytes), and parquet-mr writes the column plain.
+    #[test]
+    fn a_dictionary_that_only_breaks_even_is_dropped() {
+        let values = StringArray::from(vec!["a", "b", "c", "d", "e", "f", "g", "a"]);
+        let batch = RecordBatch::try_from_iter([("s", Arc::new(values) as ArrayRef)]).unwrap();
+        assert!(dictionary_columns(properties(20_000, 1024 * 1024, 1024), &[batch]).is_empty());
+    }
+
+    #[test]
+    fn wants_rows_until_a_page_or_a_row_group_is_held() {
+        let schema = ArrowSchema::new(vec![Field::new("id", DataType::Int64, false)]);
+        let chooser =
+            DictionaryChooser::try_new(properties(20_000, 1024 * 1024, 1024), &schema).unwrap();
+        assert!(chooser.wants_more(19_999, 1024));
+        assert!(!chooser.wants_more(20_000, 1024));
+        assert!(!chooser.wants_more(10, 128 * 1024 * 1024));
+    }
+
+    /// parquet-rs never dictionary-encodes booleans or fixed-length binary, and a column whose
+    /// dictionary is already off stays off, so none of them is worth holding rows for.
+    #[test]
+    fn holds_no_rows_when_there_is_nothing_to_choose() {
+        let schema = Arc::new(ArrowSchema::new(vec![
+            Field::new("flag", DataType::Boolean, false),
+            Field::new("uuid", DataType::FixedSizeBinary(16), false),
+            Field::new("id", DataType::Int64, false),
+        ]));
+        let properties = properties(20_000, 1024 * 1024, 1024)
+            .into_builder()
+            .set_column_dictionary_enabled(ColumnPath::from("id"), false)
+            .build();
+        let chooser = DictionaryChooser::try_new(properties, &schema).unwrap();
+        assert!(!chooser.wants_more(0, 0));
+        let batch = RecordBatch::try_new(
+            schema,
+            vec![
+                Arc::new(BooleanArray::from(vec![true, false])),
+                Arc::new(
+                    FixedSizeBinaryArray::try_from_iter([[0u8; 16], [1u8; 16]].into_iter())
+                        .unwrap(),
+                ),
+                Arc::new(Int64Array::from(vec![1, 1])),
+            ],
+        )
+        .unwrap();
+        assert!(!chooser
+            .choose(&[batch])
+            .dictionary_enabled(&ColumnPath::from("id")));
+    }
+
+    /// Only the columns there is a choice for are read. The boolean lists here, 1000 rows of 8192
+    /// entries each in the batch below, are never walked, and a schema with nothing else holds no
+    /// rows back and chooses without reading anything.
+    #[test]
+    fn columns_with_no_choice_are_never_read() {
+        let flags = || DataType::List(Arc::new(Field::new("element", DataType::Boolean, true)));
+        let schema = ArrowSchema::new(vec![
+            Field::new("flags", flags(), true),
+            Field::new(
+                "s",
+                DataType::Struct(Fields::from(vec![
+                    Field::new("more_flags", flags(), true),
+                    Field::new("n", DataType::Int64, true),
+                ])),
+                true,
+            ),
+            Field::new("id", DataType::Int64, false),
+        ]);
+        let chooser =
+            DictionaryChooser::try_new(properties(20_000, 1024 * 1024, 1024), &schema).unwrap();
+        let read: Vec<String> = chooser
+            .candidates
+            .iter()
+            .map(|candidate| candidate.path.string())
+            .collect();
+        assert_eq!(read, ["s.n", "id"]);
+
+        let list = ListArray::new(
+            Arc::new(Field::new("element", DataType::Boolean, true)),
+            OffsetBuffer::from_lengths(std::iter::repeat_n(8192, 1000)),
+            Arc::new(BooleanArray::from(vec![true; 8192 * 1000])),
+            None,
+        );
+        let batch = RecordBatch::try_from_iter([("flags", Arc::new(list) as ArrayRef)]).unwrap();
+        let chooser =
+            DictionaryChooser::try_new(properties(20_000, 1024 * 1024, 1024), batch.schema_ref())
+                .unwrap();
+        assert!(chooser.candidates.is_empty());
+        assert!(!chooser.wants_more(0, 0));
+        assert!(chooser
+            .choose(&[batch])
+            .dictionary_enabled(&ColumnPath::from("flags")));
+    }
+
+    /// A column is read only as far as its first page reaches, one entry at a time. Each row here
+    /// lists 8192 ints, so the page is full at row 100, parquet-mr's first size check, and the
+    /// entries of the other 300 rows are never produced.
+    #[test]
+    fn a_column_is_read_only_as_far_as_its_first_page() {
+        let rows = 400;
+        let list = ListArray::new(
+            Arc::new(Field::new("element", DataType::Int32, true)),
+            OffsetBuffer::from_lengths(std::iter::repeat_n(8192, rows)),
+            Arc::new(Int32Array::from(vec![0; 8192 * rows])),
+            None,
+        );
+        let batch = RecordBatch::try_from_iter([("l", Arc::new(list) as ArrayRef)]).unwrap();
+        let chooser = DictionaryChooser::try_new(
+            properties(20_000, 1024 * 1024, 2 * 1024 * 1024),
+            batch.schema_ref(),
+        )
+        .unwrap();
+        let candidate = &chooser.candidates[0];
+        let rows_walked = Cell::new(0);
+        let top: Entries = Box::new((0..rows).map(|row| {
+            rows_walked.set(rows_walked.get() + 1);
+            (row as u32, row)
+        }));
+        let (array, entries) = candidate.leaf.entries(&batch, top).unwrap();
+        let mut page = FirstPage::new(candidate, chooser.page_bytes);
+        page.extend(array, entries, rows);
+        assert!(page.full);
+        assert_eq!(page.rows, 100);
+        // The 100 rows the page took, and the next, whose first entry told it row 100 was done.
+        assert_eq!(rows_walked.get(), 101);
+    }
+
+    /// A null list can still span child values. They are not part of the column chunk, so they
+    /// must not count toward its first page.
+    #[test]
+    fn values_under_a_null_list_are_not_counted() {
+        // Row 0 is a null list spanning 1000 unique values, which would overflow the 1 KiB
+        // dictionary if they were counted. Rows 1 to 1000 hold one 7 each, so the page the column
+        // chunk actually has is 1000 repeats of one value.
+        let mut offsets = vec![0i32, 1000];
+        offsets.extend((1..=1000).map(|row| 1000 + row));
+        let list = ListArray::new(
+            Arc::new(Field::new("element", DataType::Int64, true)),
+            OffsetBuffer::new(offsets.into()),
+            Arc::new(Int64Array::from_iter_values((0..2000).map(|i| {
+                if i < 1000 {
+                    i
+                } else {
+                    7
+                }
+            }))),
+            Some(NullBuffer::from_iter((0..=1000).map(|row| row != 0))),
+        );
+        let batch = RecordBatch::try_from_iter([("l", Arc::new(list) as ArrayRef)]).unwrap();
+        assert_eq!(
+            dictionary_columns(properties(20_000, 1024 * 1024, 1024), &[batch]),
+            ["l.list.element"]
+        );
+    }
+
+    /// A batch whose leaves do not line up with the schema is not guessed at.
+    #[test]
+    fn leaves_every_column_as_configured_when_the_batch_does_not_match_the_schema() {
+        let schema = ArrowSchema::new(vec![Field::new("id", DataType::Int64, false)]);
+        let chooser =
+            DictionaryChooser::try_new(properties(20_000, 1024 * 1024, 1024), &schema).unwrap();
+        let batch = RecordBatch::try_from_iter([
+            (
+                "id",
+                Arc::new(Int64Array::from_iter_values(0..100)) as ArrayRef,
+            ),
+            (
+                "extra",
+                Arc::new(Int64Array::from_iter_values(0..100)) as ArrayRef,
+            ),
+        ])
+        .unwrap();
+        assert!(chooser
+            .choose(&[batch])
+            .dictionary_enabled(&ColumnPath::from("id")));
+    }
+}
diff --git a/native/core/src/execution/operators/iceberg_write.rs b/native/core/src/execution/operators/iceberg_write.rs
index 499db1ff0d..857f3ef129 100644
--- a/native/core/src/execution/operators/iceberg_write.rs
+++ b/native/core/src/execution/operators/iceberg_write.rs
@@ -18,11 +18,12 @@
 //! Native Iceberg write operator using iceberg-rust.
 //!
 //! Drains the upstream Arrow stream through iceberg-rust's writer stack
-//! (`ParquetWriterBuilder` -> `RollingFileWriterBuilder` -> `DataFileWriterBuilder`
-//! -> `Unpartitioned`/`Fanout`/`Clustered`Writer) and emits a single-row, single-column
-//! Arrow batch carrying the `Vec<DataFile>` produced for the task, packed as an Iceberg V2
-//! data manifest via iceberg-rust's `ManifestWriter` against an in-memory `FileIO`. The JVM
-//! decodes the bytes with `ManifestFiles.read(...)` to recover the `DataFile`s for commit.
+//! (`ParquetWriterBuilder` -> `RollingFileWriterBuilder` -> `DataFileWriterBuilder`, assembled
+//! per partition by [`PartitionWriterBuilder`], under an `Unpartitioned`/`Fanout`/`Clustered`
+//! writer) and emits a single-row, single-column Arrow batch carrying the `Vec<DataFile>`
+//! produced for the task, packed as an Iceberg V2 data manifest via iceberg-rust's
+//! `ManifestWriter` against an in-memory `FileIO`. The JVM decodes the bytes with
+//! `ManifestFiles.read(...)` to recover the `DataFile`s for commit.
 
 use std::collections::HashMap;
 use std::fmt;
@@ -51,7 +52,7 @@ use iceberg::spec::{
     PartitionSpecRef, Schema as IcebergSchema, SchemaRef as IcebergSchemaRef,
     Struct as IcebergStruct, StructType,
 };
-use iceberg::writer::base_writer::data_file_writer::DataFileWriterBuilder;
+use iceberg::writer::base_writer::data_file_writer::{DataFileWriter, DataFileWriterBuilder};
 use iceberg::writer::file_writer::location_generator::{
     DefaultFileNameGenerator, LocationGenerator,
 };
@@ -60,6 +61,7 @@ use iceberg::writer::file_writer::ParquetWriterBuilder;
 use iceberg::writer::partitioning::clustered_writer::ClusteredWriter;
 use iceberg::writer::partitioning::fanout_writer::FanoutWriter;
 use iceberg::writer::partitioning::unpartitioned_writer::UnpartitionedWriter;
+use iceberg::writer::IcebergWriterBuilder;
 use iceberg::ErrorKind;
 #[cfg(test)]
 use parquet::arrow::PARQUET_FIELD_ID_META_KEY;
@@ -74,17 +76,100 @@ use datafusion_comet_proto::spark_operator::{
 use crate::cloud::s3::credential_bridge::AccessMode;
 use crate::errors::CometError;
 use crate::execution::operators::iceberg_common::load_file_io;
+use crate::execution::operators::iceberg_dictionary::DictionaryChooser;
 use crate::execution::operators::iceberg_partition_path::{
     partition_to_path, CometLocationGenerator,
 };
 use crate::execution::operators::iceberg_partition_value::PartitionValueCalculator;
 
-/// Builder chain instantiated once per task and handed to the partitioning wrapper.
-type IcebergDataFileWriterBuilder = DataFileWriterBuilder<
-    ParquetWriterBuilder,
-    TrackingLocationGenerator,
-    DefaultFileNameGenerator,
->;
+/// The writer every partition gets: iceberg-rust's own data file writer, rolling Parquet files.
+type PartitionDataFileWriter =
+    DataFileWriter<ParquetWriterBuilder, TrackingLocationGenerator, DefaultFileNameGenerator>;
+
+/// Opens each partition's data file writer with the Parquet writer properties chosen for that
+/// partition.
+///
+/// A `ParquetWriterBuilder` fixes one set of writer properties for every file it opens, but which
+/// columns keep dictionary encoding is decided per partition (see [`DictionaryChooser`]). So the
+/// partitioning writers get this builder instead, and it assembles iceberg-rust's usual
+/// `ParquetWriterBuilder` -> `RollingFileWriterBuilder` -> `DataFileWriterBuilder` chain once per
+/// partition, with that partition's properties. Every file the partition rolls into keeps them.
+/// The location and file name generators are shared, so file names still count up across the
+/// task and every location is still tracked for cleanup.
+struct PartitionWriterBuilder {
+    schema: IcebergSchemaRef,
+    target_file_size: usize,
+    file_io: FileIO,
+    location_generator: TrackingLocationGenerator,
+    file_name_generator: DefaultFileNameGenerator,
+    properties: Arc<PartitionProperties>,
+}
+
+#[async_trait::async_trait]
+impl IcebergWriterBuilder for PartitionWriterBuilder {
+    type R = PartitionDataFileWriter;
+
+    async fn build(&self, partition_key: Option<PartitionKey>) -> iceberg::Result<Self::R> {
+        let properties = self
+            .properties
+            .take(partition_key.as_ref().map(PartitionKey::data));
+        let rolling_builder = RollingFileWriterBuilder::new(
+            ParquetWriterBuilder::new(properties, Arc::clone(&self.schema)),
+            self.target_file_size,
+            self.file_io.clone(),
+            self.location_generator.clone(),
+            self.file_name_generator.clone(),
+        );
+        DataFileWriterBuilder::new(rolling_builder)
+            .build(partition_key)
+            .await
+    }
+}
+
+/// The Parquet writer properties each partition's files are written with: the table's, with the
+/// dictionary choice [`DictionaryChooser`] made from the partition's first rows.
+///
+/// A partition's [`PartitionFeed`] records the choice before it hands the partition's first rows
+/// to the writer, and [`PartitionWriterBuilder`] takes it back out when that first write opens the
+/// partition's writer. The partitioning writers open a partition's writer exactly once per task,
+/// on its first write.
+struct PartitionProperties {
+    chooser: DictionaryChooser,
+    /// Keyed by partition value; `None` is the whole task of an unpartitioned write.
+    chosen: Mutex<HashMap<Option<IcebergStruct>, WriterProperties>>,
+}
+
+impl PartitionProperties {
+    fn new(chooser: DictionaryChooser) -> Self {
+        Self {
+            chooser,
+            chosen: Mutex::new(HashMap::new()),
+        }
+    }
+
+    /// Chooses the properties for `partition`, whose rows start with `sample`.
+    fn choose(&self, partition: &Option<IcebergStruct>, sample: &[RecordBatch]) {
+        let properties = self.chooser.choose(sample);
+        self.chosen
+            .lock()
+            .unwrap_or_else(|poisoned| poisoned.into_inner())
+            .insert(partition.clone(), properties);
+    }
+
+    /// The properties chosen for `partition`, or the table's own if none were.
+    fn take(&self, partition: Option<&IcebergStruct>) -> WriterProperties {
+        let chosen = self
+            .chosen
+            .lock()
+            .unwrap_or_else(|poisoned| poisoned.into_inner())
+            .remove(&partition.cloned());
+        debug_assert!(
+            chosen.is_some(),
+            "a partition's writer opened before its dictionary choice was made"
+        );
+        chosen.unwrap_or_else(|| self.chooser.base().clone())
+    }
+}
 
 /// [`CometLocationGenerator`] that records every location it hands to a file writer.
 ///
@@ -522,26 +607,28 @@ async fn run_write_task(
         None,
         DataFileFormat::Parquet,
     );
-    let parquet_builder = ParquetWriterBuilder::new(writer_properties, Arc::clone(&iceberg_schema));
-    let rolling_builder = RollingFileWriterBuilder::new(
-        parquet_builder,
-        common.target_file_size_bytes as usize,
-        file_io.clone(),
-        location_generator.clone(),
+    // Build the field-id-decorated target schema once per task; every batch is cast against it.
+    let target_schema =
+        Arc::new(iceberg::arrow::schema_to_arrow_schema(&iceberg_schema).map_err(iceberg_err)?);
+    let slicer = RowSlicer::for_schema(&target_schema);
+    let properties = Arc::new(PartitionProperties::new(DictionaryChooser::try_new(
+        writer_properties,
+        &target_schema,
+    )?));
+    let data_file_builder = PartitionWriterBuilder {
+        schema: Arc::clone(&iceberg_schema),
+        target_file_size: common.target_file_size_bytes as usize,
+        file_io: file_io.clone(),
+        location_generator: location_generator.clone(),
         file_name_generator,
-    );
-    let data_file_builder = DataFileWriterBuilder::new(rolling_builder);
+        properties: Arc::clone(&properties),
+    };
     let mut abort_guard = AbortOnDrop {
         file_io,
         generator: location_generator,
         armed: true,
     };
 
-    // Build the field-id-decorated target schema once per task; every batch is cast against it.
-    let target_schema =
-        Arc::new(iceberg::arrow::schema_to_arrow_schema(&iceberg_schema).map_err(iceberg_err)?);
-    let slicer = RowSlicer::for_schema(&target_schema);
-
     let unpartitioned = partition_spec.is_unpartitioned();
     let splitter = || {
         PartitionSplitter::try_new(
@@ -553,12 +640,12 @@ async fn run_write_task(
     let mut writer = match (unpartitioned, writer_mode) {
         (true, ProtoIcebergWriterMode::IcebergWriterUnpartitioned) => InnerWriter::Unpartitioned(
             UnpartitionedWriter::new(data_file_builder),
-            RowPacer::new(slicer),
+            PartitionFeed::new(None, slicer),
         ),
         (false, ProtoIcebergWriterMode::IcebergWriterFanout) => InnerWriter::Fanout(
             FanoutWriter::new(data_file_builder),
             splitter()?,
-            HashMap::new(),
+            FanoutFeeds::default(),
         ),
         (false, ProtoIcebergWriterMode::IcebergWriterClustered) => {
             InnerWriter::Clustered(ClusteredWriter::new(data_file_builder), splitter()?, None)
@@ -575,10 +662,10 @@ async fn run_write_task(
         while let Some(batch) = input.try_next().await? {
             let decorated = decorate_batch_with_field_ids(batch, &target_schema)?;
             let _timer = write_time.timer();
-            writer.write(decorated).await?;
+            writer.write(decorated, &properties).await?;
         }
         let _timer = write_time.timer();
-        writer.close().await
+        writer.close(&properties).await
     }
     .await;
     match outcome {
@@ -597,72 +684,92 @@ async fn run_write_task(
 }
 
 /// Enum-based dispatch over the three iceberg-rust partitioning writers, each paired with the
-/// [`RowPacer`] state that keeps its rolling writer on iceberg-java's row grid, and the two
-/// partitioned ones with the [`PartitionSplitter`] that cuts their input. Each variant takes the
-/// same builder chain so we can keep the type fixed.
+/// [`PartitionFeed`] state that holds a partition's first rows back for its dictionary choice and
+/// keeps its rolling writer on iceberg-java's row grid, and the two partitioned ones with the
+/// [`PartitionSplitter`] that cuts their input. Each variant takes the same builder so we can keep
+/// the type fixed.
 #[allow(clippy::large_enum_variant)]
 enum InnerWriter {
-    Unpartitioned(UnpartitionedWriter<IcebergDataFileWriterBuilder>, RowPacer),
-    /// The fanout writer keeps one file open per partition, so every partition paces separately.
-    /// The `PartitionKey` is kept alongside so leftovers can still be written out at close.
+    Unpartitioned(UnpartitionedWriter<PartitionWriterBuilder>, PartitionFeed),
+    /// The fanout writer keeps one file open per partition, so every partition is fed separately.
     Fanout(
-        FanoutWriter<IcebergDataFileWriterBuilder>,
+        FanoutWriter<PartitionWriterBuilder>,
         PartitionSplitter,
-        HashMap<IcebergStruct, (PartitionKey, RowPacer)>,
+        FanoutFeeds,
     ),
     /// The clustered writer closes a partition's file as soon as the next key arrives, so only the
-    /// current key's leftovers are live; they are written out before the switch.
+    /// current key's held rows are live; they are written out before the switch.
     Clustered(
-        ClusteredWriter<IcebergDataFileWriterBuilder>,
+        ClusteredWriter<PartitionWriterBuilder>,
         PartitionSplitter,
-        Option<(PartitionKey, RowPacer)>,
+        Option<(PartitionKey, PartitionFeed)>,
     ),
 }
 
 impl InnerWriter {
     /// Writes `batch` through the writer this task built, in the [`ROWS_DIVISOR`]-row units the
     /// rolling writer needs to re-check the target file size on iceberg-java's cadence. Rows are
-    /// paced after partition splitting, so each partition's file is measured against its own row
-    /// count the way the JVM writer measures it.
-    async fn write(&mut self, batch: RecordBatch) -> DFResult<()> {
+    /// fed after partition splitting, so each partition's dictionary choice is made from its own
+    /// rows, and each partition's file is measured against its own row count the way the JVM
+    /// writer measures it.
+    async fn write(
+        &mut self,
+        batch: RecordBatch,
+        properties: &PartitionProperties,
+    ) -> DFResult<()> {
         use iceberg::writer::partitioning::PartitioningWriter;
         match self {
-            InnerWriter::Unpartitioned(w, pacer) => {
-                for unit in pacer.push(batch)? {
+            InnerWriter::Unpartitioned(w, feed) => {
+                for unit in feed.push(batch, properties)? {
                     w.write(unit).await.map_err(iceberg_err)?;
                 }
                 Ok(())
             }
-            InnerWriter::Fanout(w, splitter, pacers) => {
+            InnerWriter::Fanout(w, splitter, fanout) => {
                 for (key, part) in splitter.split_groups(&batch)? {
-                    let (_, pacer) = pacers
-                        .entry(key.data().clone())
-                        .or_insert_with(|| (key.clone(), RowPacer::new(splitter.slicer)));
-                    for unit in pacer.push(part)? {
+                    let (key, feed) = fanout.feeds.entry(key.data().clone()).or_insert_with(|| {
+                        let partition = Some(key.data().clone());
+                        (key, PartitionFeed::new(partition, splitter.slicer))
+                    });
+                    let held_before = feed.held_bytes();
+                    let units = feed.push(part, properties)?;
+                    fanout.held_bytes = fanout.held_bytes - held_before + feed.held_bytes();
+                    for unit in units {
                         w.write(key.clone(), unit).await.map_err(iceberg_err)?;
                     }
                 }
+                // Every partition stays open, so the rows they hold back share one limit.
+                if !properties.chooser.may_hold(fanout.held_bytes) {
+                    for (key, feed) in fanout.feeds.values_mut() {
+                        for unit in feed.release(properties)? {
+                            w.write(key.clone(), unit).await.map_err(iceberg_err)?;
+                        }
+                    }
+                    fanout.held_bytes = 0;
+                }
                 Ok(())
             }
             InnerWriter::Clustered(w, splitter, live) => {
                 for (key, part) in splitter.split_runs(&batch)? {
-                    // A new key closes the previous partition's file, so its leftovers have to go
-                    // out first -- and its row grid does not carry over to the new partition.
-                    if let Some((open, mut pacer)) =
+                    // A new key closes the previous partition's file, so its held rows have to go
+                    // out first -- and neither its row grid nor its dictionary choice carries over
+                    // to the new partition.
+                    if let Some((open, mut feed)) =
                         live.take_if(|(open, _)| open.data() != key.data())
                     {
-                        if let Some(rest) = pacer.flush()? {
-                            // `write` consumes the key, but the unclustered-input error needs it
-                            // for the partition path.
-                            let key_for_error = open.clone();
-                            w.write(open, rest)
+                        for unit in feed.finish(properties)? {
+                            w.write(open.clone(), unit)
                                 .await
-                                .map_err(|e| clustered_write_err(e, &key_for_error, splitter))?;
+                                .map_err(|e| clustered_write_err(e, &open, splitter))?;
                         }
                     }
-                    let (_, pacer) =
-                        live.get_or_insert_with(|| (key.clone(), RowPacer::new(splitter.slicer)));
-                    for unit in pacer.push(part)? {
+                    let (_, feed) = live.get_or_insert_with(|| {
+                        (
+                            key.clone(),
+                            PartitionFeed::new(Some(key.data().clone()), splitter.slicer),
+                        )
+                    });
+                    for unit in feed.push(part, properties)? {
                         w.write(key.clone(), unit)
                             .await
                             .map_err(|e| clustered_write_err(e, &key, splitter))?;
@@ -676,19 +783,19 @@ impl InnerWriter {
     /// Hands over whatever rows are still waiting, then closes. iceberg-java's writer does the
     /// same at close: the leftovers land in the file that is open at that point, unless the
     /// pending target-size check rolls first -- exactly as they would have on the JVM path.
-    async fn close(self) -> DFResult<Vec<DataFile>> {
+    async fn close(self, properties: &PartitionProperties) -> DFResult<Vec<DataFile>> {
         use iceberg::writer::partitioning::PartitioningWriter;
         match self {
-            InnerWriter::Unpartitioned(mut w, mut pacer) => {
-                if let Some(rest) = pacer.flush()? {
-                    w.write(rest).await.map_err(iceberg_err)?;
+            InnerWriter::Unpartitioned(mut w, mut feed) => {
+                for unit in feed.finish(properties)? {
+                    w.write(unit).await.map_err(iceberg_err)?;
                 }
                 w.close().await.map_err(iceberg_err)
             }
-            InnerWriter::Fanout(mut w, _, pacers) => {
-                for (_, (key, mut pacer)) in pacers {
-                    if let Some(rest) = pacer.flush()? {
-                        w.write(key, rest).await.map_err(iceberg_err)?;
+            InnerWriter::Fanout(mut w, _, fanout) => {
+                for (_, (key, mut feed)) in fanout.feeds {
+                    for unit in feed.finish(properties)? {
+                        w.write(key.clone(), unit).await.map_err(iceberg_err)?;
                     }
                 }
                 let mut data_files = w.close().await.map_err(iceberg_err)?;
@@ -709,15 +816,14 @@ impl InnerWriter {
                 Ok(data_files)
             }
             InnerWriter::Clustered(mut w, splitter, live) => {
-                if let Some((key, mut pacer)) = live {
-                    if let Some(rest) = pacer.flush()? {
-                        // Pacing defers a partition's rows to here, so the unclustered-input
-                        // rejection can surface at close rather than during `write`. It still has
-                        // to read as iceberg-java's error.
-                        let key_for_error = key.clone();
-                        w.write(key, rest)
+                if let Some((key, mut feed)) = live {
+                    // Feeding defers a partition's rows to here, so the unclustered-input
+                    // rejection can surface at close rather than during `write`. It still has to
+                    // read as iceberg-java's error.
+                    for unit in feed.finish(properties)? {
+                        w.write(key.clone(), unit)
                             .await
-                            .map_err(|e| clustered_write_err(e, &key_for_error, &splitter))?;
+                            .map_err(|e| clustered_write_err(e, &key, &splitter))?;
                     }
                 }
                 w.close().await.map_err(iceberg_err)
@@ -1152,6 +1258,104 @@ impl RowPacer {
     }
 }
 
+/// One partition's rows on their way to its writer.
+///
+/// The partition's first rows are held back until they make up the first data page of every
+/// column, or the partition runs out of rows, so that its dictionary choice sees what parquet-mr's
+/// would. The choice is recorded before any row reaches the writer, since the first write opens
+/// the partition's writer with it. From then on rows are paced by a [`RowPacer`]. Holding rows back
+/// does not move a roll point: the rolling writer receives the same units, cut at the same rows and
+/// in the same order, only later.
+///
+/// Held batches are kept as they arrive. The fanout splitter's parts are already copies, and only
+/// the live partition of a clustered write holds any, so the slices it holds pin at most the input
+/// batches that one partition's first page spans.
+struct PartitionFeed {
+    /// The partition value, or `None` for the whole task of an unpartitioned write.
+    partition: Option<IcebergStruct>,
+    pacer: RowPacer,
+    /// The rows held back while the choice is pending, or `None` once it is made.
+    held: Option<HeldRows>,
+}
+
+#[derive(Default)]
+struct HeldRows {
+    batches: Vec<RecordBatch>,
+    rows: usize,
+    bytes: usize,
+}
+
+impl PartitionFeed {
+    fn new(partition: Option<IcebergStruct>, slicer: RowSlicer) -> Self {
+        Self {
+            partition,
+            pacer: RowPacer::new(slicer),
+            held: Some(HeldRows::default()),
+        }
+    }
+
+    /// Memory taken by the rows held back.
+    fn held_bytes(&self) -> usize {
+        self.held.as_ref().map_or(0, |held| held.bytes)
+    }
+
+    /// The complete units `batch` makes available, in row order.
+    fn push(
+        &mut self,
+        batch: RecordBatch,
+        properties: &PartitionProperties,
+    ) -> DFResult<Vec<RecordBatch>> {
+        let Some(held) = &mut self.held else {
+            return self.pacer.push(batch);
+        };
+        held.rows += batch.num_rows();
+        held.bytes += batch.get_array_memory_size();
+        held.batches.push(batch);
+        if properties.chooser.wants_more(held.rows, held.bytes) {
+            return Ok(Vec::new());
+        }
+        self.release(properties)
+    }
+
+    /// Every row still waiting, as the units it makes up followed by any partial unit, so a close
+    /// or a partition switch does not lose it. A partition that never filled a first page makes
+    /// its choice from the rows it has.
+    fn finish(&mut self, properties: &PartitionProperties) -> DFResult<Vec<RecordBatch>> {
+        let mut units = self.release(properties)?;
+        units.extend(self.pacer.flush()?);
+        Ok(units)
+    }
+
+    /// Makes the choice from the held rows, if it is still pending, and hands them to the pacer,
+    /// returning the complete units they make up.
+    fn release(&mut self, properties: &PartitionProperties) -> DFResult<Vec<RecordBatch>> {
+        let Some(held) = self.held.take() else {
+            return Ok(Vec::new());
+        };
+        properties.choose(&self.partition, &held.batches);
+        let mut units = Vec::new();
+        for batch in held.batches {
+            units.extend(self.pacer.push(batch)?);
+        }
+        Ok(units)
+    }
+}
+
+/// The feeds of a fanout write, one per partition, all open at once.
+///
+/// Between them they hold back no more than a single partition may: once the rows held across all
+/// partitions reach that, every partition still holding makes its choice from the rows it has. A
+/// task fanning out to many partitions that each get less than a page would otherwise hold all of
+/// its rows back, uncompressed, until it closed.
+#[derive(Default)]
+struct FanoutFeeds {
+    /// The `PartitionKey` is kept alongside each feed so held rows can still be written out at
+    /// close.
+    feeds: HashMap<IcebergStruct, (PartitionKey, PartitionFeed)>,
+    /// Bytes held back across all feeds.
+    held_bytes: usize,
+}
+
 /// `true` when `data_type` puts a float or double under a list or map, where a slice's offset
 /// window is invisible to iceberg-rust's NaN-count visitor.
 fn float_under_list_or_map(data_type: &DataType) -> bool {
@@ -2023,6 +2227,25 @@ mod tests {
             spec: PartitionSpec,
             writer_mode: ProtoIcebergWriterMode,
             batches: Vec<RecordBatch>,
+        ) -> DFResult<Vec<DataFile>> {
+            run_with_properties(
+                common,
+                schema,
+                spec,
+                writer_mode,
+                batches,
+                WriterProperties::builder().build(),
+            )
+            .await
+        }
+
+        async fn run_with_properties(
+            common: Arc<IcebergWriteCommon>,
+            schema: Schema,
+            spec: PartitionSpec,
+            writer_mode: ProtoIcebergWriterMode,
+            batches: Vec<RecordBatch>,
+            writer_properties: WriterProperties,
         ) -> DFResult<Vec<DataFile>> {
             let (data_files, mut abort_guard) = run_write_task(
                 input_stream(batches),
@@ -2030,7 +2253,7 @@ mod tests {
                 Arc::new(schema),
                 Arc::new(spec),
                 writer_mode,
-                WriterProperties::builder().build(),
+                writer_properties,
                 Some(0),
                 Some(0),
                 Time::default(),
@@ -2777,6 +3000,190 @@ mod tests {
             assert_eq!(rows, vec![1000, 500, 1000, 500]);
         }
 
+        /// Writes `batches` through a fresh table partitioned by `region` (or not, for the
+        /// unpartitioned mode) and returns its data files, with the directory they live in.
+        async fn write_files(
+            writer_mode: ProtoIcebergWriterMode,
+            batches: Vec<RecordBatch>,
+        ) -> (TempDir, Vec<DataFile>) {
+            write_files_with_properties(writer_mode, batches, WriterProperties::builder().build())
+                .await
+        }
+
+        async fn write_files_with_properties(
+            writer_mode: ProtoIcebergWriterMode,
+            batches: Vec<RecordBatch>,
+            writer_properties: WriterProperties,
+        ) -> (TempDir, Vec<DataFile>) {
+            let temp_dir = TempDir::new().unwrap();
+            let data_location = format!("file://{}", temp_dir.path().display());
+            let schema = iceberg_user_schema();
+            let spec = match writer_mode {
+                ProtoIcebergWriterMode::IcebergWriterUnpartitioned => {
+                    PartitionSpec::builder(Arc::new(schema.clone()))
+                        .build()
+                        .unwrap()
+                }
+                _ => identity_region_spec(&schema),
+            };
+            let common = common(
+                data_location,
+                serde_json::to_string(&spec).unwrap(),
+                serde_json::to_string(&schema).unwrap(),
+                writer_mode,
+            );
+            let data_files = run_with_properties(
+                common,
+                schema,
+                spec,
+                writer_mode,
+                batches,
+                writer_properties,
+            )
+            .await
+            .unwrap();
+            (temp_dir, data_files)
+        }
+
+        /// For every column chunk of `data_file`, its column and whether it has a dictionary page.
+        fn dictionary_pages(data_file: &DataFile) -> Vec<(String, bool)> {
+            use parquet::file::reader::{FileReader, SerializedFileReader};
+            let path = data_file.file_path().trim_start_matches("file://");
+            let reader = SerializedFileReader::new(std::fs::File::open(path).unwrap()).unwrap();
+            reader
+                .metadata()
+                .row_groups()
+                .iter()
+                .flat_map(|row_group| row_group.columns())
+                .map(|chunk| {
+                    (
+                        chunk.column_path().string(),
+                        chunk.dictionary_page_offset().is_some(),
+                    )
+                })
+                .collect()
+        }
+
+        /// A column of unique values gets no dictionary page, as iceberg-java writes it, while a
+        /// repetitive one keeps its dictionary (apache/datafusion-comet#6114). parquet-rs alone
+        /// would dictionary-encode `id` until its dictionary filled.
+        #[tokio::test]
+        async fn a_unique_column_is_written_without_a_dictionary() {
+            let batches = (0..3)
+                .map(|batch| {
+                    let ids: Vec<i32> = (batch * 8192..(batch + 1) * 8192).collect();
+                    let regions: Vec<&str> = ids
+                        .iter()
+                        .map(|id| if id % 2 == 0 { "us" } else { "eu" })
+                        .collect();
+                    self::batch(&ids, &regions)
+                })
+                .collect();
+            let (_dir, data_files) =
+                write_files(ProtoIcebergWriterMode::IcebergWriterUnpartitioned, batches).await;
+            assert_eq!(data_files.len(), 1);
+            assert_eq!(
+                dictionary_pages(&data_files[0]),
+                vec![("id".to_string(), false), ("region".to_string(), true)]
+            );
+        }
+
+        /// Every partition chooses from its own rows. `id` is unique in `us` but cycles through
+        /// three values in `eu`, so only the `eu` file keeps a dictionary for it, as it would from
+        /// iceberg-java's per-partition writers. A choice made once for the whole task would treat
+        /// both partitions alike.
+        #[tokio::test]
+        async fn each_partition_chooses_its_own_dictionaries() {
+            let id = |row: i32, region: &str| if region == "us" { row } else { row % 3 };
+            let interleaved: Vec<RecordBatch> = (0..2)
+                .map(|batch| {
+                    let rows = batch * 3000..(batch + 1) * 3000;
+                    let regions: Vec<&str> = rows
+                        .clone()
+                        .map(|row| if row % 2 == 0 { "us" } else { "eu" })
+                        .collect();
+                    let ids: Vec<i32> = rows
+                        .zip(&regions)
+                        .map(|(row, region)| id(row, region))
+                        .collect();
+                    self::batch(&ids, &regions)
+                })
+                .collect();
+            let sorted: Vec<RecordBatch> = ["eu", "us"]
+                .into_iter()
+                .map(|region| {
+                    let ids: Vec<i32> = (0..3000).map(|row| id(row, region)).collect();
+                    self::batch(&ids, &vec![region; 3000])
+                })
+                .collect();
+            for (writer_mode, batches) in [
+                (ProtoIcebergWriterMode::IcebergWriterFanout, interleaved),
+                (ProtoIcebergWriterMode::IcebergWriterClustered, sorted),
+            ] {
+                let (_dir, data_files) = write_files(writer_mode, batches).await;
+                let mut layouts: Vec<(bool, Vec<(String, bool)>)> = data_files
+                    .iter()
+                    .map(|file| {
+                        (
+                            file.file_path().contains("region=eu"),
+                            dictionary_pages(file),
+                        )
+                    })
+                    .collect();
+                layouts.sort();
+                assert_eq!(
+                    layouts,
+                    vec![
+                        (
+                            false,
+                            vec![("id".to_string(), false), ("region".to_string(), true)]
+                        ),
+                        (
+                            true,
+                            vec![("id".to_string(), true), ("region".to_string(), true)]
+                        ),
+                    ],
+                    "{writer_mode:?}"
+                );
+            }
+        }
+
+        /// The partitions of a fanout write hold back no more rows between them than one partition
+        /// may. Ten partitions of 1000 unique ids fill the 50 KiB row group size between them,
+        /// though none fills it alone, so each makes its choice from those rows and writes `id`
+        /// plain -- where the 1000 repeats of one id each gets next would have kept a dictionary.
+        #[tokio::test]
+        async fn fanout_partitions_share_one_limit_on_held_rows() {
+            let regions: Vec<String> = (0..10).map(|p| format!("r{p}")).collect();
+            let batch_of = |id: &dyn Fn(i32) -> i32| {
+                let ids: Vec<i32> = (0..10_000).map(id).collect();
+                let rows: Vec<&str> = (0..10_000)
+                    .map(|row| regions[row as usize / 1000].as_str())
+                    .collect();
+                self::batch(&ids, &rows)
+            };
+            let batches = vec![batch_of(&|row| row), batch_of(&|_| 7)];
+            let properties = WriterProperties::builder()
+                .set_max_row_group_bytes(Some(50 * 1024))
+                .build();
+            let (_dir, data_files) = write_files_with_properties(
+                ProtoIcebergWriterMode::IcebergWriterFanout,
+                batches,
+                properties,
+            )
+            .await;
+            assert_eq!(record_counts(&data_files), vec![2000; 10]);
+            for file in &data_files {
+                assert!(
+                    dictionary_pages(file)
+                        .iter()
+                        .all(|(column, dictionary)| column != "id" || !dictionary),
+                    "{} keeps a dictionary for id",
+                    file.file_path()
+                );
+            }
+        }
+
         /// Cutting a batch into pieces must not disturb the NaN counts the JVM carries into the
         /// manifest. iceberg-rust's visitor reads list children through `values()`, which ignores
         /// a slice's offset window, so a zero-copy slice would report every NaN in the batch once
diff --git a/native/core/src/execution/operators/mod.rs b/native/core/src/execution/operators/mod.rs
index 7d5c82a880..a15a528f90 100644
--- a/native/core/src/execution/operators/mod.rs
+++ b/native/core/src/execution/operators/mod.rs
@@ -26,6 +26,7 @@ mod dynamic_filter;
 pub(crate) use dynamic_filter::{DynamicFilterJoinExec, TopKReaderFilterExec};
 pub(crate) mod iceberg_common;
 pub use iceberg_common::clear_file_io_cache;
+mod iceberg_dictionary;
 pub(crate) mod iceberg_location_scoped;
 mod iceberg_partition_path;
 mod iceberg_partition_value;
diff --git a/spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala b/spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala
index 27c38e075e..b45eaea193 100644
--- a/spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala
@@ -35,6 +35,9 @@ import scala.util.control.NonFatal
 import org.json4s.{DefaultFormats, Formats}
 import org.json4s.jackson.JsonMethods.parse
 
+import org.apache.hadoop.fs.{Path => HadoopPath}
+import org.apache.parquet.hadoop.ParquetFileReader
+import org.apache.parquet.hadoop.util.HadoopInputFile
 import org.apache.spark.{CometListenerBusUtils, SparkConf, SparkException, Success, TaskContext}
 import org.apache.spark.rdd.RDD
 import org.apache.spark.scheduler.{SparkListener, SparkListenerTaskEnd}
@@ -2114,6 +2117,170 @@ class CometIcebergWriteActionSuite
     }
   }
 
+  // https://github.com/apache/datafusion-comet/issues/6114. parquet-mr writes a column chunk
+  // plain, with no dictionary page, when its first page shows the dictionary saving nothing;
+  // parquet-rs on its own keeps a dictionary until it fills. Both writers write the same rows here
+  // and their footers are compared column by column. `near_below` and `near_above` sit either side
+  // of parquet-mr's cut-off for its 20000-row first page: a choice made from a 1000-row sample
+  // would write `near_below` plain, and one made from all 30000 rows would keep a dictionary for
+  // `near_above`.
+  test("native acceleration: data files have dictionary pages exactly where iceberg-java's do") {
+    assumeNativeAcceleration()
+    withIcebergCatalog { warehouseDir =>
+      // A single Parquet file read back as one task: both writers see the same rows in the same
+      // order, and the native write gets a native child.
+      val source = new File(warehouseDir, "dictionary_src").getAbsolutePath
+      withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+        spark
+          .range(0, 30000, 1, 1)
+          .selectExpr(
+            "id",
+            "substr(sha2(cast(id AS STRING), 256), 1, 16) AS payload",
+            "concat('c', cast(id % 50 AS STRING)) AS category",
+            "cast(id % 10 AS INT) AS small",
+            "cast(pmod(hash(id), 5000) AS INT) AS medium",
+            "cast(pmod(hash(id), 14000) AS INT) AS near_below",
+            "cast(pmod(hash(id), 17000) AS INT) AS near_above",
+            "timestamp_micros(1700000000000000 + id * 1000003) AS ts",
+            "CASE WHEN id % 2 = 0 THEN id END AS half_null",
+            "CAST(NULL AS INT) AS all_null",
+            "cast(id AS DOUBLE) / 7 AS ratio",
+            "cast(id % 100 AS FLOAT) AS pct",
+            "CAST((id % 1000) / 100 AS DECIMAL(10, 2)) AS amount",
+            "date_add(DATE'2020-01-01', cast(id % 365 AS INT)) AS day",
+            "id % 3 = 0 AS flag",
+            "named_struct('a', id * 7, 'b', cast(id % 7 AS INT)) AS st",
+            "array(cast(id % 5 AS INT), cast(id % 3 AS INT)) AS small_list",
+            "array(id * 3, id * 3 + 1) AS unique_list",
+            "map(concat('k', cast(id % 4 AS STRING)), id) AS attrs",
+            "repeat(sha2(cast(id AS STRING), 256), 3) AS wide_unique",
+            "concat(repeat('x', 200), cast(id % 20 AS STRING)) AS wide_small")
+          .write
+          .parquet(source)
+      }
+      Seq("dict_native", "dict_jvm").foreach { t =>
+        spark.sql(s"""
+          CREATE TABLE $catalog.$ns.$t (
+            id BIGINT, payload STRING, category STRING, small INT, medium INT,
+            near_below INT, near_above INT, ts TIMESTAMP, half_null BIGINT, all_null INT,
+            ratio DOUBLE, pct FLOAT, amount DECIMAL(10, 2), day DATE, flag BOOLEAN,
+            st STRUCT<a: BIGINT, b: INT>, small_list ARRAY<INT>, unique_list ARRAY<BIGINT>,
+            attrs MAP<STRING, BIGINT>, wide_unique STRING, wide_small STRING
+          ) USING iceberg
+        """)
+      }
+      def insert(t: String): Unit =
+        spark.read.parquet(source).coalesce(1).writeTo(s"$catalog.$ns.$t").append()
+
+      val snapshot = withNativeEnabled { captureWrite("dict_native")(insert("dict_native")) }
+      assert(snapshot.snapshotDelta == 1L)
+      val nativeExecs = snapshot.plans.flatMap { p =>
+        collectWithSubqueries(p) { case e: CometIcebergWriteExec => e }
+      }
+      assert(nativeExecs.nonEmpty, "expected the write to engage the native path")
+      insert("dict_jvm")
+
+      def layout(t: String): Map[String, Boolean] = {
+        val files = dataFileDictionaryPages(t)
+        assert(files.size == 1, s"expected one data file for $t, got ${files.keys}")
+        files.values.head
+      }
+      val native = layout("dict_native")
+      val jvm = layout("dict_jvm")
+      assert(native == jvm, s"native $native != iceberg-java $jvm")
+      // Neither side may be degenerate: the table has columns of both kinds.
+      val plain = Set(
+        "id",
+        "payload",
+        "near_above",
+        "ts",
+        "half_null",
+        "all_null",
+        "ratio",
+        "flag",
+        "st.a",
+        "unique_list.list.element",
+        "attrs.key_value.value",
+        "wide_unique")
+      assert(native.collect { case (column, false) => column }.toSet == plain)
+    }
+  }
+
+  test("native acceleration: every partition keeps or drops a dictionary like iceberg-java") {
+    assumeNativeAcceleration()
+    withIcebergCatalog { warehouseDir =>
+      // `x` is unique in partition 0 and takes three values in partition 1, so a choice made once
+      // for the whole write would get one of the two partitions wrong.
+      val source = new File(warehouseDir, "partition_dictionary_src").getAbsolutePath
+      withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+        spark
+          .range(0, 20000, 1, 1)
+          .selectExpr(
+            "id",
+            "cast(id % 2 AS INT) AS p",
+            "CASE WHEN id % 2 = 0 THEN id ELSE id % 3 END AS x")
+          .write
+          .parquet(source)
+      }
+      Seq("pdict_native", "pdict_jvm").foreach { t =>
+        spark.sql(s"""
+          CREATE TABLE $catalog.$ns.$t (id BIGINT, p INT, x BIGINT)
+          USING iceberg PARTITIONED BY (p)
+        """)
+      }
+      def insert(t: String): Unit =
+        spark.read.parquet(source).writeTo(s"$catalog.$ns.$t").append()
+
+      val snapshot = withNativeEnabled { captureWrite("pdict_native")(insert("pdict_native")) }
+      assert(snapshot.snapshotDelta == 1L)
+      val nativeExecs = snapshot.plans.flatMap { p =>
+        collectWithSubqueries(p) { case e: CometIcebergWriteExec => e }
+      }
+      assert(nativeExecs.nonEmpty, "expected the write to engage the native path")
+      insert("pdict_jvm")
+
+      def layout(t: String): Map[Boolean, Map[String, Boolean]] =
+        dataFileDictionaryPages(t).map { case (path, columns) =>
+          path.contains("p=1") -> columns
+        }
+      val native = layout("pdict_native")
+      assert(native == layout("pdict_jvm"))
+      assert(native(false)("x") == false, s"partition 0 keeps a dictionary for x: $native")
+      assert(native(true)("x") == true, s"partition 1 drops the dictionary for x: $native")
+    }
+  }
+
+  /**
+   * For every data file of `table`, by path: whether each of its columns has a dictionary page in
+   * any row group, according to the encoding stats in the footer.
+   */
+  private def dataFileDictionaryPages(table: String): Map[String, Map[String, Boolean]] = {
+    val paths = spark
+      .sql(s"SELECT file_path FROM $catalog.$ns.$table.files")
+      .collect()
+      .map(_.getString(0))
+    paths.map { path =>
+      val reader = ParquetFileReader.open(
+        HadoopInputFile.fromPath(new HadoopPath(path), spark.sessionState.newHadoopConf()))
+      val columns =
+        try {
+          reader.getFooter.getBlocks.asScala
+            .flatMap(_.getColumns.asScala)
+            .groupBy(_.getPath.toDotString)
+            .map { case (column, chunks) =>
+              column -> chunks.exists { chunk =>
+                val stats = chunk.getEncodingStats
+                assert(stats != null, s"$column in $path has no encoding stats")
+                stats.hasDictionaryPages
+              }
+            }
+        } finally {
+          reader.close()
+        }
+      path -> columns
+    }.toMap
+  }
+
   test("native acceleration: empty append commits exactly once with zero data files") {
     assumeNativeAcceleration()
     withIcebergCatalog { warehouseDir =>
@@ -2592,11 +2759,15 @@ class CometIcebergWriteActionSuite
         .toDF("id", "region", "amount")
         .coalesce(1)
         .createOrReplaceTempView("retry_src")
+      // The native writer holds back a partition's first data page of rows for its dictionary
+      // choice and writes nothing before then. A 1000-row page lets it finalize files well before
+      // the failure at row 7000, which the default 20000-row page would not.
       createTable(
         warehouseDir,
         "retry_target",
         partitionSpec = "",
-        properties = Some("'write.target-file-size-bytes'='1'"))
+        properties =
+          Some("'write.target-file-size-bytes'='1', 'write.parquet.page-row-limit'='1000'"))
       NativeWriteRetryProbe.reset()
       val dataLocation = dataDir("retry_target").getAbsolutePath
       spark.udf.register(


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org