You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Kurtiscwright (via GitHub)" <gi...@apache.org> on 2026/04/29 07:31:35 UTC
[PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Kurtiscwright opened a new pull request, #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384
## Which issue does this PR close?
- Closes #2382
## What changes are included in this PR?
- Adds new File Format API RFC doc.
- From the Issue:
This RFC proposes a File Format API for the iceberg Rust crate that decouples Iceberg's read and write paths from any single file format. Today, iceberg-rust can only read and write Parquet data files: the format is hard-wired into ArrowReader, ParquetWriter, and every layer that touches them. The Java project shipped an analogous abstraction (FormatModel) in February 2026 via PR https://github.com/apache/iceberg/pull/12774, and PyIceberg has an open proposal (https://github.com/apache/iceberg-python/issues/3100) for the same concept.
## Are these changes tested?
- Just a doc update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "Kurtiscwright (via GitHub)" <gi...@apache.org>.
Kurtiscwright commented on PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#issuecomment-4389905034
@pvary @CTTY @xanderbailey thank you all for the feedback instead of responding in-line, I chose to work on a re-write with additional diagrams and code examples. I will send out a new commit with those additions and I am also changing the approach so we can have 3 layers of configuration data file format (parquet, orc) -> in-memory format (today only arrow-rs) -> engine format (Spark, Datafusion, Trino).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3160148755
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
Review Comment:
Which I think means I could loose https://github.com/apache/iceberg-rust/pull/2383/changes#diff-d7541c771a8632c6fb1644f8ec49a8e3efddae40c68197ca3657125059417454R96
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "dannycjones (via GitHub)" <gi...@apache.org>.
dannycjones commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3465997673
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
+
+The registry exposes `reader::<B>(format)` and `writer::<B>(format)`, where `B` is the batch type the engine chose. Errors are eager: if `(format, B)` is not registered, the call fails immediately at scan planning time rather than mid-stream during I/O. This is a runtime contract — the `TypeId` tells the registry what batch type the caller expects, but the type system does not enforce that the reader actually produces that type. A reader that returns the wrong type fails at the first downcast with a clear error. Java makes the same tradeoff with `Class<?>`.
+
+A process-wide default registry is available via `OnceLock`. Custom registries can be provided to `TableScanBuilder` for tests or restricted configurations.
+
+**Default scan loop (hybrid execution model).** The kernel provides a default scan loop that pulls batches from the format reader, checks `ReadResult` to see what the format already handled, and applies any remaining operations (residual filtering, schema evolution, constant injection, delete application). Engines that want full control (DataFusion, Comet) bypass this loop and drive execution themselves — they use the format reader as a stream source and apply their own processing. The `DataBatch` methods are available as a toolbox, not mandated as a pipeline.
+
+**Supporting types:** `ColumnMetrics` stores bounds as typed `Datum` values, matching how `DataFile` already stores bounds. `MetricsConfig` and `MetricsMode` control collection granularity. `FileContent` distinguishes data from equality and position deletes. All config structs and enums are `#[non_exhaustive]`.
+
+### Feature flags
+
+Format implementations are gated behind feature flags (`format-parquet` on by default, `format-orc` when ORC lands). The default feature set includes every implemented format. No configuration is needed for downstream crates using default features. Feature flags exist for crates that want to minimize binary size or compile time. This matches `datafusion`, `opendal`, and `reqwest`.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # Module declarations and public re-exports
+├── traits.rs # DataBatch, FormatReader, FormatWriter,
+│ # FormatFileWriter, ReadOptions, WriteOptions,
+│ # ColumnMetrics, MetricsConfig, MetricsMode,
+│ # FileContent, WriterResult,
+│ # VariantShredding (future extension, not yet wired)
+├── registry.rs # FormatRegistry, default_format_registry()
+└── parquet/
+ ├── mod.rs # Submodule declarations and re-exports
+ ├── model.rs # ParquetArrowModel (implements FormatReader + FormatWriter)
+ ├── reader.rs # Parquet reader internals
+ └── writer.rs # Parquet writer (implements FormatFileWriter)
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, each following the same structure.
+
+### Target crate architecture
+
+The end-state after crate separation (follow-up work, not part of Phase 1). All crates live in the same repository and workspace. Phase 1 ships everything in the existing `iceberg` crate.
+
+```
+iceberg (core kernel) - traits only, no arrow/parquet dependency
+iceberg-arrow - impl DataBatch for RecordBatch
+iceberg-format-parquet - ParquetArrowModel
+iceberg-format-orc - OrcArrowModel
+iceberg-datafusion - TableProvider, depends on above
+```
+
+This separation enables engines to pin their own Arrow version independently. For example, a Comet integration could depend on `iceberg` (the kernel) and provide its own `DataBatch` implementation and format readers using a newer Arrow version, without conflicting with `iceberg-arrow` or `iceberg-format-parquet`.
+
+## Design Rationale
+
+### Three layers, Arrow as default
+
+Java's File Format API has two layers: format and engine. Each format-engine combination requires a direct conversion path, producing an N-by-M matrix. The Java community's performance analysis ([benchmark branch](https://github.com/pvary/iceberg/tree/perf_bench)) measured 20% overhead for double conversion (ORC to Spark ColumnarBatch versus direct VectorizedRowBatch access) and recommended a hybrid: provide an Arrow-based intermediate for interoperability, allow engines to replace it for performance-sensitive paths.
+
+This RFC adopts the hybrid. Arrow is the shipped default. The `DataBatch` trait is the extension point for representations that cannot or should not convert to Arrow. If both format and engine happen to use Arrow (the common case today), there is zero intermediate conversion.
+
+### Why DataBatch is a trait
+
+The kernel ambition ([#1817](https://github.com/apache/iceberg-rust/issues/1817)) requires support for multiple in-memory representations. Arrow version coupling forces lockstep upgrades across the dependency graph — an engine that wants Arrow 60 cannot coexist with a kernel pinned to Arrow 58 unless the kernel is representation-agnostic. `DataBatch` as a trait solves both while preserving Arrow as the shipped default.
+
+The original version of this RFC fixed the data type to `RecordBatch`. Community feedback raised two concerns: that Arrow's dominance in the Rust ecosystem may reflect selection pressure rather than a permanent constraint, and that a kernel-level API should remain open to other ecosystems. This rewrite makes the in-memory type a trait boundary rather than a fixed type.
+
+The name `DataBatch` was chosen over alternatives like `ProcessableBatch` or `ExecutionBatch` because it names what flows through the system (batches of data) rather than what the trait demands (processability). The trait's contract is documented in the method signatures and the TCK; the name stays short and general since it will appear in every engine integration, error message, and type signature.
+
+### Concrete types for processing, trait objects for dispatch
+
+The processing layer works with the concrete batch type the engine chose (like `RecordBatch`). The registry uses trait objects (`Arc<dyn FormatReader>`) because it must hand back a reader for any format at runtime — Parquet for one table, ORC for another — without the caller naming the concrete reader type.
+
+The cost of trait objects is one type check per batch when pulling from the reader stream. The benefit is that the processing layer can fuse operations, inline method calls, and avoid heap allocations for intermediate results. This is the same split `object_store` uses (`dyn ObjectStore` at the boundary, concrete streams returned) and `datafusion` uses (`dyn TableProvider` at the boundary, concrete `RecordBatchStream` returned).
+
+`async fn` in traits does not work through `dyn Trait`. Since the registry returns trait objects, async methods return `BoxFuture`. The allocation happens once per file operation, not per batch.
+
+### Coarse trait methods, rich TCK
+
+`DataBatch::filter` accepts the full `BoundPredicate` tree. `DataBatch::evolve_schema` takes source and target schemas. The trait does not decompose operations into evaluate-per-leaf-then-compose-masks or compute-diff-then-apply-ops. Decomposition forces intermediate materializations. The existing Arrow implementation evaluates predicates as fused closures during I/O, avoiding materialized boolean masks. A decomposed interface would force Arrow to abandon this optimization or work around the trait.
+
+The cost of coarse interfaces is implementor burden. A new `DataBatch` implementation must correctly evaluate recursive predicates and handle schema evolution semantics. The mitigation is the TCK Layer 1 suite, which covers every operator, every primitive type, compound predicates, null handling, and schema evolution scenarios. A new implementation iterates against the TCK until all scenarios pass. Correctness is validated by testing, not constrained by interface shape.
+
+### No engine_schema parameter
+
+Java's `ReadBuilder.engineProjection(S)` and `ModelWriteBuilder.engineSchema(S)` serve two purposes: engine type widening and variant shredding configuration.
+
+Variant shredding is a format-layer concern. The physical shredding layout is defined by the Parquet spec ([VariantShredding.md](https://github.com/apache/parquet-format/blob/master/VariantShredding.md)). The decision of what to shred comes from table properties, data statistics, or engine-provided configuration — but in all cases it is expressed as typed Iceberg-level fields (which paths to extract, as what types), not as an opaque engine-specific schema. Java's Spark implements its own shredding inference internally (PR [#49234](https://github.com/apache/spark/pull/49234), [#52406](https://github.com/apache/spark/pull/52406)); Iceberg later built a shared analyzer (PR [#14297](https://github.com/apache/iceberg/pull/14297)). Neither passes a shredding schema through `engineSchema`. The format handles the physical encoding.
+
+Engine type widening (read an `int` column as `long`) is expressible as a typed `HashMap<i32, PrimitiveType>` on `ReadOptions` when needed.
+
+A type-erased `Box<dyn Any>` was rejected because it allows hidden coupling: a caller going through the registry could pass an object only one format understands. The failure mode is code that works in tests (Parquet-only) and breaks in production (table has ORC files), with a runtime error that surfaces in the reader, not where the wrong object was passed. Both `ReadOptions` and `WriteOptions` are `#[non_exhaustive]`; typed fields are added without a breaking change.
+
+### Options structs, not builder traits
+
+An earlier iteration used `FormatReadBuilder` and `FormatWriteBuilder` traits with fluent `&mut self` methods. Options structs were chosen instead: fewer traits, matches `object_store` precedent (`GetOptions`, `PutOptions`), composable without `Box<dyn>` allocations at each handoff point.
+
+### Divergence from Java
+
+| Aspect | Java | This RFC | Rationale |
+|--------|------|----------|-----------|
+| Type parameters | `FormatModel<D, S>` | Dyn-compatible traits, no type params | Registry requires trait objects |
+| Processing layer | Implicit (engine-specific) | Explicit `DataBatch` trait | Separates concerns, avoids N*M |
+| Registry key | `(FileFormat, Class<?>)` | `(DataFileFormat, TypeId)` | Same concept; batch type is the second dimension |
+| Format traits | Single `FormatModel` | Separate `FormatReader` + `FormatWriter` | A format may support one without the other |
+| Engine schema | Generic parameter `S` | Deferred; typed Iceberg-level fields when needed | Shredding is a format-layer concern, not engine |
+| Residual predicate | Not reported | `ReadResult::residual_predicate` | Kernel needs to know what format handled |
+| Old API coexistence | Kept indefinitely | Hard cutover (pre-1.0) | Clean end state while cost is low |
+
+## Conformance Tests (TCK)
+
+### Layered test architecture
+
+```
+Layer 3: Integration (format x batch x scenario)
+Layer 2: Format (one format, known-good batch type)
+Layer 1: DataBatch (one batch type, no format involvement)
+```
+
+When a Layer 3 test fails, Layer 1 and Layer 2 results identify whether the bug is in the batch implementation, the format implementation, or their interaction.
+
+### Layer 1: DataBatch conformance
+
+Layer 1 validates that a `DataBatch` implementation correctly handles Iceberg semantics without any file format involvement. A new batch type must pass every Layer 1 test before it can participate in the format API.
+
+**Filter correctness:**
+- Every binary operator (Eq, NotEq, Lt, LtEq, Gt, GtEq, StartsWith, NotStartsWith) against every primitive type.
+- Every unary operator (IsNull, NotNull, IsNan, NotNan).
+- Set operators (In, NotIn) with varying set sizes.
+- Compound predicates: AND, OR, NOT at multiple nesting depths.
+- Edge cases: all-null columns, all-matching, none-matching, NaN handling, empty batches.
+- Property-based: for random data and random predicates, verify `filter(p).num_rows()` matches naive row-by-row evaluation.
+
+**Schema evolution correctness:**
+- Type promotion: int to long, float to double, decimal widening.
+- Add column with default value (every primitive type).
+- Add column with null (optional field).
+- Column reordering (same fields, different order).
+- Drop column (project narrows).
+- Combined: promote + add + reorder in one operation.
+
+**Inject constants correctness:**
+- Every primitive Datum type as constant.
+- Multiple constants at once.
+- Zero-row batch.
+
+**Project correctness:**
+- Single column, multiple columns, all columns.
+- Non-existent field ID produces an error.
+- Idempotency: project(ids).project(ids) == project(ids).
+
+**Metrics correctness:**
+- Every primitive type: verify min/max match hand-computed values.
+- Null handling: null_count matches actual nulls.
+- NaN handling: nan_count for float/double.
+- All-null column: bounds are None.
+- Single-value column: lower == upper.
+
+### Layer 2: Format conformance
+
+Layer 2 validates that a format reader/writer correctly round-trips data and produces correct metadata. Uses Arrow RecordBatch (a known-good Layer 1 implementation) as the batch type.
+
+**Phase 1 scenarios (land with initial PR):**
+
+1. Round-trip every `PrimitiveType` with null and non-null values.
+2. Nested types: struct, list, map, combinations.
+3. Projection: write wide, read narrow.
+4. Null handling: all-null, mixed, required versus optional.
+5. Filter pushdown with row group elimination.
+6. Split reading across multiple splits.
+7. Statistics: column sizes, value counts, null counts, NaN counts, min/max bounds.
+8. Metrics modes: None, Counts, Full, Truncate.
+
+**Follow-up scenarios (before 1.0):**
+
+9. Schema evolution with defaults.
+10. Delete files (equality and position).
+11. Metadata columns.
+12. Case sensitivity.
+13. Encryption round-trip.
+14. Name mapping (reading files without embedded field IDs).
+15. Variant ([#2188](https://github.com/apache/iceberg-rust/pull/2188)).
+
+### Merge gates
+
+- New `DataBatch` implementation: must pass all Layer 1 tests.
+- New `FormatReader`/`FormatWriter` implementation: must pass all Layer 2 tests.
+- New format-batch combination: must pass all Layer 3 tests.
+
+## Migration / Rollout Plan
+
+### Phase 1: Land the abstraction (additive)
+
+Add `formats/` module with all traits, registry, and Phase 1 TCK. No existing code changes. The `ParquetArrowModel` implementation wraps the existing `ParquetWriter` and `ArrowReader` internally.
+
+### Phase 2: Migrate internal callers (breaking)
+
+Modify `TableScan` to dispatch through `FormatRegistry`. Modify `DataFileWriterBuilder` and `EqualityDeleteFileWriterBuilder` to use the registry. Update the DataFusion integration.
+
+### Phase 3: Relocate and separate (breaking)
+
+Move Parquet types into `formats/parquet/`. Establish crate boundaries per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819). Add `#[non_exhaustive]` to `DataFileFormat`.
+
+## Breaking Changes
+
+| # | What breaks | Phase | Migration |
+|---|-------------|-------|-----------|
+| 1 | `DataFileWriterBuilder` generic parameters removed | 2 | Use registry-based constructor |
+| 2 | `EqualityDeleteFileWriterBuilder` generic parameters removed | 2 | Use registry-based constructor |
+| 3 | `ParquetWriterBuilder` module path changes | 3 | Update `use` statements |
+| 4 | `ArrowReaderBuilder` path changes, renamed to `ParquetReaderBuilder` | 3 | Update `use` statements |
+| 5 | `DataFileFormat` gains `#[non_exhaustive]` | 3 | Add wildcard arm to `match` blocks |
+| 6 | `TableScan::to_arrow` dispatches through registry (behavior change) | 2 | No change for default registry users |
+
+## Open Questions
+
+### 1. Delete application
+
+**Current lean:** Delete application is a kernel responsibility, not part of `DataBatch` or the format reader. The kernel's default scan loop loads delete files, builds equality delete predicates and positional delete row selections, and applies them after the format reader produces batches. The format reader never sees delete files.
+
+**Tradeoff:** Handling deletes in the kernel means every `DataBatch` implementation gets merge-on-read for free — delete files are resolved before the engine sees batches. The downside is that a format reader cannot fuse delete application with I/O (for example, applying positional deletes during Parquet decompression to skip rows before materializing them). If a format implementation demonstrates that fusing delete application with I/O provides meaningful benefit, a future `ReadOptions` field could pass delete information to the format reader as an optimization hint.
+
+**Question:** Is kernel-owned delete application the right starting point, or should format readers have the option to handle deletes during I/O from the start?
Review Comment:
We still need to support the V2 use case and the continued equality delete use case, right? At least in the long-term.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "dannycjones (via GitHub)" <gi...@apache.org>.
dannycjones commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3463064012
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
Review Comment:
Ah, `std::any`, my bad! I haven't played much with type reflection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3828927512
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
Review Comment:
```suggestion
The `iceberg` crate is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
```
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
Review Comment:
I think should be part of the goal.
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
+
+The registry exposes `reader::<B>(format)` and `writer::<B>(format)`, where `B` is the batch type the engine chose. Errors are eager: if `(format, B)` is not registered, the call fails immediately at scan planning time rather than mid-stream during I/O. This is a runtime contract — the `TypeId` tells the registry what batch type the caller expects, but the type system does not enforce that the reader actually produces that type. A reader that returns the wrong type fails at the first downcast with a clear error. Java makes the same tradeoff with `Class<?>`.
+
+A process-wide default registry is available via `OnceLock`. Custom registries can be provided to `TableScanBuilder` for tests or restricted configurations.
+
+**Default scan loop (hybrid execution model).** The kernel provides a default scan loop that pulls batches from the format reader, checks `ReadResult` to see what the format already handled, and applies any remaining operations (residual filtering, schema evolution, constant injection, delete application). Engines that want full control (DataFusion, Comet) bypass this loop and drive execution themselves — they use the format reader as a stream source and apply their own processing. The `DataBatch` methods are available as a toolbox, not mandated as a pipeline.
+
+**Supporting types:** `ColumnMetrics` stores bounds as typed `Datum` values, matching how `DataFile` already stores bounds. `MetricsConfig` and `MetricsMode` control collection granularity. `FileContent` distinguishes data from equality and position deletes. All config structs and enums are `#[non_exhaustive]`.
+
+### Feature flags
+
+Format implementations are gated behind feature flags (`format-parquet` on by default, `format-orc` when ORC lands). The default feature set includes every implemented format. No configuration is needed for downstream crates using default features. Feature flags exist for crates that want to minimize binary size or compile time. This matches `datafusion`, `opendal`, and `reqwest`.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # Module declarations and public re-exports
+├── traits.rs # DataBatch, FormatReader, FormatWriter,
+│ # FormatFileWriter, ReadOptions, WriteOptions,
+│ # ColumnMetrics, MetricsConfig, MetricsMode,
+│ # FileContent, WriterResult,
+│ # VariantShredding (future extension, not yet wired)
+├── registry.rs # FormatRegistry, default_format_registry()
+└── parquet/
Review Comment:
I prefer to move parquet out of core crate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3164586355
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
Review Comment:
I think this is more of an implementation details
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
Review Comment:
Is this really necessary for the initial implementation? we only support parquet for now
TCK is not even completed on the java side afaik
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
Review Comment:
We don't have to hardcode it to use Arrow's `RecordBatch` even. We can use a generic type for in-memory representation and arrow can be the default value
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
+
+### The read and write builders
+
+`FormatReadBuilder` and `FormatWriteBuilder` configure a single read or write. `FormatModel` produces them, and the caller consumes them with `build`.
+
+```rust
+pub trait FormatReadBuilder: Send {
+ fn project(&mut self, schema: Schema) -> &mut Self;
+ fn filter(&mut self, predicate: BoundPredicate) -> &mut Self;
+ fn split(&mut self, start: u64, length: u64) -> &mut Self;
+ fn case_sensitive(&mut self, case_sensitive: bool) -> &mut Self;
+ fn batch_size(&mut self, batch_size: usize) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<ArrowRecordBatchStream>>;
+}
+
+pub trait FormatWriteBuilder: Send {
+ fn schema(&mut self, schema: Schema) -> &mut Self;
+ fn set(&mut self, key: String, value: String) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<Box<dyn FormatFileWriter>>>;
+}
+```
+
+Both builders take Iceberg `Schema` values. Format implementations convert to physical schemas internally using `schema_to_arrow_schema` from `arrow/schema.rs`. This deliberately departs from Java's `ReadBuilder<D, S>`, which has a separate `engineProjection(S)` method for variant shredding and similar per-engine type mapping. Rust's builders expose one projection surface, and the hook for a variant-shredding "engine projection" is a future-extension point described in "Design rationale."
+
+`FormatWriteBuilder::build` produces a `FormatFileWriter`:
+
+```rust
+pub trait FormatFileWriter: Send {
+ fn write(&mut self, batch: &RecordBatch) -> BoxFuture<'_, Result<()>>;
+ fn close(self: Box<Self>) -> BoxFuture<'static, Result<Vec<DataFileBuilder>>>;
+}
+```
+
+The async methods return `BoxFuture` rather than using `async fn` in traits. The `self: Box<Self>` signature on `build` and `close` lets those methods consume the value while keeping the traits object-safe. Both patterns are forced by the trait-object boundary at the registry. The "BoxFuture instead of async fn in traits" and "Dynamic dispatch at the registry, static inside the format" subsections under "Design rationale" explain why.
+
+### The FormatRegistry
+
+`FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances.
+
+```rust
+pub struct FormatRegistry {
+ models: HashMap<DataFileFormat, Box<dyn FormatModel>>,
+}
+
+impl FormatRegistry {
+ pub fn new() -> Self { ... }
+ pub fn register(&mut self, model: Box<dyn FormatModel>) { ... }
+ pub fn read_builder(
+ &self,
+ format: DataFileFormat,
+ input: InputFile,
+ ) -> Result<Box<dyn FormatReadBuilder>> { ... }
+ pub fn write_builder(
+ &self,
+ format: DataFileFormat,
+ output: OutputFile,
+ ) -> Result<Box<dyn FormatWriteBuilder>> { ... }
+}
+```
+
+Using `DataFileFormat` as a `HashMap` key requires adding `#[derive(Hash)]` to the enum. That is a non-breaking addition.
+
+The registry is an owned value, not a global static. Tests construct their own. Applications construct one at startup and pass it to scan planners and write orchestrators. For the common case of a single registry for the lifetime of a process, `default_format_registry()` returns a `&'static FormatRegistry` initialized through `OnceLock` on first call.
+
+`read_builder` and `write_builder` return `Err(Error { kind: ErrorKind::FeatureUnsupported, .. })` for unregistered formats. The error message distinguishes two cases: the format is implemented but its feature flag is disabled in this build, or the format has no implementation in this crate.
+
+### Feature flags
Review Comment:
This could be a non-goal, we only plan to support parquet for now and parquet is essential
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
+
+### The read and write builders
+
+`FormatReadBuilder` and `FormatWriteBuilder` configure a single read or write. `FormatModel` produces them, and the caller consumes them with `build`.
+
+```rust
+pub trait FormatReadBuilder: Send {
+ fn project(&mut self, schema: Schema) -> &mut Self;
+ fn filter(&mut self, predicate: BoundPredicate) -> &mut Self;
+ fn split(&mut self, start: u64, length: u64) -> &mut Self;
+ fn case_sensitive(&mut self, case_sensitive: bool) -> &mut Self;
+ fn batch_size(&mut self, batch_size: usize) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<ArrowRecordBatchStream>>;
+}
+
+pub trait FormatWriteBuilder: Send {
+ fn schema(&mut self, schema: Schema) -> &mut Self;
+ fn set(&mut self, key: String, value: String) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<Box<dyn FormatFileWriter>>>;
+}
+```
+
+Both builders take Iceberg `Schema` values. Format implementations convert to physical schemas internally using `schema_to_arrow_schema` from `arrow/schema.rs`. This deliberately departs from Java's `ReadBuilder<D, S>`, which has a separate `engineProjection(S)` method for variant shredding and similar per-engine type mapping. Rust's builders expose one projection surface, and the hook for a variant-shredding "engine projection" is a future-extension point described in "Design rationale."
+
+`FormatWriteBuilder::build` produces a `FormatFileWriter`:
+
+```rust
+pub trait FormatFileWriter: Send {
+ fn write(&mut self, batch: &RecordBatch) -> BoxFuture<'_, Result<()>>;
+ fn close(self: Box<Self>) -> BoxFuture<'static, Result<Vec<DataFileBuilder>>>;
+}
+```
+
+The async methods return `BoxFuture` rather than using `async fn` in traits. The `self: Box<Self>` signature on `build` and `close` lets those methods consume the value while keeping the traits object-safe. Both patterns are forced by the trait-object boundary at the registry. The "BoxFuture instead of async fn in traits" and "Dynamic dispatch at the registry, static inside the format" subsections under "Design rationale" explain why.
+
+### The FormatRegistry
+
+`FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances.
+
+```rust
+pub struct FormatRegistry {
+ models: HashMap<DataFileFormat, Box<dyn FormatModel>>,
+}
+
+impl FormatRegistry {
+ pub fn new() -> Self { ... }
+ pub fn register(&mut self, model: Box<dyn FormatModel>) { ... }
+ pub fn read_builder(
+ &self,
+ format: DataFileFormat,
+ input: InputFile,
+ ) -> Result<Box<dyn FormatReadBuilder>> { ... }
+ pub fn write_builder(
+ &self,
+ format: DataFileFormat,
+ output: OutputFile,
+ ) -> Result<Box<dyn FormatWriteBuilder>> { ... }
+}
+```
+
+Using `DataFileFormat` as a `HashMap` key requires adding `#[derive(Hash)]` to the enum. That is a non-breaking addition.
+
+The registry is an owned value, not a global static. Tests construct their own. Applications construct one at startup and pass it to scan planners and write orchestrators. For the common case of a single registry for the lifetime of a process, `default_format_registry()` returns a `&'static FormatRegistry` initialized through `OnceLock` on first call.
+
+`read_builder` and `write_builder` return `Err(Error { kind: ErrorKind::FeatureUnsupported, .. })` for unregistered formats. The error message distinguishes two cases: the format is implemented but its feature flag is disabled in this build, or the format has no implementation in this crate.
+
+### Feature flags
+
+Format implementations live in `iceberg::formats::{format}` and are gated behind a feature flag per format: `format-parquet` on by default, `format-orc` when an ORC implementation lands, and so on. The default feature set includes every format the crate implements. Users who build from source and want a smaller binary disable what they do not need.
+
+`FormatRegistry::default()` registers every format enabled by the current feature set at compile time:
+
+```rust
+impl Default for FormatRegistry {
+ fn default() -> Self {
+ let mut registry = Self::new();
+ #[cfg(feature = "format-parquet")]
+ registry.register(Box::new(ParquetFormatModel::new()));
+ #[cfg(feature = "format-orc")]
+ registry.register(Box::new(OrcFormatModel::new()));
+ registry
+ }
+}
+```
+
+Format implementations live inside the `iceberg` crate rather than as separate dependencies. This matches how the Java project keeps its `Parquet`, `Avro`, and `ORC` modules inside the `apache/iceberg` repository. In-tree implementations let the community control each format's quality and release cadence and keep the build graph narrow for downstream consumers. Nothing in the trait design prevents a downstream crate from defining its own `FormatModel` and registering it with a custom `FormatRegistry`, but in-tree is the recommended path for formats intended to ship in iceberg-rust itself.
+
+The `parquet` crate remains an unconditional dependency of `iceberg` for now, because non-format code (page index evaluators, row group metric evaluators, delete file loaders) still uses `parquet` types directly. A later pass gates the `parquet` dependency on the `format-parquet` feature once those callers move to format-agnostic abstractions.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # FormatModel, FormatReadBuilder, FormatWriteBuilder, FormatFileWriter
+├── registry.rs # FormatRegistry, default_format_registry
+└── parquet.rs # ParquetFormatModel, wrapping existing ParquetWriter and ArrowReader
Review Comment:
nit: This should be formats/parquet/mod.rs?
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
+
+### The read and write builders
+
+`FormatReadBuilder` and `FormatWriteBuilder` configure a single read or write. `FormatModel` produces them, and the caller consumes them with `build`.
+
+```rust
+pub trait FormatReadBuilder: Send {
+ fn project(&mut self, schema: Schema) -> &mut Self;
+ fn filter(&mut self, predicate: BoundPredicate) -> &mut Self;
+ fn split(&mut self, start: u64, length: u64) -> &mut Self;
+ fn case_sensitive(&mut self, case_sensitive: bool) -> &mut Self;
+ fn batch_size(&mut self, batch_size: usize) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<ArrowRecordBatchStream>>;
+}
+
+pub trait FormatWriteBuilder: Send {
+ fn schema(&mut self, schema: Schema) -> &mut Self;
+ fn set(&mut self, key: String, value: String) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<Box<dyn FormatFileWriter>>>;
+}
+```
+
+Both builders take Iceberg `Schema` values. Format implementations convert to physical schemas internally using `schema_to_arrow_schema` from `arrow/schema.rs`. This deliberately departs from Java's `ReadBuilder<D, S>`, which has a separate `engineProjection(S)` method for variant shredding and similar per-engine type mapping. Rust's builders expose one projection surface, and the hook for a variant-shredding "engine projection" is a future-extension point described in "Design rationale."
+
+`FormatWriteBuilder::build` produces a `FormatFileWriter`:
+
+```rust
+pub trait FormatFileWriter: Send {
+ fn write(&mut self, batch: &RecordBatch) -> BoxFuture<'_, Result<()>>;
+ fn close(self: Box<Self>) -> BoxFuture<'static, Result<Vec<DataFileBuilder>>>;
+}
+```
+
+The async methods return `BoxFuture` rather than using `async fn` in traits. The `self: Box<Self>` signature on `build` and `close` lets those methods consume the value while keeping the traits object-safe. Both patterns are forced by the trait-object boundary at the registry. The "BoxFuture instead of async fn in traits" and "Dynamic dispatch at the registry, static inside the format" subsections under "Design rationale" explain why.
+
+### The FormatRegistry
+
+`FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances.
+
+```rust
+pub struct FormatRegistry {
+ models: HashMap<DataFileFormat, Box<dyn FormatModel>>,
+}
+
+impl FormatRegistry {
+ pub fn new() -> Self { ... }
+ pub fn register(&mut self, model: Box<dyn FormatModel>) { ... }
+ pub fn read_builder(
+ &self,
+ format: DataFileFormat,
+ input: InputFile,
+ ) -> Result<Box<dyn FormatReadBuilder>> { ... }
+ pub fn write_builder(
+ &self,
+ format: DataFileFormat,
+ output: OutputFile,
+ ) -> Result<Box<dyn FormatWriteBuilder>> { ... }
+}
+```
+
+Using `DataFileFormat` as a `HashMap` key requires adding `#[derive(Hash)]` to the enum. That is a non-breaking addition.
+
+The registry is an owned value, not a global static. Tests construct their own. Applications construct one at startup and pass it to scan planners and write orchestrators. For the common case of a single registry for the lifetime of a process, `default_format_registry()` returns a `&'static FormatRegistry` initialized through `OnceLock` on first call.
+
+`read_builder` and `write_builder` return `Err(Error { kind: ErrorKind::FeatureUnsupported, .. })` for unregistered formats. The error message distinguishes two cases: the format is implemented but its feature flag is disabled in this build, or the format has no implementation in this crate.
+
+### Feature flags
+
+Format implementations live in `iceberg::formats::{format}` and are gated behind a feature flag per format: `format-parquet` on by default, `format-orc` when an ORC implementation lands, and so on. The default feature set includes every format the crate implements. Users who build from source and want a smaller binary disable what they do not need.
+
+`FormatRegistry::default()` registers every format enabled by the current feature set at compile time:
+
+```rust
+impl Default for FormatRegistry {
+ fn default() -> Self {
+ let mut registry = Self::new();
+ #[cfg(feature = "format-parquet")]
+ registry.register(Box::new(ParquetFormatModel::new()));
+ #[cfg(feature = "format-orc")]
+ registry.register(Box::new(OrcFormatModel::new()));
+ registry
+ }
+}
+```
+
+Format implementations live inside the `iceberg` crate rather than as separate dependencies. This matches how the Java project keeps its `Parquet`, `Avro`, and `ORC` modules inside the `apache/iceberg` repository. In-tree implementations let the community control each format's quality and release cadence and keep the build graph narrow for downstream consumers. Nothing in the trait design prevents a downstream crate from defining its own `FormatModel` and registering it with a custom `FormatRegistry`, but in-tree is the recommended path for formats intended to ship in iceberg-rust itself.
+
+The `parquet` crate remains an unconditional dependency of `iceberg` for now, because non-format code (page index evaluators, row group metric evaluators, delete file loaders) still uses `parquet` types directly. A later pass gates the `parquet` dependency on the `format-parquet` feature once those callers move to format-agnostic abstractions.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # FormatModel, FormatReadBuilder, FormatWriteBuilder, FormatFileWriter
+├── registry.rs # FormatRegistry, default_format_registry
+└── parquet.rs # ParquetFormatModel, wrapping existing ParquetWriter and ArrowReader
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, and so on, with directory-per-format for anything larger than a few hundred lines.
+
+In the initial implementation, `ParquetWriter`, `ParquetWriterBuilder`, `ArrowReader`, and `ArrowReaderBuilder` stay at their current module paths. `ParquetFormatModel` wraps them. Phase 3 of the Migration Plan moves them into `formats/parquet/` and retires the old paths.
+
+### Design rationale
+
+#### No generic over the data type
+
+Java's `FormatModel<D, S>` uses `D` for Spark `InternalRow`, Flink `RowData`, Arrow `ColumnarBatch`, and other engine-native row types. iceberg-rust has one row type: Arrow `RecordBatch`. Every writer accepts it, and every reader returns it. No format on the near-term queue (ORC, Avro data-file, Vortex, Lance) produces anything other than `RecordBatch` at the Iceberg-facing boundary. No engine integration in iceberg-rust today brings an engine-native row type the way Spark and Flink do in Java.
+
+Adding a `D` parameter today means writing `<RecordBatch>` everywhere the trait is used, for no present caller benefit. If a future format cannot bridge to Arrow, the trait can gain an associated type with a default, which is a semver-compatible addition.
+
+#### No generic over the engine schema
+
+Java's `S` parameter serves Spark's `StructType`, Flink's `RowType`, and the other engine-native schema types. iceberg-rust has `iceberg::spec::Schema` for the logical schema and `arrow::datatypes::SchemaRef` for the physical schema, with conversion in `arrow/schema.rs`. There is no third schema type a generic parameter would serve.
+
+Variant shredding is the concrete use case that Java's `engineProjection(S)` method addresses. `FormatReadBuilder` can later gain an `engine_projection(&mut self, schema: ArrowSchemaRef) -> &mut Self` method with a default no-op implementation. Format implementations that support shredding override it. The parameter is `ArrowSchemaRef`, not a generic `S`, because Arrow is the only engine schema in iceberg-rust.
+
+#### Dynamic dispatch at the registry, static inside the format
+
+Registry lookup is inherently dynamic: the caller has a `DataFileFormat` value at runtime. The read or write hot loop should use static dispatch and inline normally. Splitting the two puts `Box<dyn FormatModel>` at the registry boundary and concrete types everywhere inside the format implementation. Dispatch through the trait object happens once per `read_builder` or `write_builder` call, which is once per file. The hot loop sees concrete types.
+
+`object_store` uses the same split (`dyn ObjectStore` at the boundary, concrete `AsyncRead` and `AsyncWrite` streams returned). `datafusion` uses the same split (`dyn TableProvider` at the boundary, concrete `RecordBatchStream` returned).
+
+#### BoxFuture instead of async fn in traits
+
+`async fn` in traits (stable since Rust 1.75) works for static dispatch. It does not work through `dyn Trait`, because the returned future has an opaque type that trait-object dispatch cannot carry. The `async-trait` crate desugars `async fn` to the same `BoxFuture` a manual implementation produces. Using `BoxFuture` explicitly keeps the future's lifetime visible in the signature and avoids one proc-macro expansion per trait.
+
+#### Feature flags instead of inventory or libloading
+
+Two other registration mechanisms were considered.
+
+The `inventory` crate collects static items at link time through platform-specific linker sections. Each format would register itself with a `submit!` macro. `inventory` drops items silently across static-library boundaries, interacts poorly with analysis tools that do not run a real linker, and registers formats that the caller did not explicitly depend on.
+
+Runtime library loading through `libloading` or similar would allow formats to ship as separate shared libraries. Rust has no stable ABI, so implementations would need to match the host's compiler version and feature flags exactly. Every use would need `unsafe`. PyIceberg has flagged runtime loading as an open question without landing a mechanism.
+
+Feature flags work across `cargo check`, `cargo miri`, cross-compilation, and static linking. `datafusion`, `opendal`, and `reqwest` use the same pattern.
+
+#### RecordBatch as the canonical data type
Review Comment:
Is this the same as the first point?
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
+
+### The read and write builders
+
+`FormatReadBuilder` and `FormatWriteBuilder` configure a single read or write. `FormatModel` produces them, and the caller consumes them with `build`.
+
+```rust
+pub trait FormatReadBuilder: Send {
+ fn project(&mut self, schema: Schema) -> &mut Self;
+ fn filter(&mut self, predicate: BoundPredicate) -> &mut Self;
+ fn split(&mut self, start: u64, length: u64) -> &mut Self;
+ fn case_sensitive(&mut self, case_sensitive: bool) -> &mut Self;
+ fn batch_size(&mut self, batch_size: usize) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<ArrowRecordBatchStream>>;
+}
+
+pub trait FormatWriteBuilder: Send {
+ fn schema(&mut self, schema: Schema) -> &mut Self;
+ fn set(&mut self, key: String, value: String) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<Box<dyn FormatFileWriter>>>;
+}
+```
+
+Both builders take Iceberg `Schema` values. Format implementations convert to physical schemas internally using `schema_to_arrow_schema` from `arrow/schema.rs`. This deliberately departs from Java's `ReadBuilder<D, S>`, which has a separate `engineProjection(S)` method for variant shredding and similar per-engine type mapping. Rust's builders expose one projection surface, and the hook for a variant-shredding "engine projection" is a future-extension point described in "Design rationale."
+
+`FormatWriteBuilder::build` produces a `FormatFileWriter`:
+
+```rust
+pub trait FormatFileWriter: Send {
+ fn write(&mut self, batch: &RecordBatch) -> BoxFuture<'_, Result<()>>;
+ fn close(self: Box<Self>) -> BoxFuture<'static, Result<Vec<DataFileBuilder>>>;
+}
+```
+
+The async methods return `BoxFuture` rather than using `async fn` in traits. The `self: Box<Self>` signature on `build` and `close` lets those methods consume the value while keeping the traits object-safe. Both patterns are forced by the trait-object boundary at the registry. The "BoxFuture instead of async fn in traits" and "Dynamic dispatch at the registry, static inside the format" subsections under "Design rationale" explain why.
+
+### The FormatRegistry
+
+`FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances.
+
+```rust
+pub struct FormatRegistry {
+ models: HashMap<DataFileFormat, Box<dyn FormatModel>>,
+}
+
+impl FormatRegistry {
+ pub fn new() -> Self { ... }
+ pub fn register(&mut self, model: Box<dyn FormatModel>) { ... }
+ pub fn read_builder(
+ &self,
+ format: DataFileFormat,
+ input: InputFile,
+ ) -> Result<Box<dyn FormatReadBuilder>> { ... }
+ pub fn write_builder(
+ &self,
+ format: DataFileFormat,
+ output: OutputFile,
+ ) -> Result<Box<dyn FormatWriteBuilder>> { ... }
+}
+```
+
+Using `DataFileFormat` as a `HashMap` key requires adding `#[derive(Hash)]` to the enum. That is a non-breaking addition.
+
+The registry is an owned value, not a global static. Tests construct their own. Applications construct one at startup and pass it to scan planners and write orchestrators. For the common case of a single registry for the lifetime of a process, `default_format_registry()` returns a `&'static FormatRegistry` initialized through `OnceLock` on first call.
+
+`read_builder` and `write_builder` return `Err(Error { kind: ErrorKind::FeatureUnsupported, .. })` for unregistered formats. The error message distinguishes two cases: the format is implemented but its feature flag is disabled in this build, or the format has no implementation in this crate.
+
+### Feature flags
+
+Format implementations live in `iceberg::formats::{format}` and are gated behind a feature flag per format: `format-parquet` on by default, `format-orc` when an ORC implementation lands, and so on. The default feature set includes every format the crate implements. Users who build from source and want a smaller binary disable what they do not need.
+
+`FormatRegistry::default()` registers every format enabled by the current feature set at compile time:
+
+```rust
+impl Default for FormatRegistry {
+ fn default() -> Self {
+ let mut registry = Self::new();
+ #[cfg(feature = "format-parquet")]
+ registry.register(Box::new(ParquetFormatModel::new()));
+ #[cfg(feature = "format-orc")]
+ registry.register(Box::new(OrcFormatModel::new()));
+ registry
+ }
+}
+```
+
+Format implementations live inside the `iceberg` crate rather than as separate dependencies. This matches how the Java project keeps its `Parquet`, `Avro`, and `ORC` modules inside the `apache/iceberg` repository. In-tree implementations let the community control each format's quality and release cadence and keep the build graph narrow for downstream consumers. Nothing in the trait design prevents a downstream crate from defining its own `FormatModel` and registering it with a custom `FormatRegistry`, but in-tree is the recommended path for formats intended to ship in iceberg-rust itself.
+
+The `parquet` crate remains an unconditional dependency of `iceberg` for now, because non-format code (page index evaluators, row group metric evaluators, delete file loaders) still uses `parquet` types directly. A later pass gates the `parquet` dependency on the `format-parquet` feature once those callers move to format-agnostic abstractions.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # FormatModel, FormatReadBuilder, FormatWriteBuilder, FormatFileWriter
+├── registry.rs # FormatRegistry, default_format_registry
+└── parquet.rs # ParquetFormatModel, wrapping existing ParquetWriter and ArrowReader
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, and so on, with directory-per-format for anything larger than a few hundred lines.
+
+In the initial implementation, `ParquetWriter`, `ParquetWriterBuilder`, `ArrowReader`, and `ArrowReaderBuilder` stay at their current module paths. `ParquetFormatModel` wraps them. Phase 3 of the Migration Plan moves them into `formats/parquet/` and retires the old paths.
+
+### Design rationale
+
+#### No generic over the data type
+
+Java's `FormatModel<D, S>` uses `D` for Spark `InternalRow`, Flink `RowData`, Arrow `ColumnarBatch`, and other engine-native row types. iceberg-rust has one row type: Arrow `RecordBatch`. Every writer accepts it, and every reader returns it. No format on the near-term queue (ORC, Avro data-file, Vortex, Lance) produces anything other than `RecordBatch` at the Iceberg-facing boundary. No engine integration in iceberg-rust today brings an engine-native row type the way Spark and Flink do in Java.
+
+Adding a `D` parameter today means writing `<RecordBatch>` everywhere the trait is used, for no present caller benefit. If a future format cannot bridge to Arrow, the trait can gain an associated type with a default, which is a semver-compatible addition.
Review Comment:
> the trait can gain an associated type with a default
I think we can do it in the initial cut, but I have not investigated the required effort for this
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
+
+### The read and write builders
+
+`FormatReadBuilder` and `FormatWriteBuilder` configure a single read or write. `FormatModel` produces them, and the caller consumes them with `build`.
+
+```rust
+pub trait FormatReadBuilder: Send {
+ fn project(&mut self, schema: Schema) -> &mut Self;
+ fn filter(&mut self, predicate: BoundPredicate) -> &mut Self;
+ fn split(&mut self, start: u64, length: u64) -> &mut Self;
+ fn case_sensitive(&mut self, case_sensitive: bool) -> &mut Self;
+ fn batch_size(&mut self, batch_size: usize) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<ArrowRecordBatchStream>>;
+}
+
+pub trait FormatWriteBuilder: Send {
+ fn schema(&mut self, schema: Schema) -> &mut Self;
+ fn set(&mut self, key: String, value: String) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<Box<dyn FormatFileWriter>>>;
+}
+```
+
+Both builders take Iceberg `Schema` values. Format implementations convert to physical schemas internally using `schema_to_arrow_schema` from `arrow/schema.rs`. This deliberately departs from Java's `ReadBuilder<D, S>`, which has a separate `engineProjection(S)` method for variant shredding and similar per-engine type mapping. Rust's builders expose one projection surface, and the hook for a variant-shredding "engine projection" is a future-extension point described in "Design rationale."
+
+`FormatWriteBuilder::build` produces a `FormatFileWriter`:
+
+```rust
+pub trait FormatFileWriter: Send {
+ fn write(&mut self, batch: &RecordBatch) -> BoxFuture<'_, Result<()>>;
+ fn close(self: Box<Self>) -> BoxFuture<'static, Result<Vec<DataFileBuilder>>>;
+}
+```
+
+The async methods return `BoxFuture` rather than using `async fn` in traits. The `self: Box<Self>` signature on `build` and `close` lets those methods consume the value while keeping the traits object-safe. Both patterns are forced by the trait-object boundary at the registry. The "BoxFuture instead of async fn in traits" and "Dynamic dispatch at the registry, static inside the format" subsections under "Design rationale" explain why.
+
+### The FormatRegistry
+
+`FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances.
+
+```rust
+pub struct FormatRegistry {
+ models: HashMap<DataFileFormat, Box<dyn FormatModel>>,
+}
+
+impl FormatRegistry {
+ pub fn new() -> Self { ... }
+ pub fn register(&mut self, model: Box<dyn FormatModel>) { ... }
+ pub fn read_builder(
+ &self,
+ format: DataFileFormat,
+ input: InputFile,
+ ) -> Result<Box<dyn FormatReadBuilder>> { ... }
+ pub fn write_builder(
+ &self,
+ format: DataFileFormat,
+ output: OutputFile,
+ ) -> Result<Box<dyn FormatWriteBuilder>> { ... }
+}
+```
+
+Using `DataFileFormat` as a `HashMap` key requires adding `#[derive(Hash)]` to the enum. That is a non-breaking addition.
+
+The registry is an owned value, not a global static. Tests construct their own. Applications construct one at startup and pass it to scan planners and write orchestrators. For the common case of a single registry for the lifetime of a process, `default_format_registry()` returns a `&'static FormatRegistry` initialized through `OnceLock` on first call.
Review Comment:
Would love to see more details on scan planner's API changes. Also what about writing?
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
+
+### The read and write builders
+
+`FormatReadBuilder` and `FormatWriteBuilder` configure a single read or write. `FormatModel` produces them, and the caller consumes them with `build`.
+
+```rust
+pub trait FormatReadBuilder: Send {
+ fn project(&mut self, schema: Schema) -> &mut Self;
+ fn filter(&mut self, predicate: BoundPredicate) -> &mut Self;
+ fn split(&mut self, start: u64, length: u64) -> &mut Self;
+ fn case_sensitive(&mut self, case_sensitive: bool) -> &mut Self;
+ fn batch_size(&mut self, batch_size: usize) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<ArrowRecordBatchStream>>;
+}
+
+pub trait FormatWriteBuilder: Send {
+ fn schema(&mut self, schema: Schema) -> &mut Self;
+ fn set(&mut self, key: String, value: String) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<Box<dyn FormatFileWriter>>>;
+}
+```
+
+Both builders take Iceberg `Schema` values. Format implementations convert to physical schemas internally using `schema_to_arrow_schema` from `arrow/schema.rs`. This deliberately departs from Java's `ReadBuilder<D, S>`, which has a separate `engineProjection(S)` method for variant shredding and similar per-engine type mapping. Rust's builders expose one projection surface, and the hook for a variant-shredding "engine projection" is a future-extension point described in "Design rationale."
+
+`FormatWriteBuilder::build` produces a `FormatFileWriter`:
+
+```rust
+pub trait FormatFileWriter: Send {
+ fn write(&mut self, batch: &RecordBatch) -> BoxFuture<'_, Result<()>>;
+ fn close(self: Box<Self>) -> BoxFuture<'static, Result<Vec<DataFileBuilder>>>;
+}
+```
+
+The async methods return `BoxFuture` rather than using `async fn` in traits. The `self: Box<Self>` signature on `build` and `close` lets those methods consume the value while keeping the traits object-safe. Both patterns are forced by the trait-object boundary at the registry. The "BoxFuture instead of async fn in traits" and "Dynamic dispatch at the registry, static inside the format" subsections under "Design rationale" explain why.
+
+### The FormatRegistry
+
+`FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances.
+
+```rust
+pub struct FormatRegistry {
+ models: HashMap<DataFileFormat, Box<dyn FormatModel>>,
+}
+
+impl FormatRegistry {
+ pub fn new() -> Self { ... }
+ pub fn register(&mut self, model: Box<dyn FormatModel>) { ... }
+ pub fn read_builder(
+ &self,
+ format: DataFileFormat,
+ input: InputFile,
+ ) -> Result<Box<dyn FormatReadBuilder>> { ... }
+ pub fn write_builder(
+ &self,
+ format: DataFileFormat,
+ output: OutputFile,
+ ) -> Result<Box<dyn FormatWriteBuilder>> { ... }
+}
+```
+
+Using `DataFileFormat` as a `HashMap` key requires adding `#[derive(Hash)]` to the enum. That is a non-breaking addition.
+
+The registry is an owned value, not a global static. Tests construct their own. Applications construct one at startup and pass it to scan planners and write orchestrators. For the common case of a single registry for the lifetime of a process, `default_format_registry()` returns a `&'static FormatRegistry` initialized through `OnceLock` on first call.
Review Comment:
Or we can inject format registry to Catalog instances directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3465566806
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
Review Comment:
In the Java ecosystem, we found that many existing engines maintain their own in-memory representations and therefore chose not to use the Iceberg-provided readers and writers. Doing so would either require them to adopt the File Format API's in-memory format or introduce an additional conversion step from the File Format API output into their native representation. The former would require a substantial rewrite of the engine to use the API's in-memory model, the latter adds overhead and resulted in performance regressions. Most of the engines considered both unacceptable.
The Rust ecosystem may be different. It is possible that Arrow has become the de facto in-memory representation, making a standardized Arrow-based API a more natural fit. However, there is also a risk of confirmation bias: today we primarily hear from users who are already comfortable consuming Arrow output. We may discover additional use cases and adopters if we provide a more flexible API that does not assume a single in-memory representation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3465583597
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
+
+The registry exposes `reader::<B>(format)` and `writer::<B>(format)`, where `B` is the batch type the engine chose. Errors are eager: if `(format, B)` is not registered, the call fails immediately at scan planning time rather than mid-stream during I/O. This is a runtime contract — the `TypeId` tells the registry what batch type the caller expects, but the type system does not enforce that the reader actually produces that type. A reader that returns the wrong type fails at the first downcast with a clear error. Java makes the same tradeoff with `Class<?>`.
+
+A process-wide default registry is available via `OnceLock`. Custom registries can be provided to `TableScanBuilder` for tests or restricted configurations.
+
+**Default scan loop (hybrid execution model).** The kernel provides a default scan loop that pulls batches from the format reader, checks `ReadResult` to see what the format already handled, and applies any remaining operations (residual filtering, schema evolution, constant injection, delete application). Engines that want full control (DataFusion, Comet) bypass this loop and drive execution themselves — they use the format reader as a stream source and apply their own processing. The `DataBatch` methods are available as a toolbox, not mandated as a pipeline.
+
+**Supporting types:** `ColumnMetrics` stores bounds as typed `Datum` values, matching how `DataFile` already stores bounds. `MetricsConfig` and `MetricsMode` control collection granularity. `FileContent` distinguishes data from equality and position deletes. All config structs and enums are `#[non_exhaustive]`.
+
+### Feature flags
+
+Format implementations are gated behind feature flags (`format-parquet` on by default, `format-orc` when ORC lands). The default feature set includes every implemented format. No configuration is needed for downstream crates using default features. Feature flags exist for crates that want to minimize binary size or compile time. This matches `datafusion`, `opendal`, and `reqwest`.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # Module declarations and public re-exports
+├── traits.rs # DataBatch, FormatReader, FormatWriter,
+│ # FormatFileWriter, ReadOptions, WriteOptions,
+│ # ColumnMetrics, MetricsConfig, MetricsMode,
+│ # FileContent, WriterResult,
+│ # VariantShredding (future extension, not yet wired)
+├── registry.rs # FormatRegistry, default_format_registry()
+└── parquet/
+ ├── mod.rs # Submodule declarations and re-exports
+ ├── model.rs # ParquetArrowModel (implements FormatReader + FormatWriter)
+ ├── reader.rs # Parquet reader internals
+ └── writer.rs # Parquet writer (implements FormatFileWriter)
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, each following the same structure.
+
+### Target crate architecture
+
+The end-state after crate separation (follow-up work, not part of Phase 1). All crates live in the same repository and workspace. Phase 1 ships everything in the existing `iceberg` crate.
+
+```
+iceberg (core kernel) - traits only, no arrow/parquet dependency
+iceberg-arrow - impl DataBatch for RecordBatch
+iceberg-format-parquet - ParquetArrowModel
+iceberg-format-orc - OrcArrowModel
+iceberg-datafusion - TableProvider, depends on above
+```
+
+This separation enables engines to pin their own Arrow version independently. For example, a Comet integration could depend on `iceberg` (the kernel) and provide its own `DataBatch` implementation and format readers using a newer Arrow version, without conflicting with `iceberg-arrow` or `iceberg-format-parquet`.
+
+## Design Rationale
+
+### Three layers, Arrow as default
+
+Java's File Format API has two layers: format and engine. Each format-engine combination requires a direct conversion path, producing an N-by-M matrix. The Java community's performance analysis ([benchmark branch](https://github.com/pvary/iceberg/tree/perf_bench)) measured 20% overhead for double conversion (ORC to Spark ColumnarBatch versus direct VectorizedRowBatch access) and recommended a hybrid: provide an Arrow-based intermediate for interoperability, allow engines to replace it for performance-sensitive paths.
+
+This RFC adopts the hybrid. Arrow is the shipped default. The `DataBatch` trait is the extension point for representations that cannot or should not convert to Arrow. If both format and engine happen to use Arrow (the common case today), there is zero intermediate conversion.
+
+### Why DataBatch is a trait
+
+The kernel ambition ([#1817](https://github.com/apache/iceberg-rust/issues/1817)) requires support for multiple in-memory representations. Arrow version coupling forces lockstep upgrades across the dependency graph — an engine that wants Arrow 60 cannot coexist with a kernel pinned to Arrow 58 unless the kernel is representation-agnostic. `DataBatch` as a trait solves both while preserving Arrow as the shipped default.
+
+The original version of this RFC fixed the data type to `RecordBatch`. Community feedback raised two concerns: that Arrow's dominance in the Rust ecosystem may reflect selection pressure rather than a permanent constraint, and that a kernel-level API should remain open to other ecosystems. This rewrite makes the in-memory type a trait boundary rather than a fixed type.
+
+The name `DataBatch` was chosen over alternatives like `ProcessableBatch` or `ExecutionBatch` because it names what flows through the system (batches of data) rather than what the trait demands (processability). The trait's contract is documented in the method signatures and the TCK; the name stays short and general since it will appear in every engine integration, error message, and type signature.
+
+### Concrete types for processing, trait objects for dispatch
+
+The processing layer works with the concrete batch type the engine chose (like `RecordBatch`). The registry uses trait objects (`Arc<dyn FormatReader>`) because it must hand back a reader for any format at runtime — Parquet for one table, ORC for another — without the caller naming the concrete reader type.
+
+The cost of trait objects is one type check per batch when pulling from the reader stream. The benefit is that the processing layer can fuse operations, inline method calls, and avoid heap allocations for intermediate results. This is the same split `object_store` uses (`dyn ObjectStore` at the boundary, concrete streams returned) and `datafusion` uses (`dyn TableProvider` at the boundary, concrete `RecordBatchStream` returned).
+
+`async fn` in traits does not work through `dyn Trait`. Since the registry returns trait objects, async methods return `BoxFuture`. The allocation happens once per file operation, not per batch.
+
+### Coarse trait methods, rich TCK
+
+`DataBatch::filter` accepts the full `BoundPredicate` tree. `DataBatch::evolve_schema` takes source and target schemas. The trait does not decompose operations into evaluate-per-leaf-then-compose-masks or compute-diff-then-apply-ops. Decomposition forces intermediate materializations. The existing Arrow implementation evaluates predicates as fused closures during I/O, avoiding materialized boolean masks. A decomposed interface would force Arrow to abandon this optimization or work around the trait.
+
+The cost of coarse interfaces is implementor burden. A new `DataBatch` implementation must correctly evaluate recursive predicates and handle schema evolution semantics. The mitigation is the TCK Layer 1 suite, which covers every operator, every primitive type, compound predicates, null handling, and schema evolution scenarios. A new implementation iterates against the TCK until all scenarios pass. Correctness is validated by testing, not constrained by interface shape.
+
+### No engine_schema parameter
+
+Java's `ReadBuilder.engineProjection(S)` and `ModelWriteBuilder.engineSchema(S)` serve two purposes: engine type widening and variant shredding configuration.
+
+Variant shredding is a format-layer concern. The physical shredding layout is defined by the Parquet spec ([VariantShredding.md](https://github.com/apache/parquet-format/blob/master/VariantShredding.md)). The decision of what to shred comes from table properties, data statistics, or engine-provided configuration — but in all cases it is expressed as typed Iceberg-level fields (which paths to extract, as what types), not as an opaque engine-specific schema. Java's Spark implements its own shredding inference internally (PR [#49234](https://github.com/apache/spark/pull/49234), [#52406](https://github.com/apache/spark/pull/52406)); Iceberg later built a shared analyzer (PR [#14297](https://github.com/apache/iceberg/pull/14297)). Neither passes a shredding schema through `engineSchema`. The format handles the physical encoding.
+
+Engine type widening (read an `int` column as `long`) is expressible as a typed `HashMap<i32, PrimitiveType>` on `ReadOptions` when needed.
+
+A type-erased `Box<dyn Any>` was rejected because it allows hidden coupling: a caller going through the registry could pass an object only one format understands. The failure mode is code that works in tests (Parquet-only) and breaks in production (table has ORC files), with a runtime error that surfaces in the reader, not where the wrong object was passed. Both `ReadOptions` and `WriteOptions` are `#[non_exhaustive]`; typed fields are added without a breaking change.
+
+### Options structs, not builder traits
+
+An earlier iteration used `FormatReadBuilder` and `FormatWriteBuilder` traits with fluent `&mut self` methods. Options structs were chosen instead: fewer traits, matches `object_store` precedent (`GetOptions`, `PutOptions`), composable without `Box<dyn>` allocations at each handoff point.
+
+### Divergence from Java
+
+| Aspect | Java | This RFC | Rationale |
+|--------|------|----------|-----------|
+| Type parameters | `FormatModel<D, S>` | Dyn-compatible traits, no type params | Registry requires trait objects |
+| Processing layer | Implicit (engine-specific) | Explicit `DataBatch` trait | Separates concerns, avoids N*M |
+| Registry key | `(FileFormat, Class<?>)` | `(DataFileFormat, TypeId)` | Same concept; batch type is the second dimension |
+| Format traits | Single `FormatModel` | Separate `FormatReader` + `FormatWriter` | A format may support one without the other |
+| Engine schema | Generic parameter `S` | Deferred; typed Iceberg-level fields when needed | Shredding is a format-layer concern, not engine |
+| Residual predicate | Not reported | `ReadResult::residual_predicate` | Kernel needs to know what format handled |
+| Old API coexistence | Kept indefinitely | Hard cutover (pre-1.0) | Clean end state while cost is low |
+
+## Conformance Tests (TCK)
+
+### Layered test architecture
+
+```
+Layer 3: Integration (format x batch x scenario)
+Layer 2: Format (one format, known-good batch type)
+Layer 1: DataBatch (one batch type, no format involvement)
+```
+
+When a Layer 3 test fails, Layer 1 and Layer 2 results identify whether the bug is in the batch implementation, the format implementation, or their interaction.
+
+### Layer 1: DataBatch conformance
+
+Layer 1 validates that a `DataBatch` implementation correctly handles Iceberg semantics without any file format involvement. A new batch type must pass every Layer 1 test before it can participate in the format API.
+
+**Filter correctness:**
+- Every binary operator (Eq, NotEq, Lt, LtEq, Gt, GtEq, StartsWith, NotStartsWith) against every primitive type.
+- Every unary operator (IsNull, NotNull, IsNan, NotNan).
+- Set operators (In, NotIn) with varying set sizes.
+- Compound predicates: AND, OR, NOT at multiple nesting depths.
+- Edge cases: all-null columns, all-matching, none-matching, NaN handling, empty batches.
+- Property-based: for random data and random predicates, verify `filter(p).num_rows()` matches naive row-by-row evaluation.
+
+**Schema evolution correctness:**
+- Type promotion: int to long, float to double, decimal widening.
+- Add column with default value (every primitive type).
+- Add column with null (optional field).
+- Column reordering (same fields, different order).
+- Drop column (project narrows).
+- Combined: promote + add + reorder in one operation.
+
+**Inject constants correctness:**
+- Every primitive Datum type as constant.
+- Multiple constants at once.
+- Zero-row batch.
+
+**Project correctness:**
+- Single column, multiple columns, all columns.
+- Non-existent field ID produces an error.
+- Idempotency: project(ids).project(ids) == project(ids).
+
+**Metrics correctness:**
+- Every primitive type: verify min/max match hand-computed values.
+- Null handling: null_count matches actual nulls.
+- NaN handling: nan_count for float/double.
+- All-null column: bounds are None.
+- Single-value column: lower == upper.
+
+### Layer 2: Format conformance
+
+Layer 2 validates that a format reader/writer correctly round-trips data and produces correct metadata. Uses Arrow RecordBatch (a known-good Layer 1 implementation) as the batch type.
+
+**Phase 1 scenarios (land with initial PR):**
+
+1. Round-trip every `PrimitiveType` with null and non-null values.
+2. Nested types: struct, list, map, combinations.
+3. Projection: write wide, read narrow.
+4. Null handling: all-null, mixed, required versus optional.
+5. Filter pushdown with row group elimination.
+6. Split reading across multiple splits.
+7. Statistics: column sizes, value counts, null counts, NaN counts, min/max bounds.
+8. Metrics modes: None, Counts, Full, Truncate.
+
+**Follow-up scenarios (before 1.0):**
+
+9. Schema evolution with defaults.
+10. Delete files (equality and position).
+11. Metadata columns.
+12. Case sensitivity.
+13. Encryption round-trip.
+14. Name mapping (reading files without embedded field IDs).
+15. Variant ([#2188](https://github.com/apache/iceberg-rust/pull/2188)).
+
+### Merge gates
+
+- New `DataBatch` implementation: must pass all Layer 1 tests.
+- New `FormatReader`/`FormatWriter` implementation: must pass all Layer 2 tests.
+- New format-batch combination: must pass all Layer 3 tests.
+
+## Migration / Rollout Plan
+
+### Phase 1: Land the abstraction (additive)
+
+Add `formats/` module with all traits, registry, and Phase 1 TCK. No existing code changes. The `ParquetArrowModel` implementation wraps the existing `ParquetWriter` and `ArrowReader` internally.
+
+### Phase 2: Migrate internal callers (breaking)
+
+Modify `TableScan` to dispatch through `FormatRegistry`. Modify `DataFileWriterBuilder` and `EqualityDeleteFileWriterBuilder` to use the registry. Update the DataFusion integration.
+
+### Phase 3: Relocate and separate (breaking)
+
+Move Parquet types into `formats/parquet/`. Establish crate boundaries per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819). Add `#[non_exhaustive]` to `DataFileFormat`.
+
+## Breaking Changes
+
+| # | What breaks | Phase | Migration |
+|---|-------------|-------|-----------|
+| 1 | `DataFileWriterBuilder` generic parameters removed | 2 | Use registry-based constructor |
+| 2 | `EqualityDeleteFileWriterBuilder` generic parameters removed | 2 | Use registry-based constructor |
+| 3 | `ParquetWriterBuilder` module path changes | 3 | Update `use` statements |
+| 4 | `ArrowReaderBuilder` path changes, renamed to `ParquetReaderBuilder` | 3 | Update `use` statements |
+| 5 | `DataFileFormat` gains `#[non_exhaustive]` | 3 | Add wildcard arm to `match` blocks |
+| 6 | `TableScan::to_arrow` dispatches through registry (behavior change) | 2 | No change for default registry users |
+
+## Open Questions
+
+### 1. Delete application
+
+**Current lean:** Delete application is a kernel responsibility, not part of `DataBatch` or the format reader. The kernel's default scan loop loads delete files, builds equality delete predicates and positional delete row selections, and applies them after the format reader produces batches. The format reader never sees delete files.
+
+**Tradeoff:** Handling deletes in the kernel means every `DataBatch` implementation gets merge-on-read for free — delete files are resolved before the engine sees batches. The downside is that a format reader cannot fuse delete application with I/O (for example, applying positional deletes during Parquet decompression to skip rows before materializing them). If a format implementation demonstrates that fusing delete application with I/O provides meaningful benefit, a future `ReadOptions` field could pass delete information to the format reader as an optimization hint.
+
+**Question:** Is kernel-owned delete application the right starting point, or should format readers have the option to handle deletes during I/O from the start?
Review Comment:
In V3 only the equality deletes remain. Position deletes are removed and Puffin DVs are used instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3167145996
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
+
+### The read and write builders
+
+`FormatReadBuilder` and `FormatWriteBuilder` configure a single read or write. `FormatModel` produces them, and the caller consumes them with `build`.
+
+```rust
+pub trait FormatReadBuilder: Send {
+ fn project(&mut self, schema: Schema) -> &mut Self;
+ fn filter(&mut self, predicate: BoundPredicate) -> &mut Self;
+ fn split(&mut self, start: u64, length: u64) -> &mut Self;
+ fn case_sensitive(&mut self, case_sensitive: bool) -> &mut Self;
+ fn batch_size(&mut self, batch_size: usize) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<ArrowRecordBatchStream>>;
+}
+
+pub trait FormatWriteBuilder: Send {
+ fn schema(&mut self, schema: Schema) -> &mut Self;
+ fn set(&mut self, key: String, value: String) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<Box<dyn FormatFileWriter>>>;
+}
+```
+
+Both builders take Iceberg `Schema` values. Format implementations convert to physical schemas internally using `schema_to_arrow_schema` from `arrow/schema.rs`. This deliberately departs from Java's `ReadBuilder<D, S>`, which has a separate `engineProjection(S)` method for variant shredding and similar per-engine type mapping. Rust's builders expose one projection surface, and the hook for a variant-shredding "engine projection" is a future-extension point described in "Design rationale."
+
+`FormatWriteBuilder::build` produces a `FormatFileWriter`:
+
+```rust
+pub trait FormatFileWriter: Send {
+ fn write(&mut self, batch: &RecordBatch) -> BoxFuture<'_, Result<()>>;
+ fn close(self: Box<Self>) -> BoxFuture<'static, Result<Vec<DataFileBuilder>>>;
+}
+```
+
+The async methods return `BoxFuture` rather than using `async fn` in traits. The `self: Box<Self>` signature on `build` and `close` lets those methods consume the value while keeping the traits object-safe. Both patterns are forced by the trait-object boundary at the registry. The "BoxFuture instead of async fn in traits" and "Dynamic dispatch at the registry, static inside the format" subsections under "Design rationale" explain why.
+
+### The FormatRegistry
+
+`FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances.
+
+```rust
+pub struct FormatRegistry {
+ models: HashMap<DataFileFormat, Box<dyn FormatModel>>,
+}
+
+impl FormatRegistry {
+ pub fn new() -> Self { ... }
+ pub fn register(&mut self, model: Box<dyn FormatModel>) { ... }
+ pub fn read_builder(
+ &self,
+ format: DataFileFormat,
+ input: InputFile,
+ ) -> Result<Box<dyn FormatReadBuilder>> { ... }
+ pub fn write_builder(
+ &self,
+ format: DataFileFormat,
+ output: OutputFile,
+ ) -> Result<Box<dyn FormatWriteBuilder>> { ... }
+}
+```
+
+Using `DataFileFormat` as a `HashMap` key requires adding `#[derive(Hash)]` to the enum. That is a non-breaking addition.
+
+The registry is an owned value, not a global static. Tests construct their own. Applications construct one at startup and pass it to scan planners and write orchestrators. For the common case of a single registry for the lifetime of a process, `default_format_registry()` returns a `&'static FormatRegistry` initialized through `OnceLock` on first call.
+
+`read_builder` and `write_builder` return `Err(Error { kind: ErrorKind::FeatureUnsupported, .. })` for unregistered formats. The error message distinguishes two cases: the format is implemented but its feature flag is disabled in this build, or the format has no implementation in this crate.
+
+### Feature flags
+
+Format implementations live in `iceberg::formats::{format}` and are gated behind a feature flag per format: `format-parquet` on by default, `format-orc` when an ORC implementation lands, and so on. The default feature set includes every format the crate implements. Users who build from source and want a smaller binary disable what they do not need.
+
+`FormatRegistry::default()` registers every format enabled by the current feature set at compile time:
+
+```rust
+impl Default for FormatRegistry {
+ fn default() -> Self {
+ let mut registry = Self::new();
+ #[cfg(feature = "format-parquet")]
+ registry.register(Box::new(ParquetFormatModel::new()));
+ #[cfg(feature = "format-orc")]
+ registry.register(Box::new(OrcFormatModel::new()));
+ registry
+ }
+}
+```
+
+Format implementations live inside the `iceberg` crate rather than as separate dependencies. This matches how the Java project keeps its `Parquet`, `Avro`, and `ORC` modules inside the `apache/iceberg` repository. In-tree implementations let the community control each format's quality and release cadence and keep the build graph narrow for downstream consumers. Nothing in the trait design prevents a downstream crate from defining its own `FormatModel` and registering it with a custom `FormatRegistry`, but in-tree is the recommended path for formats intended to ship in iceberg-rust itself.
+
+The `parquet` crate remains an unconditional dependency of `iceberg` for now, because non-format code (page index evaluators, row group metric evaluators, delete file loaders) still uses `parquet` types directly. A later pass gates the `parquet` dependency on the `format-parquet` feature once those callers move to format-agnostic abstractions.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # FormatModel, FormatReadBuilder, FormatWriteBuilder, FormatFileWriter
+├── registry.rs # FormatRegistry, default_format_registry
+└── parquet.rs # ParquetFormatModel, wrapping existing ParquetWriter and ArrowReader
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, and so on, with directory-per-format for anything larger than a few hundred lines.
+
+In the initial implementation, `ParquetWriter`, `ParquetWriterBuilder`, `ArrowReader`, and `ArrowReaderBuilder` stay at their current module paths. `ParquetFormatModel` wraps them. Phase 3 of the Migration Plan moves them into `formats/parquet/` and retires the old paths.
+
+### Design rationale
+
+#### No generic over the data type
+
+Java's `FormatModel<D, S>` uses `D` for Spark `InternalRow`, Flink `RowData`, Arrow `ColumnarBatch`, and other engine-native row types. iceberg-rust has one row type: Arrow `RecordBatch`. Every writer accepts it, and every reader returns it. No format on the near-term queue (ORC, Avro data-file, Vortex, Lance) produces anything other than `RecordBatch` at the Iceberg-facing boundary. No engine integration in iceberg-rust today brings an engine-native row type the way Spark and Flink do in Java.
+
+Adding a `D` parameter today means writing `<RecordBatch>` everywhere the trait is used, for no present caller benefit. If a future format cannot bridge to Arrow, the trait can gain an associated type with a default, which is a semver-compatible addition.
+
+#### No generic over the engine schema
+
+Java's `S` parameter serves Spark's `StructType`, Flink's `RowType`, and the other engine-native schema types. iceberg-rust has `iceberg::spec::Schema` for the logical schema and `arrow::datatypes::SchemaRef` for the physical schema, with conversion in `arrow/schema.rs`. There is no third schema type a generic parameter would serve.
+
+Variant shredding is the concrete use case that Java's `engineProjection(S)` method addresses. `FormatReadBuilder` can later gain an `engine_projection(&mut self, schema: ArrowSchemaRef) -> &mut Self` method with a default no-op implementation. Format implementations that support shredding override it. The parameter is `ArrowSchemaRef`, not a generic `S`, because Arrow is the only engine schema in iceberg-rust.
+
+#### Dynamic dispatch at the registry, static inside the format
+
+Registry lookup is inherently dynamic: the caller has a `DataFileFormat` value at runtime. The read or write hot loop should use static dispatch and inline normally. Splitting the two puts `Box<dyn FormatModel>` at the registry boundary and concrete types everywhere inside the format implementation. Dispatch through the trait object happens once per `read_builder` or `write_builder` call, which is once per file. The hot loop sees concrete types.
+
+`object_store` uses the same split (`dyn ObjectStore` at the boundary, concrete `AsyncRead` and `AsyncWrite` streams returned). `datafusion` uses the same split (`dyn TableProvider` at the boundary, concrete `RecordBatchStream` returned).
+
+#### BoxFuture instead of async fn in traits
+
+`async fn` in traits (stable since Rust 1.75) works for static dispatch. It does not work through `dyn Trait`, because the returned future has an opaque type that trait-object dispatch cannot carry. The `async-trait` crate desugars `async fn` to the same `BoxFuture` a manual implementation produces. Using `BoxFuture` explicitly keeps the future's lifetime visible in the signature and avoids one proc-macro expansion per trait.
+
+#### Feature flags instead of inventory or libloading
+
+Two other registration mechanisms were considered.
+
+The `inventory` crate collects static items at link time through platform-specific linker sections. Each format would register itself with a `submit!` macro. `inventory` drops items silently across static-library boundaries, interacts poorly with analysis tools that do not run a real linker, and registers formats that the caller did not explicitly depend on.
+
+Runtime library loading through `libloading` or similar would allow formats to ship as separate shared libraries. Rust has no stable ABI, so implementations would need to match the host's compiler version and feature flags exactly. Every use would need `unsafe`. PyIceberg has flagged runtime loading as an open question without landing a mechanism.
+
+Feature flags work across `cargo check`, `cargo miri`, cross-compilation, and static linking. `datafusion`, `opendal`, and `reqwest` use the same pattern.
+
+#### RecordBatch as the canonical data type
+
+Arrow `RecordBatch` is the interchange format for every Rust data ecosystem iceberg-rust plausibly integrates with. DataFusion consumes it directly. Polars bridges to Arrow with zero-copy conversion for primitive columns. Lance uses its own columnar layout internally and exposes Arrow at read and write boundaries. Every integration and near-term format proposal either produces Arrow or bridges to it.
Review Comment:
I’m skeptical about this.
Is the Rust ecosystem really that centered around Arrow, or are we just seeing survival bias? It feels possible that we mainly hear from Arrow users simply because anyone who wasn’t comfortable with Arrow has already abandoned the Iceberg Rust connector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3166791355
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
Review Comment:
Writing V2 tables (position delete files) is also out of scope. Am I right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3166613715
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
Review Comment:
Wonder if we need to make that abstraction at this stage? Java engines do have different representations that they operate on but to my knowledge arrow is pretty ubiquitous in rust engines / applications?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "Kurtiscwright (via GitHub)" <gi...@apache.org>.
Kurtiscwright commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3163286353
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
Review Comment:
Thank you for the review. In regards to integrating the File Format and Encryption interface, the first thing that jumps out to me is the following question. Does Encryption have different traits or impls based on the File Format in question?
Essentially, will you need `OrcEncryption` that is separate from `ParquetEncryption` modules. If yes then later in the doc when it lists the Format specific traits living in the `formats/` folder. I think the format specific encryption trait can be called from there. Otherwise we can integrate the Encryption libraries directly into the WriterBuilder and ReaderBuilder with respect to Encrypt vs Decrypt calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3169477080
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
Review Comment:
+1 to what Peter says above, for a low-level API like file format, we want to keep the iceberg-rust open to other ecosystems and enable it as a kernel implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "Kurtiscwright (via GitHub)" <gi...@apache.org>.
Kurtiscwright commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3320866114
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
+
+The registry exposes `reader::<B>(format)` and `writer::<B>(format)`, where `B` is the batch type the engine chose. Errors are eager: if `(format, B)` is not registered, the call fails immediately at scan planning time rather than mid-stream during I/O. This is a runtime contract — the `TypeId` tells the registry what batch type the caller expects, but the type system does not enforce that the reader actually produces that type. A reader that returns the wrong type fails at the first downcast with a clear error. Java makes the same tradeoff with `Class<?>`.
+
+A process-wide default registry is available via `OnceLock`. Custom registries can be provided to `TableScanBuilder` for tests or restricted configurations.
+
+**Default scan loop (hybrid execution model).** The kernel provides a default scan loop that pulls batches from the format reader, checks `ReadResult` to see what the format already handled, and applies any remaining operations (residual filtering, schema evolution, constant injection, delete application). Engines that want full control (DataFusion, Comet) bypass this loop and drive execution themselves — they use the format reader as a stream source and apply their own processing. The `DataBatch` methods are available as a toolbox, not mandated as a pipeline.
+
+**Supporting types:** `ColumnMetrics` stores bounds as typed `Datum` values, matching how `DataFile` already stores bounds. `MetricsConfig` and `MetricsMode` control collection granularity. `FileContent` distinguishes data from equality and position deletes. All config structs and enums are `#[non_exhaustive]`.
+
+### Feature flags
+
+Format implementations are gated behind feature flags (`format-parquet` on by default, `format-orc` when ORC lands). The default feature set includes every implemented format. No configuration is needed for downstream crates using default features. Feature flags exist for crates that want to minimize binary size or compile time. This matches `datafusion`, `opendal`, and `reqwest`.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # Module declarations and public re-exports
+├── traits.rs # DataBatch, FormatReader, FormatWriter,
Review Comment:
I didn't feel strongly either way, I can break them out into individual traits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "dannycjones (via GitHub)" <gi...@apache.org>.
dannycjones commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3460700261
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
Review Comment:
What about tables that have a mix of file formats? For example, table was created with ORC and then migrated to Parquet but there are still ORC data files out there.
I don't see a strong reason to temporarily limit scope here, at least not in the RFC, so I think we need to either call it out as a long-term limitation or we scope it in?
I can imagine a use case where we end up writing a rewrite data file procedure that takes all non-Parquet files and rewrites them as Parquet (or pick your favorite file format).
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
Review Comment:
I think it would be good to define what we mean by "session".
Presumably, this is a particular instance of a table? I'm not sure.
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
Review Comment:
first mention of TCK, would be good to define upfront.
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
Review Comment:
Should we have `DataFileBuilder`, or should we have our own struct at this layer? We expose a bunch of things (like sequence number) which the implementer has no business modifying.
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
+
+The registry exposes `reader::<B>(format)` and `writer::<B>(format)`, where `B` is the batch type the engine chose. Errors are eager: if `(format, B)` is not registered, the call fails immediately at scan planning time rather than mid-stream during I/O. This is a runtime contract — the `TypeId` tells the registry what batch type the caller expects, but the type system does not enforce that the reader actually produces that type. A reader that returns the wrong type fails at the first downcast with a clear error. Java makes the same tradeoff with `Class<?>`.
+
+A process-wide default registry is available via `OnceLock`. Custom registries can be provided to `TableScanBuilder` for tests or restricted configurations.
+
+**Default scan loop (hybrid execution model).** The kernel provides a default scan loop that pulls batches from the format reader, checks `ReadResult` to see what the format already handled, and applies any remaining operations (residual filtering, schema evolution, constant injection, delete application). Engines that want full control (DataFusion, Comet) bypass this loop and drive execution themselves — they use the format reader as a stream source and apply their own processing. The `DataBatch` methods are available as a toolbox, not mandated as a pipeline.
+
+**Supporting types:** `ColumnMetrics` stores bounds as typed `Datum` values, matching how `DataFile` already stores bounds. `MetricsConfig` and `MetricsMode` control collection granularity. `FileContent` distinguishes data from equality and position deletes. All config structs and enums are `#[non_exhaustive]`.
+
+### Feature flags
+
+Format implementations are gated behind feature flags (`format-parquet` on by default, `format-orc` when ORC lands). The default feature set includes every implemented format. No configuration is needed for downstream crates using default features. Feature flags exist for crates that want to minimize binary size or compile time. This matches `datafusion`, `opendal`, and `reqwest`.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # Module declarations and public re-exports
+├── traits.rs # DataBatch, FormatReader, FormatWriter,
+│ # FormatFileWriter, ReadOptions, WriteOptions,
+│ # ColumnMetrics, MetricsConfig, MetricsMode,
+│ # FileContent, WriterResult,
+│ # VariantShredding (future extension, not yet wired)
+├── registry.rs # FormatRegistry, default_format_registry()
+└── parquet/
+ ├── mod.rs # Submodule declarations and re-exports
+ ├── model.rs # ParquetArrowModel (implements FormatReader + FormatWriter)
+ ├── reader.rs # Parquet reader internals
+ └── writer.rs # Parquet writer (implements FormatFileWriter)
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, each following the same structure.
+
+### Target crate architecture
+
+The end-state after crate separation (follow-up work, not part of Phase 1). All crates live in the same repository and workspace. Phase 1 ships everything in the existing `iceberg` crate.
+
+```
+iceberg (core kernel) - traits only, no arrow/parquet dependency
+iceberg-arrow - impl DataBatch for RecordBatch
Review Comment:
Should we have a convention on in-memory representation too?
For example: `iceberg-repr-arrow`, `iceberg-repr-vortex`
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
Review Comment:
Is there an option where we use generics for the in-memory format? I'm struggling to think of use cases right now where we want to dynamically change in-memory format.
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
Review Comment:
We separate these two traits to allow to implement reading but not writing... Can we discuss the motivation a little, but more importantly: what's the failure mode look like if someone tries to append to a table using a file format that's unsupported?
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
Review Comment:
I think we need to include V2 in scope. (V3? I agree we should try and limit the scope creep somehow - but we have plenty of people interested and working on V3.)
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
+
+The registry exposes `reader::<B>(format)` and `writer::<B>(format)`, where `B` is the batch type the engine chose. Errors are eager: if `(format, B)` is not registered, the call fails immediately at scan planning time rather than mid-stream during I/O. This is a runtime contract — the `TypeId` tells the registry what batch type the caller expects, but the type system does not enforce that the reader actually produces that type. A reader that returns the wrong type fails at the first downcast with a clear error. Java makes the same tradeoff with `Class<?>`.
+
+A process-wide default registry is available via `OnceLock`. Custom registries can be provided to `TableScanBuilder` for tests or restricted configurations.
+
+**Default scan loop (hybrid execution model).** The kernel provides a default scan loop that pulls batches from the format reader, checks `ReadResult` to see what the format already handled, and applies any remaining operations (residual filtering, schema evolution, constant injection, delete application). Engines that want full control (DataFusion, Comet) bypass this loop and drive execution themselves — they use the format reader as a stream source and apply their own processing. The `DataBatch` methods are available as a toolbox, not mandated as a pipeline.
+
+**Supporting types:** `ColumnMetrics` stores bounds as typed `Datum` values, matching how `DataFile` already stores bounds. `MetricsConfig` and `MetricsMode` control collection granularity. `FileContent` distinguishes data from equality and position deletes. All config structs and enums are `#[non_exhaustive]`.
+
+### Feature flags
+
+Format implementations are gated behind feature flags (`format-parquet` on by default, `format-orc` when ORC lands). The default feature set includes every implemented format. No configuration is needed for downstream crates using default features. Feature flags exist for crates that want to minimize binary size or compile time. This matches `datafusion`, `opendal`, and `reqwest`.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # Module declarations and public re-exports
+├── traits.rs # DataBatch, FormatReader, FormatWriter,
+│ # FormatFileWriter, ReadOptions, WriteOptions,
+│ # ColumnMetrics, MetricsConfig, MetricsMode,
+│ # FileContent, WriterResult,
+│ # VariantShredding (future extension, not yet wired)
+├── registry.rs # FormatRegistry, default_format_registry()
+└── parquet/
+ ├── mod.rs # Submodule declarations and re-exports
+ ├── model.rs # ParquetArrowModel (implements FormatReader + FormatWriter)
+ ├── reader.rs # Parquet reader internals
+ └── writer.rs # Parquet writer (implements FormatFileWriter)
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, each following the same structure.
+
+### Target crate architecture
+
+The end-state after crate separation (follow-up work, not part of Phase 1). All crates live in the same repository and workspace. Phase 1 ships everything in the existing `iceberg` crate.
+
+```
+iceberg (core kernel) - traits only, no arrow/parquet dependency
+iceberg-arrow - impl DataBatch for RecordBatch
+iceberg-format-parquet - ParquetArrowModel
+iceberg-format-orc - OrcArrowModel
+iceberg-datafusion - TableProvider, depends on above
+```
+
+This separation enables engines to pin their own Arrow version independently. For example, a Comet integration could depend on `iceberg` (the kernel) and provide its own `DataBatch` implementation and format readers using a newer Arrow version, without conflicting with `iceberg-arrow` or `iceberg-format-parquet`.
+
+## Design Rationale
+
+### Three layers, Arrow as default
+
+Java's File Format API has two layers: format and engine. Each format-engine combination requires a direct conversion path, producing an N-by-M matrix. The Java community's performance analysis ([benchmark branch](https://github.com/pvary/iceberg/tree/perf_bench)) measured 20% overhead for double conversion (ORC to Spark ColumnarBatch versus direct VectorizedRowBatch access) and recommended a hybrid: provide an Arrow-based intermediate for interoperability, allow engines to replace it for performance-sensitive paths.
+
+This RFC adopts the hybrid. Arrow is the shipped default. The `DataBatch` trait is the extension point for representations that cannot or should not convert to Arrow. If both format and engine happen to use Arrow (the common case today), there is zero intermediate conversion.
+
+### Why DataBatch is a trait
+
+The kernel ambition ([#1817](https://github.com/apache/iceberg-rust/issues/1817)) requires support for multiple in-memory representations. Arrow version coupling forces lockstep upgrades across the dependency graph — an engine that wants Arrow 60 cannot coexist with a kernel pinned to Arrow 58 unless the kernel is representation-agnostic. `DataBatch` as a trait solves both while preserving Arrow as the shipped default.
+
+The original version of this RFC fixed the data type to `RecordBatch`. Community feedback raised two concerns: that Arrow's dominance in the Rust ecosystem may reflect selection pressure rather than a permanent constraint, and that a kernel-level API should remain open to other ecosystems. This rewrite makes the in-memory type a trait boundary rather than a fixed type.
+
+The name `DataBatch` was chosen over alternatives like `ProcessableBatch` or `ExecutionBatch` because it names what flows through the system (batches of data) rather than what the trait demands (processability). The trait's contract is documented in the method signatures and the TCK; the name stays short and general since it will appear in every engine integration, error message, and type signature.
+
+### Concrete types for processing, trait objects for dispatch
+
+The processing layer works with the concrete batch type the engine chose (like `RecordBatch`). The registry uses trait objects (`Arc<dyn FormatReader>`) because it must hand back a reader for any format at runtime — Parquet for one table, ORC for another — without the caller naming the concrete reader type.
+
+The cost of trait objects is one type check per batch when pulling from the reader stream. The benefit is that the processing layer can fuse operations, inline method calls, and avoid heap allocations for intermediate results. This is the same split `object_store` uses (`dyn ObjectStore` at the boundary, concrete streams returned) and `datafusion` uses (`dyn TableProvider` at the boundary, concrete `RecordBatchStream` returned).
+
+`async fn` in traits does not work through `dyn Trait`. Since the registry returns trait objects, async methods return `BoxFuture`. The allocation happens once per file operation, not per batch.
+
+### Coarse trait methods, rich TCK
+
+`DataBatch::filter` accepts the full `BoundPredicate` tree. `DataBatch::evolve_schema` takes source and target schemas. The trait does not decompose operations into evaluate-per-leaf-then-compose-masks or compute-diff-then-apply-ops. Decomposition forces intermediate materializations. The existing Arrow implementation evaluates predicates as fused closures during I/O, avoiding materialized boolean masks. A decomposed interface would force Arrow to abandon this optimization or work around the trait.
+
+The cost of coarse interfaces is implementor burden. A new `DataBatch` implementation must correctly evaluate recursive predicates and handle schema evolution semantics. The mitigation is the TCK Layer 1 suite, which covers every operator, every primitive type, compound predicates, null handling, and schema evolution scenarios. A new implementation iterates against the TCK until all scenarios pass. Correctness is validated by testing, not constrained by interface shape.
+
+### No engine_schema parameter
+
+Java's `ReadBuilder.engineProjection(S)` and `ModelWriteBuilder.engineSchema(S)` serve two purposes: engine type widening and variant shredding configuration.
+
+Variant shredding is a format-layer concern. The physical shredding layout is defined by the Parquet spec ([VariantShredding.md](https://github.com/apache/parquet-format/blob/master/VariantShredding.md)). The decision of what to shred comes from table properties, data statistics, or engine-provided configuration — but in all cases it is expressed as typed Iceberg-level fields (which paths to extract, as what types), not as an opaque engine-specific schema. Java's Spark implements its own shredding inference internally (PR [#49234](https://github.com/apache/spark/pull/49234), [#52406](https://github.com/apache/spark/pull/52406)); Iceberg later built a shared analyzer (PR [#14297](https://github.com/apache/iceberg/pull/14297)). Neither passes a shredding schema through `engineSchema`. The format handles the physical encoding.
+
+Engine type widening (read an `int` column as `long`) is expressible as a typed `HashMap<i32, PrimitiveType>` on `ReadOptions` when needed.
+
+A type-erased `Box<dyn Any>` was rejected because it allows hidden coupling: a caller going through the registry could pass an object only one format understands. The failure mode is code that works in tests (Parquet-only) and breaks in production (table has ORC files), with a runtime error that surfaces in the reader, not where the wrong object was passed. Both `ReadOptions` and `WriteOptions` are `#[non_exhaustive]`; typed fields are added without a breaking change.
+
+### Options structs, not builder traits
+
+An earlier iteration used `FormatReadBuilder` and `FormatWriteBuilder` traits with fluent `&mut self` methods. Options structs were chosen instead: fewer traits, matches `object_store` precedent (`GetOptions`, `PutOptions`), composable without `Box<dyn>` allocations at each handoff point.
+
+### Divergence from Java
+
+| Aspect | Java | This RFC | Rationale |
+|--------|------|----------|-----------|
+| Type parameters | `FormatModel<D, S>` | Dyn-compatible traits, no type params | Registry requires trait objects |
+| Processing layer | Implicit (engine-specific) | Explicit `DataBatch` trait | Separates concerns, avoids N*M |
+| Registry key | `(FileFormat, Class<?>)` | `(DataFileFormat, TypeId)` | Same concept; batch type is the second dimension |
+| Format traits | Single `FormatModel` | Separate `FormatReader` + `FormatWriter` | A format may support one without the other |
+| Engine schema | Generic parameter `S` | Deferred; typed Iceberg-level fields when needed | Shredding is a format-layer concern, not engine |
+| Residual predicate | Not reported | `ReadResult::residual_predicate` | Kernel needs to know what format handled |
+| Old API coexistence | Kept indefinitely | Hard cutover (pre-1.0) | Clean end state while cost is low |
+
+## Conformance Tests (TCK)
+
+### Layered test architecture
+
+```
+Layer 3: Integration (format x batch x scenario)
+Layer 2: Format (one format, known-good batch type)
+Layer 1: DataBatch (one batch type, no format involvement)
+```
+
+When a Layer 3 test fails, Layer 1 and Layer 2 results identify whether the bug is in the batch implementation, the format implementation, or their interaction.
+
+### Layer 1: DataBatch conformance
+
+Layer 1 validates that a `DataBatch` implementation correctly handles Iceberg semantics without any file format involvement. A new batch type must pass every Layer 1 test before it can participate in the format API.
+
+**Filter correctness:**
+- Every binary operator (Eq, NotEq, Lt, LtEq, Gt, GtEq, StartsWith, NotStartsWith) against every primitive type.
+- Every unary operator (IsNull, NotNull, IsNan, NotNan).
+- Set operators (In, NotIn) with varying set sizes.
+- Compound predicates: AND, OR, NOT at multiple nesting depths.
+- Edge cases: all-null columns, all-matching, none-matching, NaN handling, empty batches.
+- Property-based: for random data and random predicates, verify `filter(p).num_rows()` matches naive row-by-row evaluation.
+
+**Schema evolution correctness:**
+- Type promotion: int to long, float to double, decimal widening.
+- Add column with default value (every primitive type).
+- Add column with null (optional field).
+- Column reordering (same fields, different order).
+- Drop column (project narrows).
+- Combined: promote + add + reorder in one operation.
+
+**Inject constants correctness:**
+- Every primitive Datum type as constant.
+- Multiple constants at once.
+- Zero-row batch.
+
+**Project correctness:**
+- Single column, multiple columns, all columns.
+- Non-existent field ID produces an error.
+- Idempotency: project(ids).project(ids) == project(ids).
+
+**Metrics correctness:**
+- Every primitive type: verify min/max match hand-computed values.
+- Null handling: null_count matches actual nulls.
+- NaN handling: nan_count for float/double.
+- All-null column: bounds are None.
+- Single-value column: lower == upper.
+
+### Layer 2: Format conformance
+
+Layer 2 validates that a format reader/writer correctly round-trips data and produces correct metadata. Uses Arrow RecordBatch (a known-good Layer 1 implementation) as the batch type.
+
+**Phase 1 scenarios (land with initial PR):**
+
+1. Round-trip every `PrimitiveType` with null and non-null values.
+2. Nested types: struct, list, map, combinations.
+3. Projection: write wide, read narrow.
+4. Null handling: all-null, mixed, required versus optional.
+5. Filter pushdown with row group elimination.
+6. Split reading across multiple splits.
+7. Statistics: column sizes, value counts, null counts, NaN counts, min/max bounds.
+8. Metrics modes: None, Counts, Full, Truncate.
+
+**Follow-up scenarios (before 1.0):**
+
+9. Schema evolution with defaults.
+10. Delete files (equality and position).
+11. Metadata columns.
+12. Case sensitivity.
+13. Encryption round-trip.
+14. Name mapping (reading files without embedded field IDs).
+15. Variant ([#2188](https://github.com/apache/iceberg-rust/pull/2188)).
+
+### Merge gates
+
+- New `DataBatch` implementation: must pass all Layer 1 tests.
+- New `FormatReader`/`FormatWriter` implementation: must pass all Layer 2 tests.
+- New format-batch combination: must pass all Layer 3 tests.
+
+## Migration / Rollout Plan
+
+### Phase 1: Land the abstraction (additive)
+
+Add `formats/` module with all traits, registry, and Phase 1 TCK. No existing code changes. The `ParquetArrowModel` implementation wraps the existing `ParquetWriter` and `ArrowReader` internally.
+
+### Phase 2: Migrate internal callers (breaking)
+
+Modify `TableScan` to dispatch through `FormatRegistry`. Modify `DataFileWriterBuilder` and `EqualityDeleteFileWriterBuilder` to use the registry. Update the DataFusion integration.
+
+### Phase 3: Relocate and separate (breaking)
+
+Move Parquet types into `formats/parquet/`. Establish crate boundaries per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819). Add `#[non_exhaustive]` to `DataFileFormat`.
+
+## Breaking Changes
+
+| # | What breaks | Phase | Migration |
+|---|-------------|-------|-----------|
+| 1 | `DataFileWriterBuilder` generic parameters removed | 2 | Use registry-based constructor |
+| 2 | `EqualityDeleteFileWriterBuilder` generic parameters removed | 2 | Use registry-based constructor |
+| 3 | `ParquetWriterBuilder` module path changes | 3 | Update `use` statements |
+| 4 | `ArrowReaderBuilder` path changes, renamed to `ParquetReaderBuilder` | 3 | Update `use` statements |
+| 5 | `DataFileFormat` gains `#[non_exhaustive]` | 3 | Add wildcard arm to `match` blocks |
+| 6 | `TableScan::to_arrow` dispatches through registry (behavior change) | 2 | No change for default registry users |
+
+## Open Questions
+
+### 1. Delete application
+
+**Current lean:** Delete application is a kernel responsibility, not part of `DataBatch` or the format reader. The kernel's default scan loop loads delete files, builds equality delete predicates and positional delete row selections, and applies them after the format reader produces batches. The format reader never sees delete files.
+
+**Tradeoff:** Handling deletes in the kernel means every `DataBatch` implementation gets merge-on-read for free — delete files are resolved before the engine sees batches. The downside is that a format reader cannot fuse delete application with I/O (for example, applying positional deletes during Parquet decompression to skip rows before materializing them). If a format implementation demonstrates that fusing delete application with I/O provides meaningful benefit, a future `ReadOptions` field could pass delete information to the format reader as an optimization hint.
+
+**Question:** Is kernel-owned delete application the right starting point, or should format readers have the option to handle deletes during I/O from the start?
Review Comment:
What about delete file format? iceberg-java allows you to configure Parquet, ORC, Avro for delete files - should this be in scope?
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
Review Comment:
Have we discussed the risk of significant refactoring with ongoing feature development - in particular the V3 work?
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
Review Comment:
It's a bit unclear what `TypeId` is. I'll take a look at some of the reference draft PRs, but I don't understand it well from the RFC.
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
+
+The registry exposes `reader::<B>(format)` and `writer::<B>(format)`, where `B` is the batch type the engine chose. Errors are eager: if `(format, B)` is not registered, the call fails immediately at scan planning time rather than mid-stream during I/O. This is a runtime contract — the `TypeId` tells the registry what batch type the caller expects, but the type system does not enforce that the reader actually produces that type. A reader that returns the wrong type fails at the first downcast with a clear error. Java makes the same tradeoff with `Class<?>`.
+
+A process-wide default registry is available via `OnceLock`. Custom registries can be provided to `TableScanBuilder` for tests or restricted configurations.
+
+**Default scan loop (hybrid execution model).** The kernel provides a default scan loop that pulls batches from the format reader, checks `ReadResult` to see what the format already handled, and applies any remaining operations (residual filtering, schema evolution, constant injection, delete application). Engines that want full control (DataFusion, Comet) bypass this loop and drive execution themselves — they use the format reader as a stream source and apply their own processing. The `DataBatch` methods are available as a toolbox, not mandated as a pipeline.
+
+**Supporting types:** `ColumnMetrics` stores bounds as typed `Datum` values, matching how `DataFile` already stores bounds. `MetricsConfig` and `MetricsMode` control collection granularity. `FileContent` distinguishes data from equality and position deletes. All config structs and enums are `#[non_exhaustive]`.
+
+### Feature flags
+
+Format implementations are gated behind feature flags (`format-parquet` on by default, `format-orc` when ORC lands). The default feature set includes every implemented format. No configuration is needed for downstream crates using default features. Feature flags exist for crates that want to minimize binary size or compile time. This matches `datafusion`, `opendal`, and `reqwest`.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # Module declarations and public re-exports
+├── traits.rs # DataBatch, FormatReader, FormatWriter,
+│ # FormatFileWriter, ReadOptions, WriteOptions,
+│ # ColumnMetrics, MetricsConfig, MetricsMode,
+│ # FileContent, WriterResult,
+│ # VariantShredding (future extension, not yet wired)
+├── registry.rs # FormatRegistry, default_format_registry()
+└── parquet/
+ ├── mod.rs # Submodule declarations and re-exports
+ ├── model.rs # ParquetArrowModel (implements FormatReader + FormatWriter)
+ ├── reader.rs # Parquet reader internals
+ └── writer.rs # Parquet writer (implements FormatFileWriter)
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, each following the same structure.
+
+### Target crate architecture
+
+The end-state after crate separation (follow-up work, not part of Phase 1). All crates live in the same repository and workspace. Phase 1 ships everything in the existing `iceberg` crate.
+
+```
+iceberg (core kernel) - traits only, no arrow/parquet dependency
+iceberg-arrow - impl DataBatch for RecordBatch
+iceberg-format-parquet - ParquetArrowModel
+iceberg-format-orc - OrcArrowModel
+iceberg-datafusion - TableProvider, depends on above
+```
+
+This separation enables engines to pin their own Arrow version independently. For example, a Comet integration could depend on `iceberg` (the kernel) and provide its own `DataBatch` implementation and format readers using a newer Arrow version, without conflicting with `iceberg-arrow` or `iceberg-format-parquet`.
+
+## Design Rationale
+
+### Three layers, Arrow as default
+
+Java's File Format API has two layers: format and engine. Each format-engine combination requires a direct conversion path, producing an N-by-M matrix. The Java community's performance analysis ([benchmark branch](https://github.com/pvary/iceberg/tree/perf_bench)) measured 20% overhead for double conversion (ORC to Spark ColumnarBatch versus direct VectorizedRowBatch access) and recommended a hybrid: provide an Arrow-based intermediate for interoperability, allow engines to replace it for performance-sensitive paths.
+
+This RFC adopts the hybrid. Arrow is the shipped default. The `DataBatch` trait is the extension point for representations that cannot or should not convert to Arrow. If both format and engine happen to use Arrow (the common case today), there is zero intermediate conversion.
+
+### Why DataBatch is a trait
+
+The kernel ambition ([#1817](https://github.com/apache/iceberg-rust/issues/1817)) requires support for multiple in-memory representations. Arrow version coupling forces lockstep upgrades across the dependency graph — an engine that wants Arrow 60 cannot coexist with a kernel pinned to Arrow 58 unless the kernel is representation-agnostic. `DataBatch` as a trait solves both while preserving Arrow as the shipped default.
+
+The original version of this RFC fixed the data type to `RecordBatch`. Community feedback raised two concerns: that Arrow's dominance in the Rust ecosystem may reflect selection pressure rather than a permanent constraint, and that a kernel-level API should remain open to other ecosystems. This rewrite makes the in-memory type a trait boundary rather than a fixed type.
+
+The name `DataBatch` was chosen over alternatives like `ProcessableBatch` or `ExecutionBatch` because it names what flows through the system (batches of data) rather than what the trait demands (processability). The trait's contract is documented in the method signatures and the TCK; the name stays short and general since it will appear in every engine integration, error message, and type signature.
+
+### Concrete types for processing, trait objects for dispatch
+
+The processing layer works with the concrete batch type the engine chose (like `RecordBatch`). The registry uses trait objects (`Arc<dyn FormatReader>`) because it must hand back a reader for any format at runtime — Parquet for one table, ORC for another — without the caller naming the concrete reader type.
+
+The cost of trait objects is one type check per batch when pulling from the reader stream. The benefit is that the processing layer can fuse operations, inline method calls, and avoid heap allocations for intermediate results. This is the same split `object_store` uses (`dyn ObjectStore` at the boundary, concrete streams returned) and `datafusion` uses (`dyn TableProvider` at the boundary, concrete `RecordBatchStream` returned).
+
+`async fn` in traits does not work through `dyn Trait`. Since the registry returns trait objects, async methods return `BoxFuture`. The allocation happens once per file operation, not per batch.
+
+### Coarse trait methods, rich TCK
+
+`DataBatch::filter` accepts the full `BoundPredicate` tree. `DataBatch::evolve_schema` takes source and target schemas. The trait does not decompose operations into evaluate-per-leaf-then-compose-masks or compute-diff-then-apply-ops. Decomposition forces intermediate materializations. The existing Arrow implementation evaluates predicates as fused closures during I/O, avoiding materialized boolean masks. A decomposed interface would force Arrow to abandon this optimization or work around the trait.
+
+The cost of coarse interfaces is implementor burden. A new `DataBatch` implementation must correctly evaluate recursive predicates and handle schema evolution semantics. The mitigation is the TCK Layer 1 suite, which covers every operator, every primitive type, compound predicates, null handling, and schema evolution scenarios. A new implementation iterates against the TCK until all scenarios pass. Correctness is validated by testing, not constrained by interface shape.
+
+### No engine_schema parameter
+
+Java's `ReadBuilder.engineProjection(S)` and `ModelWriteBuilder.engineSchema(S)` serve two purposes: engine type widening and variant shredding configuration.
+
+Variant shredding is a format-layer concern. The physical shredding layout is defined by the Parquet spec ([VariantShredding.md](https://github.com/apache/parquet-format/blob/master/VariantShredding.md)). The decision of what to shred comes from table properties, data statistics, or engine-provided configuration — but in all cases it is expressed as typed Iceberg-level fields (which paths to extract, as what types), not as an opaque engine-specific schema. Java's Spark implements its own shredding inference internally (PR [#49234](https://github.com/apache/spark/pull/49234), [#52406](https://github.com/apache/spark/pull/52406)); Iceberg later built a shared analyzer (PR [#14297](https://github.com/apache/iceberg/pull/14297)). Neither passes a shredding schema through `engineSchema`. The format handles the physical encoding.
+
+Engine type widening (read an `int` column as `long`) is expressible as a typed `HashMap<i32, PrimitiveType>` on `ReadOptions` when needed.
+
+A type-erased `Box<dyn Any>` was rejected because it allows hidden coupling: a caller going through the registry could pass an object only one format understands. The failure mode is code that works in tests (Parquet-only) and breaks in production (table has ORC files), with a runtime error that surfaces in the reader, not where the wrong object was passed. Both `ReadOptions` and `WriteOptions` are `#[non_exhaustive]`; typed fields are added without a breaking change.
+
+### Options structs, not builder traits
+
+An earlier iteration used `FormatReadBuilder` and `FormatWriteBuilder` traits with fluent `&mut self` methods. Options structs were chosen instead: fewer traits, matches `object_store` precedent (`GetOptions`, `PutOptions`), composable without `Box<dyn>` allocations at each handoff point.
+
+### Divergence from Java
+
+| Aspect | Java | This RFC | Rationale |
+|--------|------|----------|-----------|
+| Type parameters | `FormatModel<D, S>` | Dyn-compatible traits, no type params | Registry requires trait objects |
+| Processing layer | Implicit (engine-specific) | Explicit `DataBatch` trait | Separates concerns, avoids N*M |
+| Registry key | `(FileFormat, Class<?>)` | `(DataFileFormat, TypeId)` | Same concept; batch type is the second dimension |
+| Format traits | Single `FormatModel` | Separate `FormatReader` + `FormatWriter` | A format may support one without the other |
+| Engine schema | Generic parameter `S` | Deferred; typed Iceberg-level fields when needed | Shredding is a format-layer concern, not engine |
+| Residual predicate | Not reported | `ReadResult::residual_predicate` | Kernel needs to know what format handled |
Review Comment:
nit
```suggestion
| Residual predicate | Not reported | `ReadResult::residual_predicate` | Kernel needs to know what format handled to avoid applying predicates twice |
```
is the Java implementation at risk of applying predicates twice?
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
+
+The registry exposes `reader::<B>(format)` and `writer::<B>(format)`, where `B` is the batch type the engine chose. Errors are eager: if `(format, B)` is not registered, the call fails immediately at scan planning time rather than mid-stream during I/O. This is a runtime contract — the `TypeId` tells the registry what batch type the caller expects, but the type system does not enforce that the reader actually produces that type. A reader that returns the wrong type fails at the first downcast with a clear error. Java makes the same tradeoff with `Class<?>`.
+
+A process-wide default registry is available via `OnceLock`. Custom registries can be provided to `TableScanBuilder` for tests or restricted configurations.
+
+**Default scan loop (hybrid execution model).** The kernel provides a default scan loop that pulls batches from the format reader, checks `ReadResult` to see what the format already handled, and applies any remaining operations (residual filtering, schema evolution, constant injection, delete application). Engines that want full control (DataFusion, Comet) bypass this loop and drive execution themselves — they use the format reader as a stream source and apply their own processing. The `DataBatch` methods are available as a toolbox, not mandated as a pipeline.
+
+**Supporting types:** `ColumnMetrics` stores bounds as typed `Datum` values, matching how `DataFile` already stores bounds. `MetricsConfig` and `MetricsMode` control collection granularity. `FileContent` distinguishes data from equality and position deletes. All config structs and enums are `#[non_exhaustive]`.
+
+### Feature flags
+
+Format implementations are gated behind feature flags (`format-parquet` on by default, `format-orc` when ORC lands). The default feature set includes every implemented format. No configuration is needed for downstream crates using default features. Feature flags exist for crates that want to minimize binary size or compile time. This matches `datafusion`, `opendal`, and `reqwest`.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # Module declarations and public re-exports
+├── traits.rs # DataBatch, FormatReader, FormatWriter,
+│ # FormatFileWriter, ReadOptions, WriteOptions,
+│ # ColumnMetrics, MetricsConfig, MetricsMode,
+│ # FileContent, WriterResult,
+│ # VariantShredding (future extension, not yet wired)
+├── registry.rs # FormatRegistry, default_format_registry()
+└── parquet/
+ ├── mod.rs # Submodule declarations and re-exports
+ ├── model.rs # ParquetArrowModel (implements FormatReader + FormatWriter)
+ ├── reader.rs # Parquet reader internals
+ └── writer.rs # Parquet writer (implements FormatFileWriter)
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, each following the same structure.
+
+### Target crate architecture
+
+The end-state after crate separation (follow-up work, not part of Phase 1). All crates live in the same repository and workspace. Phase 1 ships everything in the existing `iceberg` crate.
+
+```
+iceberg (core kernel) - traits only, no arrow/parquet dependency
+iceberg-arrow - impl DataBatch for RecordBatch
+iceberg-format-parquet - ParquetArrowModel
+iceberg-format-orc - OrcArrowModel
+iceberg-datafusion - TableProvider, depends on above
+```
+
+This separation enables engines to pin their own Arrow version independently. For example, a Comet integration could depend on `iceberg` (the kernel) and provide its own `DataBatch` implementation and format readers using a newer Arrow version, without conflicting with `iceberg-arrow` or `iceberg-format-parquet`.
+
+## Design Rationale
+
+### Three layers, Arrow as default
+
+Java's File Format API has two layers: format and engine. Each format-engine combination requires a direct conversion path, producing an N-by-M matrix. The Java community's performance analysis ([benchmark branch](https://github.com/pvary/iceberg/tree/perf_bench)) measured 20% overhead for double conversion (ORC to Spark ColumnarBatch versus direct VectorizedRowBatch access) and recommended a hybrid: provide an Arrow-based intermediate for interoperability, allow engines to replace it for performance-sensitive paths.
+
+This RFC adopts the hybrid. Arrow is the shipped default. The `DataBatch` trait is the extension point for representations that cannot or should not convert to Arrow. If both format and engine happen to use Arrow (the common case today), there is zero intermediate conversion.
+
+### Why DataBatch is a trait
+
+The kernel ambition ([#1817](https://github.com/apache/iceberg-rust/issues/1817)) requires support for multiple in-memory representations. Arrow version coupling forces lockstep upgrades across the dependency graph — an engine that wants Arrow 60 cannot coexist with a kernel pinned to Arrow 58 unless the kernel is representation-agnostic. `DataBatch` as a trait solves both while preserving Arrow as the shipped default.
+
+The original version of this RFC fixed the data type to `RecordBatch`. Community feedback raised two concerns: that Arrow's dominance in the Rust ecosystem may reflect selection pressure rather than a permanent constraint, and that a kernel-level API should remain open to other ecosystems. This rewrite makes the in-memory type a trait boundary rather than a fixed type.
+
+The name `DataBatch` was chosen over alternatives like `ProcessableBatch` or `ExecutionBatch` because it names what flows through the system (batches of data) rather than what the trait demands (processability). The trait's contract is documented in the method signatures and the TCK; the name stays short and general since it will appear in every engine integration, error message, and type signature.
+
+### Concrete types for processing, trait objects for dispatch
+
+The processing layer works with the concrete batch type the engine chose (like `RecordBatch`). The registry uses trait objects (`Arc<dyn FormatReader>`) because it must hand back a reader for any format at runtime — Parquet for one table, ORC for another — without the caller naming the concrete reader type.
+
+The cost of trait objects is one type check per batch when pulling from the reader stream. The benefit is that the processing layer can fuse operations, inline method calls, and avoid heap allocations for intermediate results. This is the same split `object_store` uses (`dyn ObjectStore` at the boundary, concrete streams returned) and `datafusion` uses (`dyn TableProvider` at the boundary, concrete `RecordBatchStream` returned).
+
+`async fn` in traits does not work through `dyn Trait`. Since the registry returns trait objects, async methods return `BoxFuture`. The allocation happens once per file operation, not per batch.
+
+### Coarse trait methods, rich TCK
+
+`DataBatch::filter` accepts the full `BoundPredicate` tree. `DataBatch::evolve_schema` takes source and target schemas. The trait does not decompose operations into evaluate-per-leaf-then-compose-masks or compute-diff-then-apply-ops. Decomposition forces intermediate materializations. The existing Arrow implementation evaluates predicates as fused closures during I/O, avoiding materialized boolean masks. A decomposed interface would force Arrow to abandon this optimization or work around the trait.
+
+The cost of coarse interfaces is implementor burden. A new `DataBatch` implementation must correctly evaluate recursive predicates and handle schema evolution semantics. The mitigation is the TCK Layer 1 suite, which covers every operator, every primitive type, compound predicates, null handling, and schema evolution scenarios. A new implementation iterates against the TCK until all scenarios pass. Correctness is validated by testing, not constrained by interface shape.
+
+### No engine_schema parameter
+
+Java's `ReadBuilder.engineProjection(S)` and `ModelWriteBuilder.engineSchema(S)` serve two purposes: engine type widening and variant shredding configuration.
+
+Variant shredding is a format-layer concern. The physical shredding layout is defined by the Parquet spec ([VariantShredding.md](https://github.com/apache/parquet-format/blob/master/VariantShredding.md)). The decision of what to shred comes from table properties, data statistics, or engine-provided configuration — but in all cases it is expressed as typed Iceberg-level fields (which paths to extract, as what types), not as an opaque engine-specific schema. Java's Spark implements its own shredding inference internally (PR [#49234](https://github.com/apache/spark/pull/49234), [#52406](https://github.com/apache/spark/pull/52406)); Iceberg later built a shared analyzer (PR [#14297](https://github.com/apache/iceberg/pull/14297)). Neither passes a shredding schema through `engineSchema`. The format handles the physical encoding.
+
+Engine type widening (read an `int` column as `long`) is expressible as a typed `HashMap<i32, PrimitiveType>` on `ReadOptions` when needed.
+
+A type-erased `Box<dyn Any>` was rejected because it allows hidden coupling: a caller going through the registry could pass an object only one format understands. The failure mode is code that works in tests (Parquet-only) and breaks in production (table has ORC files), with a runtime error that surfaces in the reader, not where the wrong object was passed. Both `ReadOptions` and `WriteOptions` are `#[non_exhaustive]`; typed fields are added without a breaking change.
+
+### Options structs, not builder traits
+
+An earlier iteration used `FormatReadBuilder` and `FormatWriteBuilder` traits with fluent `&mut self` methods. Options structs were chosen instead: fewer traits, matches `object_store` precedent (`GetOptions`, `PutOptions`), composable without `Box<dyn>` allocations at each handoff point.
+
+### Divergence from Java
+
+| Aspect | Java | This RFC | Rationale |
+|--------|------|----------|-----------|
+| Type parameters | `FormatModel<D, S>` | Dyn-compatible traits, no type params | Registry requires trait objects |
+| Processing layer | Implicit (engine-specific) | Explicit `DataBatch` trait | Separates concerns, avoids N*M |
+| Registry key | `(FileFormat, Class<?>)` | `(DataFileFormat, TypeId)` | Same concept; batch type is the second dimension |
+| Format traits | Single `FormatModel` | Separate `FormatReader` + `FormatWriter` | A format may support one without the other |
+| Engine schema | Generic parameter `S` | Deferred; typed Iceberg-level fields when needed | Shredding is a format-layer concern, not engine |
+| Residual predicate | Not reported | `ReadResult::residual_predicate` | Kernel needs to know what format handled |
+| Old API coexistence | Kept indefinitely | Hard cutover (pre-1.0) | Clean end state while cost is low |
+
+## Conformance Tests (TCK)
+
+### Layered test architecture
+
+```
+Layer 3: Integration (format x batch x scenario)
+Layer 2: Format (one format, known-good batch type)
+Layer 1: DataBatch (one batch type, no format involvement)
+```
+
+When a Layer 3 test fails, Layer 1 and Layer 2 results identify whether the bug is in the batch implementation, the format implementation, or their interaction.
+
+### Layer 1: DataBatch conformance
+
+Layer 1 validates that a `DataBatch` implementation correctly handles Iceberg semantics without any file format involvement. A new batch type must pass every Layer 1 test before it can participate in the format API.
+
+**Filter correctness:**
+- Every binary operator (Eq, NotEq, Lt, LtEq, Gt, GtEq, StartsWith, NotStartsWith) against every primitive type.
+- Every unary operator (IsNull, NotNull, IsNan, NotNan).
+- Set operators (In, NotIn) with varying set sizes.
+- Compound predicates: AND, OR, NOT at multiple nesting depths.
+- Edge cases: all-null columns, all-matching, none-matching, NaN handling, empty batches.
+- Property-based: for random data and random predicates, verify `filter(p).num_rows()` matches naive row-by-row evaluation.
+
+**Schema evolution correctness:**
+- Type promotion: int to long, float to double, decimal widening.
+- Add column with default value (every primitive type).
+- Add column with null (optional field).
+- Column reordering (same fields, different order).
+- Drop column (project narrows).
+- Combined: promote + add + reorder in one operation.
+
+**Inject constants correctness:**
+- Every primitive Datum type as constant.
+- Multiple constants at once.
+- Zero-row batch.
+
+**Project correctness:**
+- Single column, multiple columns, all columns.
+- Non-existent field ID produces an error.
+- Idempotency: project(ids).project(ids) == project(ids).
+
+**Metrics correctness:**
+- Every primitive type: verify min/max match hand-computed values.
+- Null handling: null_count matches actual nulls.
+- NaN handling: nan_count for float/double.
+- All-null column: bounds are None.
+- Single-value column: lower == upper.
+
+### Layer 2: Format conformance
+
+Layer 2 validates that a format reader/writer correctly round-trips data and produces correct metadata. Uses Arrow RecordBatch (a known-good Layer 1 implementation) as the batch type.
+
+**Phase 1 scenarios (land with initial PR):**
+
+1. Round-trip every `PrimitiveType` with null and non-null values.
+2. Nested types: struct, list, map, combinations.
+3. Projection: write wide, read narrow.
+4. Null handling: all-null, mixed, required versus optional.
+5. Filter pushdown with row group elimination.
+6. Split reading across multiple splits.
+7. Statistics: column sizes, value counts, null counts, NaN counts, min/max bounds.
+8. Metrics modes: None, Counts, Full, Truncate.
+
+**Follow-up scenarios (before 1.0):**
+
+9. Schema evolution with defaults.
+10. Delete files (equality and position).
+11. Metadata columns.
+12. Case sensitivity.
+13. Encryption round-trip.
+14. Name mapping (reading files without embedded field IDs).
+15. Variant ([#2188](https://github.com/apache/iceberg-rust/pull/2188)).
+
+### Merge gates
+
+- New `DataBatch` implementation: must pass all Layer 1 tests.
+- New `FormatReader`/`FormatWriter` implementation: must pass all Layer 2 tests.
+- New format-batch combination: must pass all Layer 3 tests.
+
+## Migration / Rollout Plan
+
+### Phase 1: Land the abstraction (additive)
+
+Add `formats/` module with all traits, registry, and Phase 1 TCK. No existing code changes. The `ParquetArrowModel` implementation wraps the existing `ParquetWriter` and `ArrowReader` internally.
+
+### Phase 2: Migrate internal callers (breaking)
+
+Modify `TableScan` to dispatch through `FormatRegistry`. Modify `DataFileWriterBuilder` and `EqualityDeleteFileWriterBuilder` to use the registry. Update the DataFusion integration.
+
+### Phase 3: Relocate and separate (breaking)
+
+Move Parquet types into `formats/parquet/`. Establish crate boundaries per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819). Add `#[non_exhaustive]` to `DataFileFormat`.
+
+## Breaking Changes
+
+| # | What breaks | Phase | Migration |
+|---|-------------|-------|-----------|
+| 1 | `DataFileWriterBuilder` generic parameters removed | 2 | Use registry-based constructor |
+| 2 | `EqualityDeleteFileWriterBuilder` generic parameters removed | 2 | Use registry-based constructor |
+| 3 | `ParquetWriterBuilder` module path changes | 3 | Update `use` statements |
+| 4 | `ArrowReaderBuilder` path changes, renamed to `ParquetReaderBuilder` | 3 | Update `use` statements |
+| 5 | `DataFileFormat` gains `#[non_exhaustive]` | 3 | Add wildcard arm to `match` blocks |
+| 6 | `TableScan::to_arrow` dispatches through registry (behavior change) | 2 | No change for default registry users |
+
+## Open Questions
+
+### 1. Delete application
+
+**Current lean:** Delete application is a kernel responsibility, not part of `DataBatch` or the format reader. The kernel's default scan loop loads delete files, builds equality delete predicates and positional delete row selections, and applies them after the format reader produces batches. The format reader never sees delete files.
+
+**Tradeoff:** Handling deletes in the kernel means every `DataBatch` implementation gets merge-on-read for free — delete files are resolved before the engine sees batches. The downside is that a format reader cannot fuse delete application with I/O (for example, applying positional deletes during Parquet decompression to skip rows before materializing them). If a format implementation demonstrates that fusing delete application with I/O provides meaningful benefit, a future `ReadOptions` field could pass delete information to the format reader as an optimization hint.
+
+**Question:** Is kernel-owned delete application the right starting point, or should format readers have the option to handle deletes during I/O from the start?
+
+### 2. Registry scope
+
+**Current lean:** Process-wide default via `OnceLock`, with `TableScanBuilder::with_format_registry` as override.
+
+**Tradeoff:** A process-wide default is convenient for single-configuration applications but makes testing harder (tests cannot isolate registry state without global mutation). Explicit construction is more testable and embeddable but adds a parameter to every scan and write call.
+
+**Question:** Is the process-wide default acceptable, or should the registry always be provided explicitly?
+
+### 3. Crate separation timing
+
+**Current lean:** Phase 1 ships in the existing `iceberg` crate. Separation is follow-up per [#1817](https://github.com/apache/iceberg-rust/issues/1817).
+
+**Question:** Should crate boundaries be established in Phase 1 or deferred?
+
Review Comment:
Very happy to see it as a follow-up - let's establish the new APIs and be happy with them first, then break it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3465579485
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
+
+The registry exposes `reader::<B>(format)` and `writer::<B>(format)`, where `B` is the batch type the engine chose. Errors are eager: if `(format, B)` is not registered, the call fails immediately at scan planning time rather than mid-stream during I/O. This is a runtime contract — the `TypeId` tells the registry what batch type the caller expects, but the type system does not enforce that the reader actually produces that type. A reader that returns the wrong type fails at the first downcast with a clear error. Java makes the same tradeoff with `Class<?>`.
+
+A process-wide default registry is available via `OnceLock`. Custom registries can be provided to `TableScanBuilder` for tests or restricted configurations.
+
+**Default scan loop (hybrid execution model).** The kernel provides a default scan loop that pulls batches from the format reader, checks `ReadResult` to see what the format already handled, and applies any remaining operations (residual filtering, schema evolution, constant injection, delete application). Engines that want full control (DataFusion, Comet) bypass this loop and drive execution themselves — they use the format reader as a stream source and apply their own processing. The `DataBatch` methods are available as a toolbox, not mandated as a pipeline.
+
+**Supporting types:** `ColumnMetrics` stores bounds as typed `Datum` values, matching how `DataFile` already stores bounds. `MetricsConfig` and `MetricsMode` control collection granularity. `FileContent` distinguishes data from equality and position deletes. All config structs and enums are `#[non_exhaustive]`.
+
+### Feature flags
+
+Format implementations are gated behind feature flags (`format-parquet` on by default, `format-orc` when ORC lands). The default feature set includes every implemented format. No configuration is needed for downstream crates using default features. Feature flags exist for crates that want to minimize binary size or compile time. This matches `datafusion`, `opendal`, and `reqwest`.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # Module declarations and public re-exports
+├── traits.rs # DataBatch, FormatReader, FormatWriter,
+│ # FormatFileWriter, ReadOptions, WriteOptions,
+│ # ColumnMetrics, MetricsConfig, MetricsMode,
+│ # FileContent, WriterResult,
+│ # VariantShredding (future extension, not yet wired)
+├── registry.rs # FormatRegistry, default_format_registry()
+└── parquet/
+ ├── mod.rs # Submodule declarations and re-exports
+ ├── model.rs # ParquetArrowModel (implements FormatReader + FormatWriter)
+ ├── reader.rs # Parquet reader internals
+ └── writer.rs # Parquet writer (implements FormatFileWriter)
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, each following the same structure.
+
+### Target crate architecture
+
+The end-state after crate separation (follow-up work, not part of Phase 1). All crates live in the same repository and workspace. Phase 1 ships everything in the existing `iceberg` crate.
+
+```
+iceberg (core kernel) - traits only, no arrow/parquet dependency
+iceberg-arrow - impl DataBatch for RecordBatch
+iceberg-format-parquet - ParquetArrowModel
+iceberg-format-orc - OrcArrowModel
+iceberg-datafusion - TableProvider, depends on above
+```
+
+This separation enables engines to pin their own Arrow version independently. For example, a Comet integration could depend on `iceberg` (the kernel) and provide its own `DataBatch` implementation and format readers using a newer Arrow version, without conflicting with `iceberg-arrow` or `iceberg-format-parquet`.
+
+## Design Rationale
+
+### Three layers, Arrow as default
+
+Java's File Format API has two layers: format and engine. Each format-engine combination requires a direct conversion path, producing an N-by-M matrix. The Java community's performance analysis ([benchmark branch](https://github.com/pvary/iceberg/tree/perf_bench)) measured 20% overhead for double conversion (ORC to Spark ColumnarBatch versus direct VectorizedRowBatch access) and recommended a hybrid: provide an Arrow-based intermediate for interoperability, allow engines to replace it for performance-sensitive paths.
+
+This RFC adopts the hybrid. Arrow is the shipped default. The `DataBatch` trait is the extension point for representations that cannot or should not convert to Arrow. If both format and engine happen to use Arrow (the common case today), there is zero intermediate conversion.
+
+### Why DataBatch is a trait
+
+The kernel ambition ([#1817](https://github.com/apache/iceberg-rust/issues/1817)) requires support for multiple in-memory representations. Arrow version coupling forces lockstep upgrades across the dependency graph — an engine that wants Arrow 60 cannot coexist with a kernel pinned to Arrow 58 unless the kernel is representation-agnostic. `DataBatch` as a trait solves both while preserving Arrow as the shipped default.
+
+The original version of this RFC fixed the data type to `RecordBatch`. Community feedback raised two concerns: that Arrow's dominance in the Rust ecosystem may reflect selection pressure rather than a permanent constraint, and that a kernel-level API should remain open to other ecosystems. This rewrite makes the in-memory type a trait boundary rather than a fixed type.
+
+The name `DataBatch` was chosen over alternatives like `ProcessableBatch` or `ExecutionBatch` because it names what flows through the system (batches of data) rather than what the trait demands (processability). The trait's contract is documented in the method signatures and the TCK; the name stays short and general since it will appear in every engine integration, error message, and type signature.
+
+### Concrete types for processing, trait objects for dispatch
+
+The processing layer works with the concrete batch type the engine chose (like `RecordBatch`). The registry uses trait objects (`Arc<dyn FormatReader>`) because it must hand back a reader for any format at runtime — Parquet for one table, ORC for another — without the caller naming the concrete reader type.
+
+The cost of trait objects is one type check per batch when pulling from the reader stream. The benefit is that the processing layer can fuse operations, inline method calls, and avoid heap allocations for intermediate results. This is the same split `object_store` uses (`dyn ObjectStore` at the boundary, concrete streams returned) and `datafusion` uses (`dyn TableProvider` at the boundary, concrete `RecordBatchStream` returned).
+
+`async fn` in traits does not work through `dyn Trait`. Since the registry returns trait objects, async methods return `BoxFuture`. The allocation happens once per file operation, not per batch.
+
+### Coarse trait methods, rich TCK
+
+`DataBatch::filter` accepts the full `BoundPredicate` tree. `DataBatch::evolve_schema` takes source and target schemas. The trait does not decompose operations into evaluate-per-leaf-then-compose-masks or compute-diff-then-apply-ops. Decomposition forces intermediate materializations. The existing Arrow implementation evaluates predicates as fused closures during I/O, avoiding materialized boolean masks. A decomposed interface would force Arrow to abandon this optimization or work around the trait.
+
+The cost of coarse interfaces is implementor burden. A new `DataBatch` implementation must correctly evaluate recursive predicates and handle schema evolution semantics. The mitigation is the TCK Layer 1 suite, which covers every operator, every primitive type, compound predicates, null handling, and schema evolution scenarios. A new implementation iterates against the TCK until all scenarios pass. Correctness is validated by testing, not constrained by interface shape.
+
+### No engine_schema parameter
+
+Java's `ReadBuilder.engineProjection(S)` and `ModelWriteBuilder.engineSchema(S)` serve two purposes: engine type widening and variant shredding configuration.
+
+Variant shredding is a format-layer concern. The physical shredding layout is defined by the Parquet spec ([VariantShredding.md](https://github.com/apache/parquet-format/blob/master/VariantShredding.md)). The decision of what to shred comes from table properties, data statistics, or engine-provided configuration — but in all cases it is expressed as typed Iceberg-level fields (which paths to extract, as what types), not as an opaque engine-specific schema. Java's Spark implements its own shredding inference internally (PR [#49234](https://github.com/apache/spark/pull/49234), [#52406](https://github.com/apache/spark/pull/52406)); Iceberg later built a shared analyzer (PR [#14297](https://github.com/apache/iceberg/pull/14297)). Neither passes a shredding schema through `engineSchema`. The format handles the physical encoding.
+
+Engine type widening (read an `int` column as `long`) is expressible as a typed `HashMap<i32, PrimitiveType>` on `ReadOptions` when needed.
+
+A type-erased `Box<dyn Any>` was rejected because it allows hidden coupling: a caller going through the registry could pass an object only one format understands. The failure mode is code that works in tests (Parquet-only) and breaks in production (table has ORC files), with a runtime error that surfaces in the reader, not where the wrong object was passed. Both `ReadOptions` and `WriteOptions` are `#[non_exhaustive]`; typed fields are added without a breaking change.
+
+### Options structs, not builder traits
+
+An earlier iteration used `FormatReadBuilder` and `FormatWriteBuilder` traits with fluent `&mut self` methods. Options structs were chosen instead: fewer traits, matches `object_store` precedent (`GetOptions`, `PutOptions`), composable without `Box<dyn>` allocations at each handoff point.
+
+### Divergence from Java
+
+| Aspect | Java | This RFC | Rationale |
+|--------|------|----------|-----------|
+| Type parameters | `FormatModel<D, S>` | Dyn-compatible traits, no type params | Registry requires trait objects |
+| Processing layer | Implicit (engine-specific) | Explicit `DataBatch` trait | Separates concerns, avoids N*M |
+| Registry key | `(FileFormat, Class<?>)` | `(DataFileFormat, TypeId)` | Same concept; batch type is the second dimension |
+| Format traits | Single `FormatModel` | Separate `FormatReader` + `FormatWriter` | A format may support one without the other |
+| Engine schema | Generic parameter `S` | Deferred; typed Iceberg-level fields when needed | Shredding is a format-layer concern, not engine |
+| Residual predicate | Not reported | `ReadResult::residual_predicate` | Kernel needs to know what format handled |
Review Comment:
In Java we don't expect the File Format to fully apply any of the filters, so in practice the Java code always applies the filter fully after reading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3166931319
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
Review Comment:
I would prefer to keep the data type open. My reasoning:
- Hive: We built our own integration earlier, when there was no straightforward way to convert to Hive’s internal format. As a result, Hive still uses its own readers and writers and effectively has to implement each feature from scratch. They might eventually move to the API if we introduce a major reader feature they don’t want to reimplement.
- Impala: similar situation. Impala also relies on its own readers and writers. They do more custom logic there, so they might not adopt the File Format API directly, but the trade‑off is that they end up lagging behind on new integrations.
- Arrow dependency: for engines that don’t natively operate on Arrow, converting FileFormat → Arrow → engine format is possible but can be quite expensive performance wise. If we only expose Arrow, we may lose those engines because of the overhead.
- Rust ecosystem: I’m aware of Rust‑based engines that don’t use Arrow internally. It would be great to keep the door open for them as well.
How flexible are the Rust API contracts in this regard? If we limited the scope now, how hard would it be to support additional (non‑Arrow) data types later? I understand the desire to keep things simple for now, but I suspect we’ll eventually need a path that allows more than just Arrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3167188798
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
Review Comment:
Also this could be an interesting read around the spec: https://iceberg.apache.org/docs/nightly/encryption/#appendix-internals-overview
@ggershinsky described the current Java state like this:
>Currently we can encrypt Parquet and Avro data files. Parquet files are encrypted via the native PME mechanism, while Avro files are encrypted via Iceberg's AES GCM Streams https://iceberg.apache.org/gcm-stream-spec/. But both kinds of encryption require the same parameters: key and aadPrefix. The best doc, I guess, is https://iceberg.apache.org/docs/nightly/encryption/#appendix-internals-overview .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3167156644
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
+
+### The read and write builders
+
+`FormatReadBuilder` and `FormatWriteBuilder` configure a single read or write. `FormatModel` produces them, and the caller consumes them with `build`.
+
+```rust
+pub trait FormatReadBuilder: Send {
+ fn project(&mut self, schema: Schema) -> &mut Self;
+ fn filter(&mut self, predicate: BoundPredicate) -> &mut Self;
+ fn split(&mut self, start: u64, length: u64) -> &mut Self;
+ fn case_sensitive(&mut self, case_sensitive: bool) -> &mut Self;
+ fn batch_size(&mut self, batch_size: usize) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<ArrowRecordBatchStream>>;
+}
+
+pub trait FormatWriteBuilder: Send {
+ fn schema(&mut self, schema: Schema) -> &mut Self;
+ fn set(&mut self, key: String, value: String) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<Box<dyn FormatFileWriter>>>;
+}
+```
+
+Both builders take Iceberg `Schema` values. Format implementations convert to physical schemas internally using `schema_to_arrow_schema` from `arrow/schema.rs`. This deliberately departs from Java's `ReadBuilder<D, S>`, which has a separate `engineProjection(S)` method for variant shredding and similar per-engine type mapping. Rust's builders expose one projection surface, and the hook for a variant-shredding "engine projection" is a future-extension point described in "Design rationale."
+
+`FormatWriteBuilder::build` produces a `FormatFileWriter`:
+
+```rust
+pub trait FormatFileWriter: Send {
+ fn write(&mut self, batch: &RecordBatch) -> BoxFuture<'_, Result<()>>;
+ fn close(self: Box<Self>) -> BoxFuture<'static, Result<Vec<DataFileBuilder>>>;
+}
+```
+
+The async methods return `BoxFuture` rather than using `async fn` in traits. The `self: Box<Self>` signature on `build` and `close` lets those methods consume the value while keeping the traits object-safe. Both patterns are forced by the trait-object boundary at the registry. The "BoxFuture instead of async fn in traits" and "Dynamic dispatch at the registry, static inside the format" subsections under "Design rationale" explain why.
+
+### The FormatRegistry
+
+`FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances.
+
+```rust
+pub struct FormatRegistry {
+ models: HashMap<DataFileFormat, Box<dyn FormatModel>>,
+}
+
+impl FormatRegistry {
+ pub fn new() -> Self { ... }
+ pub fn register(&mut self, model: Box<dyn FormatModel>) { ... }
+ pub fn read_builder(
+ &self,
+ format: DataFileFormat,
+ input: InputFile,
+ ) -> Result<Box<dyn FormatReadBuilder>> { ... }
+ pub fn write_builder(
+ &self,
+ format: DataFileFormat,
+ output: OutputFile,
+ ) -> Result<Box<dyn FormatWriteBuilder>> { ... }
+}
+```
+
+Using `DataFileFormat` as a `HashMap` key requires adding `#[derive(Hash)]` to the enum. That is a non-breaking addition.
+
+The registry is an owned value, not a global static. Tests construct their own. Applications construct one at startup and pass it to scan planners and write orchestrators. For the common case of a single registry for the lifetime of a process, `default_format_registry()` returns a `&'static FormatRegistry` initialized through `OnceLock` on first call.
+
+`read_builder` and `write_builder` return `Err(Error { kind: ErrorKind::FeatureUnsupported, .. })` for unregistered formats. The error message distinguishes two cases: the format is implemented but its feature flag is disabled in this build, or the format has no implementation in this crate.
+
+### Feature flags
+
+Format implementations live in `iceberg::formats::{format}` and are gated behind a feature flag per format: `format-parquet` on by default, `format-orc` when an ORC implementation lands, and so on. The default feature set includes every format the crate implements. Users who build from source and want a smaller binary disable what they do not need.
+
+`FormatRegistry::default()` registers every format enabled by the current feature set at compile time:
+
+```rust
+impl Default for FormatRegistry {
+ fn default() -> Self {
+ let mut registry = Self::new();
+ #[cfg(feature = "format-parquet")]
+ registry.register(Box::new(ParquetFormatModel::new()));
+ #[cfg(feature = "format-orc")]
+ registry.register(Box::new(OrcFormatModel::new()));
+ registry
+ }
+}
+```
+
+Format implementations live inside the `iceberg` crate rather than as separate dependencies. This matches how the Java project keeps its `Parquet`, `Avro`, and `ORC` modules inside the `apache/iceberg` repository. In-tree implementations let the community control each format's quality and release cadence and keep the build graph narrow for downstream consumers. Nothing in the trait design prevents a downstream crate from defining its own `FormatModel` and registering it with a custom `FormatRegistry`, but in-tree is the recommended path for formats intended to ship in iceberg-rust itself.
+
+The `parquet` crate remains an unconditional dependency of `iceberg` for now, because non-format code (page index evaluators, row group metric evaluators, delete file loaders) still uses `parquet` types directly. A later pass gates the `parquet` dependency on the `format-parquet` feature once those callers move to format-agnostic abstractions.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # FormatModel, FormatReadBuilder, FormatWriteBuilder, FormatFileWriter
+├── registry.rs # FormatRegistry, default_format_registry
+└── parquet.rs # ParquetFormatModel, wrapping existing ParquetWriter and ArrowReader
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, and so on, with directory-per-format for anything larger than a few hundred lines.
+
+In the initial implementation, `ParquetWriter`, `ParquetWriterBuilder`, `ArrowReader`, and `ArrowReaderBuilder` stay at their current module paths. `ParquetFormatModel` wraps them. Phase 3 of the Migration Plan moves them into `formats/parquet/` and retires the old paths.
+
+### Design rationale
+
+#### No generic over the data type
+
+Java's `FormatModel<D, S>` uses `D` for Spark `InternalRow`, Flink `RowData`, Arrow `ColumnarBatch`, and other engine-native row types. iceberg-rust has one row type: Arrow `RecordBatch`. Every writer accepts it, and every reader returns it. No format on the near-term queue (ORC, Avro data-file, Vortex, Lance) produces anything other than `RecordBatch` at the Iceberg-facing boundary. No engine integration in iceberg-rust today brings an engine-native row type the way Spark and Flink do in Java.
+
+Adding a `D` parameter today means writing `<RecordBatch>` everywhere the trait is used, for no present caller benefit. If a future format cannot bridge to Arrow, the trait can gain an associated type with a default, which is a semver-compatible addition.
+
+#### No generic over the engine schema
+
+Java's `S` parameter serves Spark's `StructType`, Flink's `RowType`, and the other engine-native schema types. iceberg-rust has `iceberg::spec::Schema` for the logical schema and `arrow::datatypes::SchemaRef` for the physical schema, with conversion in `arrow/schema.rs`. There is no third schema type a generic parameter would serve.
+
+Variant shredding is the concrete use case that Java's `engineProjection(S)` method addresses. `FormatReadBuilder` can later gain an `engine_projection(&mut self, schema: ArrowSchemaRef) -> &mut Self` method with a default no-op implementation. Format implementations that support shredding override it. The parameter is `ArrowSchemaRef`, not a generic `S`, because Arrow is the only engine schema in iceberg-rust.
+
+#### Dynamic dispatch at the registry, static inside the format
+
+Registry lookup is inherently dynamic: the caller has a `DataFileFormat` value at runtime. The read or write hot loop should use static dispatch and inline normally. Splitting the two puts `Box<dyn FormatModel>` at the registry boundary and concrete types everywhere inside the format implementation. Dispatch through the trait object happens once per `read_builder` or `write_builder` call, which is once per file. The hot loop sees concrete types.
+
+`object_store` uses the same split (`dyn ObjectStore` at the boundary, concrete `AsyncRead` and `AsyncWrite` streams returned). `datafusion` uses the same split (`dyn TableProvider` at the boundary, concrete `RecordBatchStream` returned).
+
+#### BoxFuture instead of async fn in traits
+
+`async fn` in traits (stable since Rust 1.75) works for static dispatch. It does not work through `dyn Trait`, because the returned future has an opaque type that trait-object dispatch cannot carry. The `async-trait` crate desugars `async fn` to the same `BoxFuture` a manual implementation produces. Using `BoxFuture` explicitly keeps the future's lifetime visible in the signature and avoids one proc-macro expansion per trait.
+
+#### Feature flags instead of inventory or libloading
+
+Two other registration mechanisms were considered.
+
+The `inventory` crate collects static items at link time through platform-specific linker sections. Each format would register itself with a `submit!` macro. `inventory` drops items silently across static-library boundaries, interacts poorly with analysis tools that do not run a real linker, and registers formats that the caller did not explicitly depend on.
+
+Runtime library loading through `libloading` or similar would allow formats to ship as separate shared libraries. Rust has no stable ABI, so implementations would need to match the host's compiler version and feature flags exactly. Every use would need `unsafe`. PyIceberg has flagged runtime loading as an open question without landing a mechanism.
+
+Feature flags work across `cargo check`, `cargo miri`, cross-compilation, and static linking. `datafusion`, `opendal`, and `reqwest` use the same pattern.
+
+#### RecordBatch as the canonical data type
+
+Arrow `RecordBatch` is the interchange format for every Rust data ecosystem iceberg-rust plausibly integrates with. DataFusion consumes it directly. Polars bridges to Arrow with zero-copy conversion for primitive columns. Lance uses its own columnar layout internally and exposes Arrow at read and write boundaries. Every integration and near-term format proposal either produces Arrow or bridges to it.
+
+#### Name: FormatModel
+
+The Java community settled on `FormatModel` after considering `ObjectModel` and `FileAccessFactory` during the 10-month review of PR [#12774](https://github.com/apache/iceberg/pull/12774). This RFC reuses the settled Java name to keep the concept addressable in cross-project discussion.
+
+## Conformance tests (TCK)
+
+A new `FormatModel` implementation should pass a shared conformance test suite before it merges. Java has one (`BaseFormatModelTests<T>`) but it is intentionally minimal: a single struct-of-primitives data generator, no V3 types, no schema evolution, no metadata columns. Issue [#15415](https://github.com/apache/iceberg/issues/15415) tracks the gaps.
Review Comment:
The BaseFormatModelTests were intentionally kept minimal to limit the scope of that large PR. We’re actively adding more tests there now. It would also be useful to align on the test cases and potentially create/run cross‑implementation tests as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3167112314
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
+
+### The read and write builders
+
+`FormatReadBuilder` and `FormatWriteBuilder` configure a single read or write. `FormatModel` produces them, and the caller consumes them with `build`.
+
+```rust
+pub trait FormatReadBuilder: Send {
+ fn project(&mut self, schema: Schema) -> &mut Self;
+ fn filter(&mut self, predicate: BoundPredicate) -> &mut Self;
+ fn split(&mut self, start: u64, length: u64) -> &mut Self;
+ fn case_sensitive(&mut self, case_sensitive: bool) -> &mut Self;
+ fn batch_size(&mut self, batch_size: usize) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<ArrowRecordBatchStream>>;
+}
+
+pub trait FormatWriteBuilder: Send {
+ fn schema(&mut self, schema: Schema) -> &mut Self;
+ fn set(&mut self, key: String, value: String) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<Box<dyn FormatFileWriter>>>;
+}
+```
+
+Both builders take Iceberg `Schema` values. Format implementations convert to physical schemas internally using `schema_to_arrow_schema` from `arrow/schema.rs`. This deliberately departs from Java's `ReadBuilder<D, S>`, which has a separate `engineProjection(S)` method for variant shredding and similar per-engine type mapping. Rust's builders expose one projection surface, and the hook for a variant-shredding "engine projection" is a future-extension point described in "Design rationale."
+
+`FormatWriteBuilder::build` produces a `FormatFileWriter`:
+
+```rust
+pub trait FormatFileWriter: Send {
+ fn write(&mut self, batch: &RecordBatch) -> BoxFuture<'_, Result<()>>;
+ fn close(self: Box<Self>) -> BoxFuture<'static, Result<Vec<DataFileBuilder>>>;
+}
+```
+
+The async methods return `BoxFuture` rather than using `async fn` in traits. The `self: Box<Self>` signature on `build` and `close` lets those methods consume the value while keeping the traits object-safe. Both patterns are forced by the trait-object boundary at the registry. The "BoxFuture instead of async fn in traits" and "Dynamic dispatch at the registry, static inside the format" subsections under "Design rationale" explain why.
+
+### The FormatRegistry
+
+`FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances.
+
+```rust
+pub struct FormatRegistry {
+ models: HashMap<DataFileFormat, Box<dyn FormatModel>>,
+}
+
+impl FormatRegistry {
+ pub fn new() -> Self { ... }
+ pub fn register(&mut self, model: Box<dyn FormatModel>) { ... }
+ pub fn read_builder(
+ &self,
+ format: DataFileFormat,
+ input: InputFile,
+ ) -> Result<Box<dyn FormatReadBuilder>> { ... }
+ pub fn write_builder(
+ &self,
+ format: DataFileFormat,
+ output: OutputFile,
+ ) -> Result<Box<dyn FormatWriteBuilder>> { ... }
+}
+```
+
+Using `DataFileFormat` as a `HashMap` key requires adding `#[derive(Hash)]` to the enum. That is a non-breaking addition.
+
+The registry is an owned value, not a global static. Tests construct their own. Applications construct one at startup and pass it to scan planners and write orchestrators. For the common case of a single registry for the lifetime of a process, `default_format_registry()` returns a `&'static FormatRegistry` initialized through `OnceLock` on first call.
+
+`read_builder` and `write_builder` return `Err(Error { kind: ErrorKind::FeatureUnsupported, .. })` for unregistered formats. The error message distinguishes two cases: the format is implemented but its feature flag is disabled in this build, or the format has no implementation in this crate.
+
+### Feature flags
+
+Format implementations live in `iceberg::formats::{format}` and are gated behind a feature flag per format: `format-parquet` on by default, `format-orc` when an ORC implementation lands, and so on. The default feature set includes every format the crate implements. Users who build from source and want a smaller binary disable what they do not need.
+
+`FormatRegistry::default()` registers every format enabled by the current feature set at compile time:
+
+```rust
+impl Default for FormatRegistry {
+ fn default() -> Self {
+ let mut registry = Self::new();
+ #[cfg(feature = "format-parquet")]
+ registry.register(Box::new(ParquetFormatModel::new()));
+ #[cfg(feature = "format-orc")]
+ registry.register(Box::new(OrcFormatModel::new()));
+ registry
+ }
+}
+```
+
+Format implementations live inside the `iceberg` crate rather than as separate dependencies. This matches how the Java project keeps its `Parquet`, `Avro`, and `ORC` modules inside the `apache/iceberg` repository. In-tree implementations let the community control each format's quality and release cadence and keep the build graph narrow for downstream consumers. Nothing in the trait design prevents a downstream crate from defining its own `FormatModel` and registering it with a custom `FormatRegistry`, but in-tree is the recommended path for formats intended to ship in iceberg-rust itself.
+
+The `parquet` crate remains an unconditional dependency of `iceberg` for now, because non-format code (page index evaluators, row group metric evaluators, delete file loaders) still uses `parquet` types directly. A later pass gates the `parquet` dependency on the `format-parquet` feature once those callers move to format-agnostic abstractions.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # FormatModel, FormatReadBuilder, FormatWriteBuilder, FormatFileWriter
+├── registry.rs # FormatRegistry, default_format_registry
+└── parquet.rs # ParquetFormatModel, wrapping existing ParquetWriter and ArrowReader
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, and so on, with directory-per-format for anything larger than a few hundred lines.
+
+In the initial implementation, `ParquetWriter`, `ParquetWriterBuilder`, `ArrowReader`, and `ArrowReaderBuilder` stay at their current module paths. `ParquetFormatModel` wraps them. Phase 3 of the Migration Plan moves them into `formats/parquet/` and retires the old paths.
+
+### Design rationale
+
+#### No generic over the data type
+
+Java's `FormatModel<D, S>` uses `D` for Spark `InternalRow`, Flink `RowData`, Arrow `ColumnarBatch`, and other engine-native row types. iceberg-rust has one row type: Arrow `RecordBatch`. Every writer accepts it, and every reader returns it. No format on the near-term queue (ORC, Avro data-file, Vortex, Lance) produces anything other than `RecordBatch` at the Iceberg-facing boundary. No engine integration in iceberg-rust today brings an engine-native row type the way Spark and Flink do in Java.
Review Comment:
I think this limitation will prevent integrations in the long run. How hard would it be to change this if we face new use-cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "Kurtiscwright (via GitHub)" <gi...@apache.org>.
Kurtiscwright commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3320861940
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
+
+The registry exposes `reader::<B>(format)` and `writer::<B>(format)`, where `B` is the batch type the engine chose. Errors are eager: if `(format, B)` is not registered, the call fails immediately at scan planning time rather than mid-stream during I/O. This is a runtime contract — the `TypeId` tells the registry what batch type the caller expects, but the type system does not enforce that the reader actually produces that type. A reader that returns the wrong type fails at the first downcast with a clear error. Java makes the same tradeoff with `Class<?>`.
+
+A process-wide default registry is available via `OnceLock`. Custom registries can be provided to `TableScanBuilder` for tests or restricted configurations.
Review Comment:
Are you referring to dependency injection or do you mean raise the registry up to the Catalog for initialization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "Kurtiscwright (via GitHub)" <gi...@apache.org>.
Kurtiscwright commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3320847816
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
Review Comment:
It can be, but it doesn't have to be. It will depend on if the engine/community implementer wants to support multiple in-memory formats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3166818793
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
Review Comment:
How equality delete writers are implemented in rust?
In Java we added an extra abstraction layer on top of FileWriterBuilder so the registry-provided writer can serve both DataFileWriters and EqualityDeleteFileWriters. This works because, aside from configuration differences, both writers ultimately do the same thing: write records.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3167125691
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
+
+### The read and write builders
+
+`FormatReadBuilder` and `FormatWriteBuilder` configure a single read or write. `FormatModel` produces them, and the caller consumes them with `build`.
+
+```rust
+pub trait FormatReadBuilder: Send {
+ fn project(&mut self, schema: Schema) -> &mut Self;
+ fn filter(&mut self, predicate: BoundPredicate) -> &mut Self;
+ fn split(&mut self, start: u64, length: u64) -> &mut Self;
+ fn case_sensitive(&mut self, case_sensitive: bool) -> &mut Self;
+ fn batch_size(&mut self, batch_size: usize) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<ArrowRecordBatchStream>>;
+}
+
+pub trait FormatWriteBuilder: Send {
+ fn schema(&mut self, schema: Schema) -> &mut Self;
+ fn set(&mut self, key: String, value: String) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<Box<dyn FormatFileWriter>>>;
+}
+```
+
+Both builders take Iceberg `Schema` values. Format implementations convert to physical schemas internally using `schema_to_arrow_schema` from `arrow/schema.rs`. This deliberately departs from Java's `ReadBuilder<D, S>`, which has a separate `engineProjection(S)` method for variant shredding and similar per-engine type mapping. Rust's builders expose one projection surface, and the hook for a variant-shredding "engine projection" is a future-extension point described in "Design rationale."
+
+`FormatWriteBuilder::build` produces a `FormatFileWriter`:
+
+```rust
+pub trait FormatFileWriter: Send {
+ fn write(&mut self, batch: &RecordBatch) -> BoxFuture<'_, Result<()>>;
+ fn close(self: Box<Self>) -> BoxFuture<'static, Result<Vec<DataFileBuilder>>>;
+}
+```
+
+The async methods return `BoxFuture` rather than using `async fn` in traits. The `self: Box<Self>` signature on `build` and `close` lets those methods consume the value while keeping the traits object-safe. Both patterns are forced by the trait-object boundary at the registry. The "BoxFuture instead of async fn in traits" and "Dynamic dispatch at the registry, static inside the format" subsections under "Design rationale" explain why.
+
+### The FormatRegistry
+
+`FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances.
+
+```rust
+pub struct FormatRegistry {
+ models: HashMap<DataFileFormat, Box<dyn FormatModel>>,
+}
+
+impl FormatRegistry {
+ pub fn new() -> Self { ... }
+ pub fn register(&mut self, model: Box<dyn FormatModel>) { ... }
+ pub fn read_builder(
+ &self,
+ format: DataFileFormat,
+ input: InputFile,
+ ) -> Result<Box<dyn FormatReadBuilder>> { ... }
+ pub fn write_builder(
+ &self,
+ format: DataFileFormat,
+ output: OutputFile,
+ ) -> Result<Box<dyn FormatWriteBuilder>> { ... }
+}
+```
+
+Using `DataFileFormat` as a `HashMap` key requires adding `#[derive(Hash)]` to the enum. That is a non-breaking addition.
+
+The registry is an owned value, not a global static. Tests construct their own. Applications construct one at startup and pass it to scan planners and write orchestrators. For the common case of a single registry for the lifetime of a process, `default_format_registry()` returns a `&'static FormatRegistry` initialized through `OnceLock` on first call.
+
+`read_builder` and `write_builder` return `Err(Error { kind: ErrorKind::FeatureUnsupported, .. })` for unregistered formats. The error message distinguishes two cases: the format is implemented but its feature flag is disabled in this build, or the format has no implementation in this crate.
+
+### Feature flags
+
+Format implementations live in `iceberg::formats::{format}` and are gated behind a feature flag per format: `format-parquet` on by default, `format-orc` when an ORC implementation lands, and so on. The default feature set includes every format the crate implements. Users who build from source and want a smaller binary disable what they do not need.
+
+`FormatRegistry::default()` registers every format enabled by the current feature set at compile time:
+
+```rust
+impl Default for FormatRegistry {
+ fn default() -> Self {
+ let mut registry = Self::new();
+ #[cfg(feature = "format-parquet")]
+ registry.register(Box::new(ParquetFormatModel::new()));
+ #[cfg(feature = "format-orc")]
+ registry.register(Box::new(OrcFormatModel::new()));
+ registry
+ }
+}
+```
+
+Format implementations live inside the `iceberg` crate rather than as separate dependencies. This matches how the Java project keeps its `Parquet`, `Avro`, and `ORC` modules inside the `apache/iceberg` repository. In-tree implementations let the community control each format's quality and release cadence and keep the build graph narrow for downstream consumers. Nothing in the trait design prevents a downstream crate from defining its own `FormatModel` and registering it with a custom `FormatRegistry`, but in-tree is the recommended path for formats intended to ship in iceberg-rust itself.
+
+The `parquet` crate remains an unconditional dependency of `iceberg` for now, because non-format code (page index evaluators, row group metric evaluators, delete file loaders) still uses `parquet` types directly. A later pass gates the `parquet` dependency on the `format-parquet` feature once those callers move to format-agnostic abstractions.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # FormatModel, FormatReadBuilder, FormatWriteBuilder, FormatFileWriter
+├── registry.rs # FormatRegistry, default_format_registry
+└── parquet.rs # ParquetFormatModel, wrapping existing ParquetWriter and ArrowReader
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, and so on, with directory-per-format for anything larger than a few hundred lines.
+
+In the initial implementation, `ParquetWriter`, `ParquetWriterBuilder`, `ArrowReader`, and `ArrowReaderBuilder` stay at their current module paths. `ParquetFormatModel` wraps them. Phase 3 of the Migration Plan moves them into `formats/parquet/` and retires the old paths.
+
+### Design rationale
+
+#### No generic over the data type
+
+Java's `FormatModel<D, S>` uses `D` for Spark `InternalRow`, Flink `RowData`, Arrow `ColumnarBatch`, and other engine-native row types. iceberg-rust has one row type: Arrow `RecordBatch`. Every writer accepts it, and every reader returns it. No format on the near-term queue (ORC, Avro data-file, Vortex, Lance) produces anything other than `RecordBatch` at the Iceberg-facing boundary. No engine integration in iceberg-rust today brings an engine-native row type the way Spark and Flink do in Java.
+
+Adding a `D` parameter today means writing `<RecordBatch>` everywhere the trait is used, for no present caller benefit. If a future format cannot bridge to Arrow, the trait can gain an associated type with a default, which is a semver-compatible addition.
+
+#### No generic over the engine schema
+
+Java's `S` parameter serves Spark's `StructType`, Flink's `RowType`, and the other engine-native schema types. iceberg-rust has `iceberg::spec::Schema` for the logical schema and `arrow::datatypes::SchemaRef` for the physical schema, with conversion in `arrow/schema.rs`. There is no third schema type a generic parameter would serve.
+
+Variant shredding is the concrete use case that Java's `engineProjection(S)` method addresses. `FormatReadBuilder` can later gain an `engine_projection(&mut self, schema: ArrowSchemaRef) -> &mut Self` method with a default no-op implementation. Format implementations that support shredding override it. The parameter is `ArrowSchemaRef`, not a generic `S`, because Arrow is the only engine schema in iceberg-rust.
Review Comment:
engineProjection(S) could also be used for requesting a `long` when the iceberg schema defines an `int`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3167035836
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
+
+### The read and write builders
+
+`FormatReadBuilder` and `FormatWriteBuilder` configure a single read or write. `FormatModel` produces them, and the caller consumes them with `build`.
+
+```rust
+pub trait FormatReadBuilder: Send {
Review Comment:
The engineSchema might also be necessary for Variant handling, if the engine wants to control how a Variant is read (e.g., specify the shredding layout). This would avoid leaking the file’s internal structure into the reader and would also eliminate the need to reassemble shredded variants if the file stores them shredded and the engine prefers to consume them in that form.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3167005156
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
+
+### The read and write builders
+
+`FormatReadBuilder` and `FormatWriteBuilder` configure a single read or write. `FormatModel` produces them, and the caller consumes them with `build`.
+
+```rust
+pub trait FormatReadBuilder: Send {
Review Comment:
There are a few methods that seem to be missing compared to the Java API:
- set / setAll – Do we want to allow pushing configuration down to the readers (e.g., prefetching, parallel column reads, or page index usage)? If so, having a set-style method to propagate such configs to the reader could be useful.
- reuseContainers – Does Rust support container reuse? In Java, reusing the same objects across reads (instead of allocating new ones each time) can bring significant performance benefits.
- idToConstants – In Java, the reader returns full rows, but some values are not stored in the data file itself and instead come from file metadata. IdToConstants supplies those values so they can be populated in the returned rows.
- nameMapping – In Java, this is used when reading data files that were not written by Iceberg, where column IDs are missing and only column names are available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "dannycjones (via GitHub)" <gi...@apache.org>.
dannycjones commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3468838984
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
+
+The registry exposes `reader::<B>(format)` and `writer::<B>(format)`, where `B` is the batch type the engine chose. Errors are eager: if `(format, B)` is not registered, the call fails immediately at scan planning time rather than mid-stream during I/O. This is a runtime contract — the `TypeId` tells the registry what batch type the caller expects, but the type system does not enforce that the reader actually produces that type. A reader that returns the wrong type fails at the first downcast with a clear error. Java makes the same tradeoff with `Class<?>`.
+
+A process-wide default registry is available via `OnceLock`. Custom registries can be provided to `TableScanBuilder` for tests or restricted configurations.
+
+**Default scan loop (hybrid execution model).** The kernel provides a default scan loop that pulls batches from the format reader, checks `ReadResult` to see what the format already handled, and applies any remaining operations (residual filtering, schema evolution, constant injection, delete application). Engines that want full control (DataFusion, Comet) bypass this loop and drive execution themselves — they use the format reader as a stream source and apply their own processing. The `DataBatch` methods are available as a toolbox, not mandated as a pipeline.
+
+**Supporting types:** `ColumnMetrics` stores bounds as typed `Datum` values, matching how `DataFile` already stores bounds. `MetricsConfig` and `MetricsMode` control collection granularity. `FileContent` distinguishes data from equality and position deletes. All config structs and enums are `#[non_exhaustive]`.
+
+### Feature flags
+
+Format implementations are gated behind feature flags (`format-parquet` on by default, `format-orc` when ORC lands). The default feature set includes every implemented format. No configuration is needed for downstream crates using default features. Feature flags exist for crates that want to minimize binary size or compile time. This matches `datafusion`, `opendal`, and `reqwest`.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # Module declarations and public re-exports
+├── traits.rs # DataBatch, FormatReader, FormatWriter,
+│ # FormatFileWriter, ReadOptions, WriteOptions,
+│ # ColumnMetrics, MetricsConfig, MetricsMode,
+│ # FileContent, WriterResult,
+│ # VariantShredding (future extension, not yet wired)
+├── registry.rs # FormatRegistry, default_format_registry()
+└── parquet/
+ ├── mod.rs # Submodule declarations and re-exports
+ ├── model.rs # ParquetArrowModel (implements FormatReader + FormatWriter)
+ ├── reader.rs # Parquet reader internals
+ └── writer.rs # Parquet writer (implements FormatFileWriter)
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, each following the same structure.
+
+### Target crate architecture
+
+The end-state after crate separation (follow-up work, not part of Phase 1). All crates live in the same repository and workspace. Phase 1 ships everything in the existing `iceberg` crate.
+
+```
+iceberg (core kernel) - traits only, no arrow/parquet dependency
+iceberg-arrow - impl DataBatch for RecordBatch
+iceberg-format-parquet - ParquetArrowModel
+iceberg-format-orc - OrcArrowModel
+iceberg-datafusion - TableProvider, depends on above
+```
+
+This separation enables engines to pin their own Arrow version independently. For example, a Comet integration could depend on `iceberg` (the kernel) and provide its own `DataBatch` implementation and format readers using a newer Arrow version, without conflicting with `iceberg-arrow` or `iceberg-format-parquet`.
+
+## Design Rationale
Review Comment:
I think it would be good to speak to how this design facilitates (or at least doesn't block) IO optimizations.
In particular, AWS integrated its AAL library with Apache Hadoop and Apache Iceberg which delivers read optimizations, primarily for Parquet. https://github.com/awslabs/analytics-accelerator-s3
For example:
- Ranged S3 GETs (presumably File IO impl detail)
- Sequential prefetching
- Parquet-aware row group prefetching
- Use larger reads for footer, to avoid Parquet footer dance fetching last 8 bytes, then seeking back to start of the footer.
FWIW, I am not aware if the iceberg-rust library has any of this already. I'd like to ensure that the new file format design provides a good way for us to implement similar optimizations. I suspect that many of these may be best suited to land in the file format layer, specifically the file format implementations themselves rather than our new format abstractions. For instance, parquet crate already has the footer optimization configurable: https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaDataReader.html#method.with_prefetch_hint
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3167011533
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
Review Comment:
This compelling to me! Thanks for the detailed explanation!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3167084001
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
+
+6. **Change the Iceberg table spec.** This proposal is a Rust-only API change. It does not modify the Iceberg spec, the manifest format, the manifest list format, or the on-disk layout of any file.
+
+7. **Modify manifest read or write paths.** Manifests and manifest lists remain in Avro and are handled by the existing `ManifestReader` and `ManifestWriter` paths. The File Format API is about data files and delete files only.
+
+
+## Design
+
+The Rust API is three traits and a registry. `FormatModel` is the trait that each format implementation provides. `FormatReadBuilder` and `FormatWriteBuilder` are the per-operation configurators that `FormatModel` returns. `FormatRegistry` maps `DataFileFormat` values to `FormatModel` instances. None of the traits carry generic parameters. The subsections below introduce each type, and a final "Design rationale" subsection explains the choices.
+
+### The FormatModel trait
+
+```rust
+pub trait FormatModel: Send + Sync + 'static {
+ fn format(&self) -> DataFileFormat;
+ fn read_builder(&self, input: InputFile) -> Box<dyn FormatReadBuilder>;
+ fn write_builder(&self, output: OutputFile) -> Box<dyn FormatWriteBuilder>;
+}
+```
+
+Each implementation registers one instance per `DataFileFormat` variant it supports. The `format` method returns that variant. `read_builder` and `write_builder` are the entry points for reading and writing a file. Both return trait objects so that the registry can hand them back from a `DataFileFormat`-keyed lookup.
+
+The data type is fixed to Arrow `RecordBatch`, the Iceberg schema type to `iceberg::spec::Schema`, and the physical schema type to `arrow::datatypes::SchemaRef`. These are the only types in iceberg-rust today that would fill the roles Java uses generic parameters for. Arguments for keeping them fixed, including comparisons with Java's `FormatModel<D, S>`, are in "Design rationale" below.
+
+### The read and write builders
+
+`FormatReadBuilder` and `FormatWriteBuilder` configure a single read or write. `FormatModel` produces them, and the caller consumes them with `build`.
+
+```rust
+pub trait FormatReadBuilder: Send {
+ fn project(&mut self, schema: Schema) -> &mut Self;
+ fn filter(&mut self, predicate: BoundPredicate) -> &mut Self;
+ fn split(&mut self, start: u64, length: u64) -> &mut Self;
+ fn case_sensitive(&mut self, case_sensitive: bool) -> &mut Self;
+ fn batch_size(&mut self, batch_size: usize) -> &mut Self;
+ fn build(self: Box<Self>) -> BoxFuture<'static, Result<ArrowRecordBatchStream>>;
+}
+
+pub trait FormatWriteBuilder: Send {
Review Comment:
There are a few methods missing compared to the Java API:
- engineSchema – The engine could specify how a Variant is currently shredded, or indicate that values are stored, for example, as int in the records. The writer could then use this information to read or materialize them differently (e.g., as long) when writing.
- meta – File-level metadata is often type-specific. In the Java implementation we record some information mainly for debuggability, such as the file type or the equality field IDs for an equality-delete file.
- metricsConfig – In Java, the writer is responsible for collecting file metrics, so the configuration must be passed down to ensure the correct metrics are produced.
- overwrite – In Java, users can request that existing output files be overwritten.
- Encryption settings – For example fileEncryptionKey and aadPrefix, which are required when encryption is enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "Kurtiscwright (via GitHub)" <gi...@apache.org>.
Kurtiscwright commented on PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#issuecomment-4503359331
@pvary @CTTY @xanderbailey I have rewritten the proposal based on the feedback. Thank you for your patience as I had a few things come up offline that delayed getting this rewrite out. I appreciate you in advance for any amount of review you can help provide. I will be sending out code stub example PRs here shortly to demonstrate what this would look like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3163837397
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
Review Comment:
ORC does have a similar concept to PME in parquet and the [spec references this ](https://iceberg.apache.org/gcm-stream-spec/) but it's not implemented in Java. In the case of Parquet, the actual encryption / decryption is handled by arrow-rs in the writer so it will differ depending on format. I would hope we could get away with just passing the key and aad down to the writer but I'm the kind of person that has to see code to work out how this might look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3160092754
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
Review Comment:
Encryption work is currently in-flight so I'd love to understand how we could incorporate that in this plan.
```
let mut wb = registry.write_builder(format, output)?;
if let Some(em) = &self.encryption_manager {
let (dek, aad) = em.create_file_key().await?;
wb.with_file_encryption_key(&dek);
wb.with_aad_prefix(&aad);
}
let writer = wb.build().await?;
``
I need to think if something like this might make sense?
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,520 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.0, Rust 1.92 as of this writing) is the core library of the Apache Iceberg Rust project. Its module layout in `crates/iceberg/src/lib.rs` exposes public modules for `spec`, `arrow`, `writer`, `scan`, `io`, `expr`, `transaction`, and others. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides a three-layer architecture described in `crates/iceberg/src/writer/mod.rs`:
+
+1. **`FileWriter` / `FileWriterBuilder`** in `writer/file_writer/mod.rs`: traits for physical file writers, generic over an output type (defaulting to `Vec<DataFileBuilder>`). `FileWriterBuilder::build` takes an `OutputFile` and returns a `FileWriter`. `FileWriter::write` takes a `&RecordBatch`.
+
+2. **`IcebergWriter` / `IcebergWriterBuilder`** in `writer/mod.rs`: traits for logical Iceberg writers (data files, equality deletes, position deletes, partitioning), defaulting to `RecordBatch` input and `Vec<DataFile>` output.
+
+3. **Concrete implementations**: `ParquetWriterBuilder` and `ParquetWriter` in `writer/file_writer/parquet_writer.rs` are the only `FileWriter` implementation. Higher-level writers such as `DataFileWriterBuilder` (`writer/base_writer/data_file_writer.rs`) and `EqualityDeleteFileWriterBuilder` (`writer/base_writer/equality_delete_writer.rs`) are generic over any `FileWriterBuilder`, but every example, test, and integration instantiates them with `ParquetWriterBuilder`.
+
+The trait layer is format-agnostic. Every concrete instantiation uses Parquet.
+
+For data file reading, the crate provides `ArrowReaderBuilder` and `ArrowReader` in `arrow/reader.rs`. Both are Parquet-specific despite the generic name. `ArrowReader::process_file_scan_task` calls `open_parquet_file` directly, constructs a `ParquetRecordBatchReaderBuilder`, and uses Parquet-specific row group filtering and page index logic. `TableScan::to_arrow` in `scan/mod.rs` wires `ArrowReaderBuilder` in as the only reader path. `FileScanTask` carries a `data_file_format: DataFileFormat` field, but `process_file_scan_task` ignores it. `DataFileFormat` appears throughout `src/`. Almost every non-test reference is `DataFileFormat::Parquet`. The only non-Parquet references are two uses of `DataFileFormat::Avro` in `transaction/snapshot.rs` for manifest files.
+
+The `DataFileFormat` enum in `spec/manifest/data_file.rs` has four variants: `Avro`, `Orc`, `Parquet`, and `Puffin`. Puffin files hold statistics and deletion vectors rather than row data. They are handled in `crates/iceberg/src/puffin/` and are out of scope for this RFC (see Non-Goals). Data file support today:
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read from Rust today, even though both are valid per the Iceberg spec.
+
+Arrow `RecordBatch` is the only in-memory data representation in the crate. It is the input type for every writer trait and the output type for every reader. The `arrow/` module provides schema conversion (`schema_to_arrow_schema`, `arrow_schema_to_schema`), `RecordBatchTransformer` for schema evolution and constant column injection, and the Parquet-to-Arrow read path. Every integration crate, including `iceberg-datafusion`, consumes Arrow. The crate does not define a generic `Record` type and does not integrate with engine-specific row types such as Java's `InternalRow` or `RowData`.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` to branch on `DataFileFormat` and threading ORC-specific logic through every layer that touches it. The write path has the same shape.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options (`with_metadata_size_hint`, `with_row_group_filtering_enabled`, `with_row_selection_enabled`) that are meaningless for ORC or Avro. The name "ArrowReader" conflates the in-memory representation (Arrow) with the on-disk format (Parquet).
+
+3. **No format-agnostic statistics.** `ParquetWriter` computes column sizes, value counts, and min/max bounds through `MinMaxColAggregator` and `NanValueCountVisitor`, both tightly coupled to Parquet's `Statistics` type. Another format cannot produce comparable manifest metadata without a shared statistics interface.
+
+4. **V3 types will need per-format serialization.** The V3 spec adds Variant and Geometry types. Each format encodes them differently: Parquet uses variant shredding, ORC uses binary, Avro uses union types. Implementing either type without a format abstraction means new `match` arms in every reader and writer code path.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)) after a 10-month review. Implementations for Parquet, Avro, ORC, and Arrow followed in PRs [#15253](https://github.com/apache/iceberg/pull/15253) through [#15258](https://github.com/apache/iceberg/pull/15258). Engine migrations for Spark ([#15328](https://github.com/apache/iceberg/pull/15328)) and Flink ([#15329](https://github.com/apache/iceberg/pull/15329)) landed shortly after.
+
+Java's `FormatModel` carries two generic parameters: a data type `D` (Java uses `Record`, Spark `InternalRow`, Flink `RowData`, and Arrow `ColumnarBatch`) and an engine schema type `S` (Spark `StructType`, Flink `RowType`, and others). A static `FormatModelRegistry` stores implementations keyed by `Pair<FileFormat, Class<?>>` and populates itself through Java reflection at startup. The old `Parquet.WriteBuilder`, `Avro.WriteBuilder`, and `ORC.WriteBuilder` are not deprecated. The new `FormatModel` implementations wrap them rather than replace them.
+
+PyIceberg has an open proposal for the equivalent capability in issue [apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100), with an in-progress PR at [apache/iceberg-python#3119](https://github.com/apache/iceberg-python/pull/3119). Because PyIceberg uses PyArrow as its only in-memory representation, the proposal drops Java's generic type parameters and keys the registry on file format alone. Two prior PyIceberg ORC PRs ([#790](https://github.com/apache/iceberg-python/pull/790), [#2236](https://github.com/apache/iceberg-python/pull/2236)) were closed as stale without merging, which reinforces the case for landing an abstraction layer before adding new formats.
+
+Design decisions in this RFC that differ from Java (no generics, single-dimension registry key, hard cutover from the old Parquet types) are justified inline where they appear. Specific Java design points that shaped those decisions are called out in Alternatives Considered.
+
+## Goals
+
+The user-facing outcome of this proposal is that every Iceberg data file and delete file flows through the same stable and extensible API. Parquet is the first format to land. ORC, Avro, and others follow on the same interface. Every goal below serves that outcome.
+
+1. Define a `FormatModel` trait that encapsulates format-specific read and write behavior independent of the on-disk format.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration. After this work, `TableScan::to_arrow` and `DataFileWriterBuilder` dispatch through the format abstraction instead of constructing Parquet types directly.
+
+3. Provide a registry that maps `DataFileFormat` values to `FormatModel` implementations, so callers obtain readers and writers without naming the concrete format type.
+
+4. Define a conformance test suite (TCK) that any `FormatModel` implementation must pass before it merges.
+
+5. Match the Java and PyIceberg designs where they align, and diverge where Rust's single-data-type ecosystem and pre-1.0 status justify it. Divergences are called out inline.
+
+## Non-Goals
+
+The items below are deliberately out of scope to keep this proposal focused on the abstraction and its Parquet implementation. Most are follow-up work that the API enables but does not itself deliver.
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet implementation. ORC, Avro data-file, Vortex, and Lance are follow-up RFCs.
+
+2. **Introduce a plugin protocol or runtime library loading.** Rust does not offer a clean mechanism for loading compiled plugins at runtime. A runtime-linking approach using `libloading` or similar would expand scope beyond what the formats currently under discussion (Parquet, ORC, Avro, Vortex, Lance) require.
+
+3. **Add Puffin support to the FormatModel.** Puffin files hold statistics and deletion vectors rather than row data. They have a different lifecycle from data files and are already handled separately in `crates/iceberg/src/puffin/`.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds a format abstraction beneath `FileWriter`, not a replacement for it.
+
+5. **Implement variant shredding or encryption.** Java exposes `engineProjection` and `engineSchema` as extension points for variant shredding and similar format-specific type mapping, and `withFileEncryptionKey` and `withAADPrefix` for Parquet encryption. Equivalent hooks are noted as future extensions in the Rust design. Implementing either requires a dedicated RFC.
Review Comment:
Encryption work is currently in-flight so I'd love to understand how we could incorporate that in this plan.
```
let mut wb = registry.write_builder(format, output)?;
if let Some(em) = &self.encryption_manager {
let (dek, aad) = em.create_file_key().await?;
wb.with_file_encryption_key(&dek);
wb.with_aad_prefix(&aad);
}
let writer = wb.build().await?;
```
I need to think if something like this might make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#issuecomment-5081545183
Don't close this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#issuecomment-5075852500
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3314102701
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
Review Comment:
this is kinda confusing. I thought in-memory format should be associated with the format model that we registered?
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
+
+The registry exposes `reader::<B>(format)` and `writer::<B>(format)`, where `B` is the batch type the engine chose. Errors are eager: if `(format, B)` is not registered, the call fails immediately at scan planning time rather than mid-stream during I/O. This is a runtime contract — the `TypeId` tells the registry what batch type the caller expects, but the type system does not enforce that the reader actually produces that type. A reader that returns the wrong type fails at the first downcast with a clear error. Java makes the same tradeoff with `Class<?>`.
+
+A process-wide default registry is available via `OnceLock`. Custom registries can be provided to `TableScanBuilder` for tests or restricted configurations.
Review Comment:
Can we inject registry into the Catalog?
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
Review Comment:
What's the difference between FormatFileWriter and FormatWriter
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
Review Comment:
nit: Maybe we should just call it in-memory layer?
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
+
+The registry exposes `reader::<B>(format)` and `writer::<B>(format)`, where `B` is the batch type the engine chose. Errors are eager: if `(format, B)` is not registered, the call fails immediately at scan planning time rather than mid-stream during I/O. This is a runtime contract — the `TypeId` tells the registry what batch type the caller expects, but the type system does not enforce that the reader actually produces that type. A reader that returns the wrong type fails at the first downcast with a clear error. Java makes the same tradeoff with `Class<?>`.
+
+A process-wide default registry is available via `OnceLock`. Custom registries can be provided to `TableScanBuilder` for tests or restricted configurations.
+
+**Default scan loop (hybrid execution model).** The kernel provides a default scan loop that pulls batches from the format reader, checks `ReadResult` to see what the format already handled, and applies any remaining operations (residual filtering, schema evolution, constant injection, delete application). Engines that want full control (DataFusion, Comet) bypass this loop and drive execution themselves — they use the format reader as a stream source and apply their own processing. The `DataBatch` methods are available as a toolbox, not mandated as a pipeline.
+
+**Supporting types:** `ColumnMetrics` stores bounds as typed `Datum` values, matching how `DataFile` already stores bounds. `MetricsConfig` and `MetricsMode` control collection granularity. `FileContent` distinguishes data from equality and position deletes. All config structs and enums are `#[non_exhaustive]`.
+
+### Feature flags
+
+Format implementations are gated behind feature flags (`format-parquet` on by default, `format-orc` when ORC lands). The default feature set includes every implemented format. No configuration is needed for downstream crates using default features. Feature flags exist for crates that want to minimize binary size or compile time. This matches `datafusion`, `opendal`, and `reqwest`.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # Module declarations and public re-exports
+├── traits.rs # DataBatch, FormatReader, FormatWriter,
+│ # FormatFileWriter, ReadOptions, WriteOptions,
+│ # ColumnMetrics, MetricsConfig, MetricsMode,
+│ # FileContent, WriterResult,
+│ # VariantShredding (future extension, not yet wired)
+├── registry.rs # FormatRegistry, default_format_registry()
+└── parquet/
+ ├── mod.rs # Submodule declarations and re-exports
+ ├── model.rs # ParquetArrowModel (implements FormatReader + FormatWriter)
+ ├── reader.rs # Parquet reader internals
+ └── writer.rs # Parquet writer (implements FormatFileWriter)
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, each following the same structure.
+
+### Target crate architecture
+
+The end-state after crate separation (follow-up work, not part of Phase 1). All crates live in the same repository and workspace. Phase 1 ships everything in the existing `iceberg` crate.
+
+```
+iceberg (core kernel) - traits only, no arrow/parquet dependency
+iceberg-arrow - impl DataBatch for RecordBatch
+iceberg-format-parquet - ParquetArrowModel
+iceberg-format-orc - OrcArrowModel
+iceberg-datafusion - TableProvider, depends on above
+```
+
+This separation enables engines to pin their own Arrow version independently. For example, a Comet integration could depend on `iceberg` (the kernel) and provide its own `DataBatch` implementation and format readers using a newer Arrow version, without conflicting with `iceberg-arrow` or `iceberg-format-parquet`.
+
+## Design Rationale
+
+### Three layers, Arrow as default
+
+Java's File Format API has two layers: format and engine. Each format-engine combination requires a direct conversion path, producing an N-by-M matrix. The Java community's performance analysis ([benchmark branch](https://github.com/pvary/iceberg/tree/perf_bench)) measured 20% overhead for double conversion (ORC to Spark ColumnarBatch versus direct VectorizedRowBatch access) and recommended a hybrid: provide an Arrow-based intermediate for interoperability, allow engines to replace it for performance-sensitive paths.
+
+This RFC adopts the hybrid. Arrow is the shipped default. The `DataBatch` trait is the extension point for representations that cannot or should not convert to Arrow. If both format and engine happen to use Arrow (the common case today), there is zero intermediate conversion.
+
+### Why DataBatch is a trait
+
+The kernel ambition ([#1817](https://github.com/apache/iceberg-rust/issues/1817)) requires support for multiple in-memory representations. Arrow version coupling forces lockstep upgrades across the dependency graph — an engine that wants Arrow 60 cannot coexist with a kernel pinned to Arrow 58 unless the kernel is representation-agnostic. `DataBatch` as a trait solves both while preserving Arrow as the shipped default.
+
+The original version of this RFC fixed the data type to `RecordBatch`. Community feedback raised two concerns: that Arrow's dominance in the Rust ecosystem may reflect selection pressure rather than a permanent constraint, and that a kernel-level API should remain open to other ecosystems. This rewrite makes the in-memory type a trait boundary rather than a fixed type.
Review Comment:
Why is DataBatch not a generic? is it because of dyn compatibility?
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
+
+The registry exposes `reader::<B>(format)` and `writer::<B>(format)`, where `B` is the batch type the engine chose. Errors are eager: if `(format, B)` is not registered, the call fails immediately at scan planning time rather than mid-stream during I/O. This is a runtime contract — the `TypeId` tells the registry what batch type the caller expects, but the type system does not enforce that the reader actually produces that type. A reader that returns the wrong type fails at the first downcast with a clear error. Java makes the same tradeoff with `Class<?>`.
+
+A process-wide default registry is available via `OnceLock`. Custom registries can be provided to `TableScanBuilder` for tests or restricted configurations.
+
+**Default scan loop (hybrid execution model).** The kernel provides a default scan loop that pulls batches from the format reader, checks `ReadResult` to see what the format already handled, and applies any remaining operations (residual filtering, schema evolution, constant injection, delete application). Engines that want full control (DataFusion, Comet) bypass this loop and drive execution themselves — they use the format reader as a stream source and apply their own processing. The `DataBatch` methods are available as a toolbox, not mandated as a pipeline.
+
+**Supporting types:** `ColumnMetrics` stores bounds as typed `Datum` values, matching how `DataFile` already stores bounds. `MetricsConfig` and `MetricsMode` control collection granularity. `FileContent` distinguishes data from equality and position deletes. All config structs and enums are `#[non_exhaustive]`.
+
+### Feature flags
+
+Format implementations are gated behind feature flags (`format-parquet` on by default, `format-orc` when ORC lands). The default feature set includes every implemented format. No configuration is needed for downstream crates using default features. Feature flags exist for crates that want to minimize binary size or compile time. This matches `datafusion`, `opendal`, and `reqwest`.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # Module declarations and public re-exports
+├── traits.rs # DataBatch, FormatReader, FormatWriter,
Review Comment:
nit: we should not put all traits under one file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "Kurtiscwright (via GitHub)" <gi...@apache.org>.
Kurtiscwright commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3320887544
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
Review Comment:
Yes I can fix that in a follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "dannycjones (via GitHub)" <gi...@apache.org>.
dannycjones commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3468875829
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
+
+The registry exposes `reader::<B>(format)` and `writer::<B>(format)`, where `B` is the batch type the engine chose. Errors are eager: if `(format, B)` is not registered, the call fails immediately at scan planning time rather than mid-stream during I/O. This is a runtime contract — the `TypeId` tells the registry what batch type the caller expects, but the type system does not enforce that the reader actually produces that type. A reader that returns the wrong type fails at the first downcast with a clear error. Java makes the same tradeoff with `Class<?>`.
+
+A process-wide default registry is available via `OnceLock`. Custom registries can be provided to `TableScanBuilder` for tests or restricted configurations.
+
+**Default scan loop (hybrid execution model).** The kernel provides a default scan loop that pulls batches from the format reader, checks `ReadResult` to see what the format already handled, and applies any remaining operations (residual filtering, schema evolution, constant injection, delete application). Engines that want full control (DataFusion, Comet) bypass this loop and drive execution themselves — they use the format reader as a stream source and apply their own processing. The `DataBatch` methods are available as a toolbox, not mandated as a pipeline.
+
+**Supporting types:** `ColumnMetrics` stores bounds as typed `Datum` values, matching how `DataFile` already stores bounds. `MetricsConfig` and `MetricsMode` control collection granularity. `FileContent` distinguishes data from equality and position deletes. All config structs and enums are `#[non_exhaustive]`.
+
+### Feature flags
+
+Format implementations are gated behind feature flags (`format-parquet` on by default, `format-orc` when ORC lands). The default feature set includes every implemented format. No configuration is needed for downstream crates using default features. Feature flags exist for crates that want to minimize binary size or compile time. This matches `datafusion`, `opendal`, and `reqwest`.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # Module declarations and public re-exports
+├── traits.rs # DataBatch, FormatReader, FormatWriter,
+│ # FormatFileWriter, ReadOptions, WriteOptions,
+│ # ColumnMetrics, MetricsConfig, MetricsMode,
+│ # FileContent, WriterResult,
+│ # VariantShredding (future extension, not yet wired)
+├── registry.rs # FormatRegistry, default_format_registry()
+└── parquet/
+ ├── mod.rs # Submodule declarations and re-exports
+ ├── model.rs # ParquetArrowModel (implements FormatReader + FormatWriter)
+ ├── reader.rs # Parquet reader internals
+ └── writer.rs # Parquet writer (implements FormatFileWriter)
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, each following the same structure.
+
+### Target crate architecture
+
+The end-state after crate separation (follow-up work, not part of Phase 1). All crates live in the same repository and workspace. Phase 1 ships everything in the existing `iceberg` crate.
+
+```
+iceberg (core kernel) - traits only, no arrow/parquet dependency
+iceberg-arrow - impl DataBatch for RecordBatch
+iceberg-format-parquet - ParquetArrowModel
+iceberg-format-orc - OrcArrowModel
+iceberg-datafusion - TableProvider, depends on above
+```
+
+This separation enables engines to pin their own Arrow version independently. For example, a Comet integration could depend on `iceberg` (the kernel) and provide its own `DataBatch` implementation and format readers using a newer Arrow version, without conflicting with `iceberg-arrow` or `iceberg-format-parquet`.
+
+## Design Rationale
Review Comment:
For reference, there was a write-up on the AAL integration and its optimizations here: https://docs.google.com/document/d/13shy0RWotwfWC_qQksb95PXdi-vSUCKQyDzjoExQEN0/edit?tab=t.0#heading=h.xt5bgwddtip0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] rfc: Implement an API for all Data File Formats [iceberg-rust]
Posted by "dannycjones (via GitHub)" <gi...@apache.org>.
dannycjones commented on code in PR #2384:
URL: https://github.com/apache/iceberg-rust/pull/2384#discussion_r3468875829
##########
docs/rfcs/0003_file_format_api.md:
##########
@@ -0,0 +1,510 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# RFC: File Format API for Apache Iceberg Rust
+
+**Authors:** Kurtis C. Wright
+**Last updated:** 2026-05-20
+
+## Background
+
+### Current state
+
+The `iceberg` crate (version 0.9.1, Rust 1.92) is the core library of the Apache Iceberg Rust project. The crate depends directly on the `parquet` crate (with the `async` feature) and on the `arrow-*` crates. It has no feature flags today.
+
+For data file writing, the crate provides `FileWriter` / `FileWriterBuilder` traits that are format-agnostic at the type level, but `ParquetWriterBuilder` and `ParquetWriter` are the only implementation. Higher-level writers (`DataFileWriterBuilder`, `EqualityDeleteFileWriterBuilder`) are generic over any `FileWriterBuilder`, but every instantiation uses Parquet.
+
+For data file reading, `ArrowReaderBuilder` and `ArrowReader` are Parquet-specific despite the generic name. `TableScan::to_arrow` wires `ArrowReaderBuilder` as the only reader path. `FileScanTask` carries a `data_file_format` field, but the reader ignores it.
+
+| Format | Data file read | Data file write | Manifests |
+|---------|----------------|-----------------|-----------|
+| Parquet | Yes | Yes | No |
+| Avro | No | No | Yes |
+| ORC | No | No | No |
+
+A table containing ORC or Avro data files cannot be read by the `iceberg` crate today, even though both are valid per the Iceberg spec.
+
+### Pain points
+
+1. **No extension point for new formats.** Adding ORC means editing `ArrowReader` and threading format-specific logic through every layer.
+
+2. **Parquet assumptions leak into generic code.** `ArrowReaderBuilder` exposes Parquet-specific options meaningless for other formats. The name conflates the in-memory representation with the on-disk format.
+
+3. **No format-agnostic statistics.** Statistics computation is tightly coupled to Parquet's `Statistics` type.
+
+4. **V3 types will need per-format serialization.** Variant uses shredding in Parquet, binary in ORC, unions in Avro. Without a format abstraction, each new type means new `match` arms everywhere.
+
+5. **Arrow version coupling.** The core crate depends on specific `arrow-*` versions. Upgrading Arrow in `datafusion` or other integrations forces lockstep upgrades across the dependency graph.
+
+### Prior work
+
+The Java project shipped `FormatModel<D, S>` in February 2026 (PR [#12774](https://github.com/apache/iceberg/pull/12774)). Java's design uses two generic parameters (data type `D` and engine schema `S`) with a registry keyed by `(FileFormat, Class<?>)`. PyIceberg has an open proposal ([#3100](https://github.com/apache/iceberg-python/issues/3100)) that drops generics entirely, keying on file format alone.
+
+This RFC proposes a composable three-layer architecture that separates the in-memory processing representation from the file format layer, using Rust's trait system for static dispatch within a layer and dynamic dispatch at layer boundaries. It aligns with the kernel architecture proposed in [#1817](https://github.com/apache/iceberg-rust/issues/1817) and the modularization tracked in [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+## Goals
+
+1. Define a composable three-layer architecture where the file format, in-memory processing representation, and engine are independent axes of variation. A `DataBatch` trait defines the processing contract. `FormatReader` and `FormatWriter` traits bridge file formats to batch types. No layer imposes a conversion on another.
+
+2. Remove hard-coded Parquet assumptions from scan and write orchestration.
+
+3. Establish the crate architecture that allows the core `iceberg` kernel to be representation-agnostic, decoupling Arrow version pinning from the core library.
+
+4. Provide interoperability with Java and Python Iceberg implementations at the conceptual level (same registry key semantics, same TCK coverage) while using Rust's trait system for zero-cost abstraction within a layer.
+
+## Non-Goals
+
+1. **Ship new format implementations.** This RFC lands the abstraction and a Parquet-with-Arrow implementation. ORC, Avro, Vortex, and Lance are follow-up work.
+
+2. **Runtime library loading.** Rust has no stable ABI. No format under discussion requires this.
+
+3. **Puffin support.** Puffin files have a different lifecycle and are handled separately.
+
+4. **Redesign the writer trait hierarchy.** The existing `IcebergWriter` and `FileWriter` layering is sound. This RFC adds beneath it, not a replacement.
+
+5. **Implement variant shredding.** The hooks are provided. Implementation depends on [#2188](https://github.com/apache/iceberg-rust/pull/2188).
+
+6. **Complete crate separation.** This RFC establishes trait boundaries. Extraction is follow-up work per [#1817](https://github.com/apache/iceberg-rust/issues/1817) and [#1819](https://github.com/apache/iceberg-rust/issues/1819).
+
+7. **Change the Iceberg table spec.** Rust-only API change.
+
+8. **Modify manifest paths.** Manifests remain in Avro via existing code.
+
+## Design
+
+### Architecture
+
+Three independent axes determine how data flows through the system:
+
+| Axis | Controlled by | Determined when | Can change mid-session? |
+|------|---------------|-----------------|-------------------------|
+| **In-memory representation** | The engine embedding Iceberg (DataFusion, Spark, Comet) or the direct library user | Session start | No |
+| **Processing operations** | The Iceberg kernel | N/A (always available) | N/A |
+| **File format** | The table creator, stored in table metadata | Table creation | No (one format per table) |
+
+An Iceberg session has one in-memory representation. A table has one data file format. A session may scan multiple tables with different formats, and a proposed multi-table transaction could span Parquet and ORC tables. In all cases, batches of data flow through three layers with no intermediate conversions:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ Engine Layer │
+│ The engine's native data representation. │
+│ Examples: Arrow RecordBatch, Vortex compressed arrays │
+│ The engine provides a type that implements DataBatch. │
+│ This choice is fixed for the session. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ Processing Layer │
+│ Iceberg operations on in-memory data: expression evaluation, │
+│ partition transforms, schema evolution, metrics collection, │
+│ constant injection, delete application. │
+│ Works with the concrete batch type chosen by the engine. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │ (same concrete type throughout)
+┌─────────────────────────────────────┼───────────────────────────┐
+│ File Format Layer │
+│ FormatReader/FormatWriter implementations read/write physical │
+│ files, producing/consuming the engine's batch type directly. │
+│ One implementation per (format, batch type) pair. │
+└─────────────────────────────────────┬───────────────────────────┘
+ │
+ Physical Files
+ (Parquet, ORC, Avro, ...)
+```
+
+### Separation of responsibilities
+
+Each implementor has a clearly defined contract. The kernel provides processing operations that format and engine developers do not need to reimplement.
+
+**Format implementor** (for example, adding ORC):
+
+| MUST implement | Handled by the kernel (unless the format opts in) |
+|---|---|
+| Decode requested columns from disk (projection is an I/O decision) | Residual row-level filtering |
+| Encode batches to disk | Schema evolution (add columns, widen types, reorder) |
+| Collect file-level metrics during write | Constant injection (metadata columns like `_file`, `_pos`) |
+| Handle format-specific optimizations (variant shredding, statistics-based chunk skipping) | Delete application (merge-on-read) |
+| Report what operations the reader already handled | Partition routing, file rolling, location generation |
+
+A format reader MAY handle operations from the right column during I/O if it can do so more efficiently (for example, Parquet reading an `int32` column directly into `int64` for schema evolution). When it does, it reports what it handled via `ReadResult` so the kernel skips the redundant pass. The kernel handles anything the format does not.
+
+**DataBatch implementor** (for example, adding GPU-native buffers):
+
+| MUST implement | Gets for free |
+|---|---|
+| `filter` — evaluate a predicate against the data | All kernel orchestration |
+| `project` — subset columns by field ID | All format readers that produce the type |
+| `evolve_schema` — widen types, add columns with defaults | TCK to validate correctness |
+| `inject_constants` — add metadata column values | |
+| `column_metrics` — compute min/max/null/NaN stats | |
+
+**Engine implementor** (for example, adding Comet):
+
+| What you want | What you implement |
+|---|---|
+| Arrow works for your engine | Nothing. Use the shipped `RecordBatch` default and existing format readers. |
+| Custom in-memory type | `impl DataBatch for YourType`. Pass TCK Layer 1. Register format readers that produce your type. |
+| Full control over execution | Bypass the kernel's default scan loop. Use format readers as a stream source and drive your own processing. |
+
+### Core traits
+
+**`DataBatch`** defines the contract for an in-memory data representation that the Iceberg kernel can process. The engine chooses a concrete batch type at session start and that choice does not change. `DataBatch` methods (`filter`, `project`, `evolve_schema`, `inject_constants`) return `Self` — they operate on the concrete type. Implementations have full freedom to fuse operations internally for performance. The kernel does not decompose operations into steps or dictate intermediate materializations.
+
+The shipped default is `impl DataBatch for RecordBatch`. Supporting a new representation requires implementing `DataBatch` and passing the TCK Layer 1 suite.
+
+**`FormatReader`** reads a file and produces a stream of batches. Its contract:
+
+- The reader MUST decode only the columns in `ReadOptions::schema` (projection is an I/O concern).
+- The reader MAY skip chunks that cannot match `ReadOptions::predicate` using format-level statistics.
+- The reader MAY support byte-range splits via `ReadOptions::split_start` / `split_length`.
+- The reader MAY handle schema evolution or other kernel operations during decode if the format can do so more efficiently (e.g., Parquet type promotion during decompression).
+- The reader MUST NOT handle partitioning, transaction management, or name mapping (field ID resolution for files not written by Iceberg). Those are always kernel responsibilities.
+
+The reader returns a `ReadResult` containing the batch stream and reporting what operations the reader handled. The kernel applies any remaining operations (residual filtering, schema evolution, constant injection, delete application) that the format did not.
+
+**`FormatWriter`** writes batches to a file. Its contract:
+
+- The writer MUST encode batches into the file format.
+- The writer MUST collect file-level metrics per `WriteOptions::metrics_config`.
+- The writer MUST handle format-specific encoding concerns (like variant shredding layout for Parquet).
+- The writer MUST NOT handle partitioning, sorting, or transaction management. The caller sends pre-partitioned, pre-sorted batches. The kernel handles commit.
+
+Both traits are dyn-compatible: no associated types, no generic parameters. The registry stores them as `Arc<dyn FormatReader>` and `Arc<dyn FormatWriter>`. This is the same pattern as `Storage` and `StorageFactory` in the IO layer.
+
+**`ReadOptions`** configures a read: Iceberg projection schema, filter predicate for pushdown, byte-range splits, case sensitivity, batch size, format-specific properties, and constant values for metadata columns.
+
+**`WriteOptions`** configures a write: Iceberg schema, format-specific properties, file-level metadata, content type, metrics collection configuration, overwrite flag, and encryption parameters.
+
+Both structs are `#[non_exhaustive]`. Fields are added without a breaking change.
+
+**`ReadResult`** contains the batch stream and communicates what the reader handled. Additional fields for reporting other handled operations (schema evolution, constant injection) will be added as formats opt in to handling them. The struct is `#[non_exhaustive]` so fields are added without a breaking change.
+
+```rust
+#[non_exhaustive]
+pub struct ReadResult {
+ /// The stream of batches with requested columns decoded.
+ pub stream: DataStream,
+ /// The portion of the predicate the reader could NOT evaluate.
+ /// None means the reader fully handled the predicate.
+ /// Some(predicate) means the kernel must apply residual filtering.
+ pub residual_predicate: Option<BoundPredicate>,
+ /// Whether the reader applied schema evolution during decode.
+ /// If true, the kernel skips the post-read schema evolution pass.
+ pub schema_evolved: bool,
+}
+```
+
+**`FormatFileWriter`** accepts batches via `write_batch(&dyn DataBatch)` and returns a `WriterResult` on `close()`. `WriterResult` contains `Vec<DataFileBuilder>` — builders rather than completed `DataFile` values because the format layer fills format-specific fields (file size, column sizes, metrics from its own metadata) while the kernel fills Iceberg-level fields (partition values, sequence number, snapshot ID).
+
+**`FormatRegistry`** maps `(DataFileFormat, TypeId)` pairs to reader and writer implementations. The first dimension is the file format (determined at runtime from table metadata). The second dimension is the batch type (fixed at session start by the engine). This matches Java's `FormatModelRegistry` key of `(FileFormat, Class<?>)`.
+
+The registry exposes `reader::<B>(format)` and `writer::<B>(format)`, where `B` is the batch type the engine chose. Errors are eager: if `(format, B)` is not registered, the call fails immediately at scan planning time rather than mid-stream during I/O. This is a runtime contract — the `TypeId` tells the registry what batch type the caller expects, but the type system does not enforce that the reader actually produces that type. A reader that returns the wrong type fails at the first downcast with a clear error. Java makes the same tradeoff with `Class<?>`.
+
+A process-wide default registry is available via `OnceLock`. Custom registries can be provided to `TableScanBuilder` for tests or restricted configurations.
+
+**Default scan loop (hybrid execution model).** The kernel provides a default scan loop that pulls batches from the format reader, checks `ReadResult` to see what the format already handled, and applies any remaining operations (residual filtering, schema evolution, constant injection, delete application). Engines that want full control (DataFusion, Comet) bypass this loop and drive execution themselves — they use the format reader as a stream source and apply their own processing. The `DataBatch` methods are available as a toolbox, not mandated as a pipeline.
+
+**Supporting types:** `ColumnMetrics` stores bounds as typed `Datum` values, matching how `DataFile` already stores bounds. `MetricsConfig` and `MetricsMode` control collection granularity. `FileContent` distinguishes data from equality and position deletes. All config structs and enums are `#[non_exhaustive]`.
+
+### Feature flags
+
+Format implementations are gated behind feature flags (`format-parquet` on by default, `format-orc` when ORC lands). The default feature set includes every implemented format. No configuration is needed for downstream crates using default features. Feature flags exist for crates that want to minimize binary size or compile time. This matches `datafusion`, `opendal`, and `reqwest`.
+
+### Module layout
+
+```
+crates/iceberg/src/formats/
+├── mod.rs # Module declarations and public re-exports
+├── traits.rs # DataBatch, FormatReader, FormatWriter,
+│ # FormatFileWriter, ReadOptions, WriteOptions,
+│ # ColumnMetrics, MetricsConfig, MetricsMode,
+│ # FileContent, WriterResult,
+│ # VariantShredding (future extension, not yet wired)
+├── registry.rs # FormatRegistry, default_format_registry()
+└── parquet/
+ ├── mod.rs # Submodule declarations and re-exports
+ ├── model.rs # ParquetArrowModel (implements FormatReader + FormatWriter)
+ ├── reader.rs # Parquet reader internals
+ └── writer.rs # Parquet writer (implements FormatFileWriter)
+```
+
+Additional formats land as `formats/orc/`, `formats/avro_data/`, each following the same structure.
+
+### Target crate architecture
+
+The end-state after crate separation (follow-up work, not part of Phase 1). All crates live in the same repository and workspace. Phase 1 ships everything in the existing `iceberg` crate.
+
+```
+iceberg (core kernel) - traits only, no arrow/parquet dependency
+iceberg-arrow - impl DataBatch for RecordBatch
+iceberg-format-parquet - ParquetArrowModel
+iceberg-format-orc - OrcArrowModel
+iceberg-datafusion - TableProvider, depends on above
+```
+
+This separation enables engines to pin their own Arrow version independently. For example, a Comet integration could depend on `iceberg` (the kernel) and provide its own `DataBatch` implementation and format readers using a newer Arrow version, without conflicting with `iceberg-arrow` or `iceberg-format-parquet`.
+
+## Design Rationale
Review Comment:
For reference, there was a write-up on the AAL integration and its optimizations here: https://docs.google.com/document/d/13shy0RWotwfWC_qQksb95PXdi-vSUCKQyDzjoExQEN0/edit?tab=t.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org