You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2026/05/19 06:17:18 UTC

[I] Spec: Clarify Avro schema for `day` partition transform fields in manifests: plain `int` vs logical `date` [iceberg]

kevinjqliu opened a new issue, #16414:
URL: https://github.com/apache/iceberg/issues/16414

   This issue is ai assisted, and I have verified all the details.
   
   ## Summary
   
   The Iceberg spec says the `day` partition transform has result type `int`, where the value is the number of days from `1970-01-01`.
   
   A literal reading of the spec plus the Avro type mapping implies that manifest partition fields produced by `day(...)` should be encoded as plain Avro `int`:
   
   ```json
   "type": "int"
   ```
   
   However, several Iceberg implementations model `day(...)` partition fields as an Iceberg `date` / Avro logical `date`, and therefore write:
   
   ```json
   {
     "type": "int",
     "logicalType": "date"
   }
   ```
   
   Both forms have the same Avro physical encoding: an integer day count. The difference is the Avro schema logical type annotation. This can still affect interoperability because some readers validate manifest schemas strictly.
   
   This issue is to document the ambiguity and raise a discussion about the intended canonical behavior.
   
   ---
   
   ## Quick comparison
   
   | Implementation | Internal result type for `day(...)` | Manifest Avro schema written | Accepts plain Avro `int`? | Accepts Avro `int` + `logicalType: date`? |
   |---|---|---|---:|---:|
   | Spec literal interpretation | `int` | plain Avro `int` | N/A | N/A |
   | Java Iceberg, `apache/iceberg` | `DateType` | Avro logical `date` | Yes | Yes |
   | PyIceberg, `apache/iceberg-python` | `DateType` | Avro logical `date` | Generally yes in normal decode path; projection caveat | Yes |
   | Go, `apache/iceberg-go` current `main` | `Int32Type` | plain Avro `int` | Yes | Not clearly guaranteed on merged `main`; open PR proposes logical `date` |
   | Rust, `apache/iceberg-rust` current `main` | `Date` | Avro logical `date` | Not clearly guaranteed in normal manifest reader | Yes |
   
   ---
   
   ## Spec references and interpretation
   
   ### Partition transform result type
   
   The partition transform table says:
   
   > `day` Extract a date or timestamp day, as days from 1970-01-01 `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` `int`
   
   Source: https://iceberg.apache.org/spec/#partition-transforms
   
   This explicitly lists the result type for `day` as `int`, not `date`.
   
   ### Manifest partition field schema
   
   The manifest section says:
   
   > A manifest is an immutable Avro file that lists data files or delete files, along with each file's partition data tuple, metrics, and tracking information.
   
   For the `partition` field in a manifest entry, the spec says:
   
   > Partition data tuple, schema based on the partition spec output using partition field ids for the struct field ids
   
   Source: https://iceberg.apache.org/spec/#manifests
   
   This means the Avro schema for the manifest `partition` struct is derived from the partition spec output. Therefore, the transform result type matters for the Avro schema used in the manifest.
   
   ### Avro type mapping
   
   Appendix A says:
   
   > Values should be stored in Avro using the Avro types and logical type annotations in the table below.
   
   The Avro mapping table distinguishes Iceberg `int` from Iceberg `date`:
   
   | Iceberg type | Avro representation | Note |
   |---|---|---|
   | `int` | `int` | |
   | `date` | `{ "type": "int", "logicalType": "date" }` | Stores days from 1970-01-01. |
   
   Source: https://iceberg.apache.org/spec/#appendix-a-format-specific-requirements
   
   ### Literal spec interpretation
   
   Because `day(...)` has result type `int`, and Iceberg `int` maps to plain Avro `int`, the literal spec interpretation appears to be:
   
   ```json
   "type": "int"
   ```
   
   not:
   
   ```json
   {
     "type": "int",
     "logicalType": "date"
   }
   ```
   
   The spec uses the word `should` for the Avro mapping, so this may not be an absolute `MUST`-style requirement. But it is unclear whether the intended canonical manifest schema for `day(...)` partition fields is plain `int` or logical `date`.
   
   ---
   
   ## Implementation behavior
   
   ## 1. Java Iceberg: `apache/iceberg`
   
   ### What Java writes
   
   Java Iceberg models `day(...)` as `DateType`, not `IntegerType`.
   
   Relevant code paths:
   
   - `Dates.getResultType(...)`: https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/transforms/Dates.java
   - `Timestamps.getResultType(...)`: https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/transforms/Timestamps.java
   - `TypeToSchema.DATE_SCHEMA`: https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/avro/TypeToSchema.java
   
   For day granularity, both date and timestamp transforms return `Types.DateType.get()`. Java's Avro conversion writes `DateType` as Avro `int` annotated with logical type `date`.
   
   So Java writes manifest partition fields for `day(...)` as:
   
   ```json
   {
     "type": "int",
     "logicalType": "date"
   }
   ```
   
   ### What Java accepts
   
   Java Iceberg accepts both forms.
   
   | Manifest Avro schema form | Accepted? | Behavior |
   |---|---:|---|
   | Plain Avro `int` | Yes | Reads as raw integer day count |
   | Avro `int` with `logicalType: date` | Yes | Reads as raw integer day count |
   
   Relevant reader path:
   
   - `InternalReader.primitive(...)`: https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/avro/InternalReader.java
   
   The Java Avro reader maps Avro logical type `date` to `ValueReaders.ints()`. Plain Avro `INT` also maps to `ValueReaders.ints()`, except for the generic int-to-long widening case.
   
   ### Java summary
   
   Java writes the de facto implementation convention:
   
   ```json
   {
     "type": "int",
     "logicalType": "date"
   }
   ```
   
   but reads both the Java-style annotated form and the spec-literal plain `int` form.
   
   ---
   
   ## 2. PyIceberg: `apache/iceberg-python`
   
   ### What PyIceberg writes
   
   PyIceberg also models `day(...)` as `DateType`.
   
   Relevant code paths:
   
   - `DayTransform.result_type`: https://github.com/apache/iceberg-python/blob/main/pyiceberg/transforms.py
   - `PartitionSpec.partition_type(...)`: https://github.com/apache/iceberg-python/blob/main/pyiceberg/partitioning.py
   - Avro schema conversion: https://github.com/apache/iceberg-python/blob/main/pyiceberg/utils/schema_conversion.py
   
   The `DayTransform.result_type` implementation returns `DateType()`. The code comment explains the intent: the physical representation still conforms because `DateType` is internally converted to an int, while returning `DateType` gives a more human-readable display representation.
   
   PyIceberg's Avro conversion maps:
   
   | PyIceberg type | Avro schema |
   |---|---|
   | `IntegerType` | `"int"` |
   | `DateType` | `{ "type": "int", "logicalType": "date" }` |
   
   So PyIceberg writes manifest partition fields for `day(...)` as:
   
   ```json
   {
     "type": "int",
     "logicalType": "date"
   }
   ```
   
   ### What PyIceberg accepts
   
   PyIceberg recognizes both schema shapes at schema-conversion level.
   
   | Manifest Avro schema form | Converted PyIceberg type | Practical read behavior |
   |---|---|---|
   | Plain Avro `int` | `IntegerType` | Integer Avro encoding is readable in normal decode path |
   | Avro `int` with `logicalType: date` | `DateType` | Integer Avro encoding is readable as date logical type |
   
   Relevant reader path:
   
   - Avro reader implementation: https://github.com/apache/iceberg-python/blob/main/pyiceberg/avro/reader.py
   
   At the primitive reader level, `DateReader` subclasses `IntegerReader`, so both forms use the same integer Avro encoding.
   
   ### PyIceberg caveat
   
   PyIceberg is less explicitly tolerant than Java when a caller forces a projected/read schema expecting `DateType` while the file schema says plain `IntegerType`. The normal file-schema decode path can handle both integer-encoded forms, but explicit int-to-date projection compatibility is not as clear as Java's tolerant reader branch.
   
   ### PyIceberg summary
   
   PyIceberg writes the same de facto convention as Java:
   
   ```json
   {
     "type": "int",
     "logicalType": "date"
   }
   ```
   
   It can read both integer-encoded forms in the normal Avro decode path, with the projection caveat above.
   
   ---
   
   ## 3. Go: `apache/iceberg-go`
   
   ### What Go writes on current `main`
   
   Current `iceberg-go` appears to follow the literal spec interpretation.
   
   Relevant code paths:
   
   - `DayTransform.ResultType(...)`: https://github.com/apache/iceberg-go/blob/main/transforms.go
   - `PartitionSpec.PartitionType(...)`: https://github.com/apache/iceberg-go/blob/main/partitions.go
   - Avro schema conversion: https://github.com/apache/iceberg-go/blob/main/schema_conversions.go
   
   `DayTransform.ResultType(...)` returns `PrimitiveTypes.Int32`. `PartitionSpec.PartitionType(...)` uses the transform result type when constructing the partition struct. The Avro conversion maps `Int32Type` to Avro `IntNode`, while `DateType` maps to Avro `DateNode`.
   
   So current Go writes manifest partition fields for `day(...)` as:
   
   ```json
   "type": "int"
   ```
   
   ### What Go accepts
   
   | Manifest Avro schema form | Current `iceberg-go` status |
   |---|---|
   | Plain Avro `int` | Confirmed: this is what it writes and expects |
   | Avro `int` with `logicalType: date` | Not clearly guaranteed on merged `main` |
   
   The Go manifest reader opens the Avro OCF file, uses the file schema, and builds logical-type metadata from the actual Avro schema, including nested partition fields. However, I would not call Java-style "accepts both" behavior confirmed without a merged explicit compatibility test/path.
   
   Relevant reader path:
   
   - Manifest code: https://github.com/apache/iceberg-go/blob/main/manifest.go
   
   ### Open Go PR changing this behavior
   
   There is an open Go PR proposing to encode `day(...)` partition fields with Avro logical `date` for interoperability with engines that expect the Java-style manifest schema:
   
   - https://github.com/apache/iceberg-go/pull/915
   
   The PR description says current Go encoded all `Int32` partition fields as plain Avro `int`, and that this caused interoperability issues with engines expecting the logical `date` annotation.
   
   As of this writing, that PR is open.
   
   ### Go summary
   
   Current Go writes the spec-literal form:
   
   ```json
   "type": "int"
   ```
   
   The open PR proposes moving Go toward the Java/PyIceberg/Rust convention:
   
   ```json
   {
     "type": "int",
     "logicalType": "date"
   }
   ```
   
   ---
   
   ## 4. Rust: `apache/iceberg-rust`
   
   ### What Rust writes
   
   Current `iceberg-rust` follows the Java/PyIceberg convention.
   
   Relevant code paths:
   
   - `Transform::Day.result_type(...)`: https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/spec/transform.rs
   - Avro schema conversion: https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/avro/schema.rs
   - Manifest writer: https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/spec/manifest/writer.rs
   
   `Transform::Day.result_type(...)` maps timestamp/date source types to `PrimitiveType::Date`. Rust's Avro schema conversion maps `PrimitiveType::Date` to Avro date, while `PrimitiveType::Int` maps to plain Avro int. The manifest writer builds the manifest Avro schema from the partition type.
   
   So Rust writes manifest partition fields for `day(...)` as:
   
   ```json
   {
     "type": "int",
     "logicalType": "date"
   }
   ```
   
   ### Why Rust changed
   
   Rust originally mapped `Transform::Day` to `Int`. That caused manifest parsing failures for Java-written manifests whose day partition field used Avro logical `date`.
   
   Related issue and PR:
   
   - Issue: https://github.com/apache/iceberg-rust/issues/478
   - PR: https://github.com/apache/iceberg-rust/pull/479
   
   PR #479 changed `Transform::Day` to map partition types to `Date` rather than `Int` for consistency with the Java/reference implementation.
   
   ### What Rust accepts
   
   | Manifest Avro schema form | Current `iceberg-rust` status |
   |---|---|
   | Avro `int` with `logicalType: date` | Confirmed: this is what current Rust writes and expects |
   | Plain Avro `int` | Not clearly guaranteed in the normal manifest reader |
   
   The normal Rust manifest reader reconstructs the expected manifest schema from table metadata and the partition spec, then reads with that expected schema. The historical bug report shows that a mismatch between the expected schema and the manifest file schema caused parsing failure. Therefore, Rust should not be assumed to accept plain Avro `int` for `day(...)` partition fields unless an explicit compatibility path or test is added.
   
   Relevant reader path:
   
   - Manifest reader code: https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/spec/manifest/mod.rs
   
   ### Rust summary
   
   Rust writes the de facto implementation convention:
   
   ```json
   {
     "type": "int",
     "logicalType": "date"
   }
   ```
   
   Unlike Java, Rust should not currently be assumed to accept both schema forms in all manifest reading paths.
   
   ---
   
   ## Why this matters
   
   Although both schemas encode the same integer day count, the Avro schema annotation can matter for readers that construct an expected manifest schema and use strict Avro schema resolution.
   
   This has led to divergent implementation behavior:
   
   - Java/PyIceberg/Rust write logical `date`.
   - Go currently writes plain `int`.
   - Java accepts both forms.
   - Rust and Go may not accept both forms in all manifest reading paths.
   
   This creates uncertainty for new implementations and for existing implementations trying to decide whether to prioritize literal spec compliance or compatibility with the dominant Java/Spark convention.
   
   ---
   
   ## Questions for clarification
   
   Could the Iceberg spec clarify the intended canonical Avro schema for manifest partition fields produced by the `day(...)` transform?
   
   Specifically:
   
   1. Should the canonical transform result type for `day(...)` remain `int`, as currently documented?
   
   2. If the result type remains `int`, should manifest partition fields for `day(...)` be written as plain Avro `int`?
   
      ```json
      "type": "int"
      ```
   
   3. Or should `day(...)` be considered a special case where the logical result is date-like and writers should use Avro logical `date`?
   
      ```json
      {
        "type": "int",
        "logicalType": "date"
      }
      ```
   
   4. Should readers be required or strongly recommended to accept both schema forms for `day(...)` partition fields?
   
   5. If both forms are accepted, should one form be documented as canonical for writers?
   
   6. Should the partition transform result type table be changed from `int` to `date` for `day(...)`, or should the Avro manifest encoding guidance be changed to explicitly allow/use logical `date` for `day(...)` despite the transform table saying `int`?
   
   ---
   
   ## Possible resolutions
   
   ### Option A: Keep the spec-literal interpretation
   
   Clarify that `day(...)` result type is `int` and writers should use plain Avro `int` for manifest partition fields.
   
   Pros:
   
   - Matches the current partition transform table.
   - Matches Appendix A's mapping for Iceberg `int`.
   - Matches current `iceberg-go` behavior.
   
   Cons:
   
   - Diverges from current Java/PyIceberg/Rust behavior.
   - May require existing implementations to change writer behavior or document a compatibility exception.
   - Could cause compatibility problems with engines that expect logical `date`.
   
   ### Option B: Change or clarify `day(...)` result type to `date`
   
   Clarify that `day(...)` returns a date-like value and should be written as Avro logical `date`.
   
   Pros:
   
   - Matches Java/PyIceberg/Rust behavior.
   - Matches the de facto ecosystem convention.
   - Provides more human-readable metadata/table output.
   
   Cons:
   
   - Requires changing or qualifying the current transform result type table.
   - The transform table currently describes the result type as `int`.
   - Could make plain-`int` writers look non-compliant despite following the current spec text.
   
   ### Option C: Keep `int` result type but explicitly allow logical `date` in manifests
   
   Clarify that the physical value of `day(...)` is still an integer day count, but manifest Avro schemas may use either:
   
   ```json
   "type": "int"
   ```
   
   or:
   
   ```json
   {
     "type": "int",
     "logicalType": "date"
   }
   ```
   
   and readers should accept both.
   
   Pros:
   
   - Acknowledges the existing ecosystem split.
   - Avoids declaring current Java/PyIceberg/Rust or Go behavior invalid.
   - Encourages interoperability.
   
   Cons:
   
   - Leaves two writer encodings unless a canonical writer form is also chosen.
   - Requires readers to implement compatibility handling.
   
   ---
   
   ## Suggested clarification
   
   A possible clarification could be:
   
   > For partition fields produced by the `day` transform, the stored value is an integer number of days from `1970-01-01`. Manifest readers should accept both plain Avro `int` and Avro `int` annotated with `logicalType: date` for these fields. Writers should use [chosen canonical form].
   
   The main clarification needed is the final sentence: which form should writers use?
   
   ---
   
   ## Related issues and PRs
   
   - Java issue asking about this mismatch: https://github.com/apache/iceberg/issues/10616
   - Older Java issue discussing timestamp/day returning date for display: https://github.com/apache/iceberg/issues/279
   - Rust issue caused by Java date-logical-type manifests: https://github.com/apache/iceberg-rust/issues/478
   - Rust PR changing `Transform::Day` to `Date`: https://github.com/apache/iceberg-rust/pull/479
   - Go PR proposing logical `date` for day-transform partition fields: https://github.com/apache/iceberg-go/pull/915
   
   ---
   
   ## Requested outcome
   
   Please clarify the intended spec behavior for `day(...)` partition fields in manifest Avro schemas:
   
   - Is plain Avro `int` the canonical writer form?
   - Is Avro logical `date` the canonical writer form?
   - Are both forms allowed?
   - Should readers be required or strongly recommended to accept both?
   
   This would help align Java, PyIceberg, Go, Rust, and other implementations around one documented compatibility expectation.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Spec ambiguity: Avro schema for `day` partition transform fields in manifests [iceberg]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #16414:
URL: https://github.com/apache/iceberg/issues/16414#issuecomment-4510315302

   Linking to previous discussion, https://github.com/apache/iceberg/issues/10616#issuecomment-2200191427
   looks like we went back and forth on this a couple of times already 😄 
   
   lets figure this out once and for all 🤞 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Spec ambiguity: Avro schema for `day` partition transform fields in manifests [iceberg]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #16414:
URL: https://github.com/apache/iceberg/issues/16414#issuecomment-4696712203

   Changed the table spec to document this in #16446
   
   Hope we don't need to re-litigate this in the future 🤞 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Spec ambiguity: Avro schema for `day` partition transform fields in manifests [iceberg]

Posted by "zhjwpku (via GitHub)" <gi...@apache.org>.
zhjwpku commented on issue #16414:
URL: https://github.com/apache/iceberg/issues/16414#issuecomment-4639558799

   FWIW, iceberg-cpp also changed the result type of DayTransform from `int` to `date`[1]
   
   [1] https://github.com/apache/iceberg-cpp/pull/285


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Spec ambiguity: Avro schema for `day` partition transform fields in manifests [iceberg]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu closed issue #16414: Spec ambiguity: Avro schema for `day` partition transform fields in manifests
URL: https://github.com/apache/iceberg/issues/16414


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org