You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2026/05/19 06:17:18 UTC
[I] Spec: Clarify Avro schema for `day` partition transform fields in manifests: plain `int` vs logical `date` [iceberg]
kevinjqliu opened a new issue, #16414:
URL: https://github.com/apache/iceberg/issues/16414
This issue is ai assisted, and I have verified all the details.
## Summary
The Iceberg spec says the `day` partition transform has result type `int`, where the value is the number of days from `1970-01-01`.
A literal reading of the spec plus the Avro type mapping implies that manifest partition fields produced by `day(...)` should be encoded as plain Avro `int`:
```json
"type": "int"
```
However, several Iceberg implementations model `day(...)` partition fields as an Iceberg `date` / Avro logical `date`, and therefore write:
```json
{
"type": "int",
"logicalType": "date"
}
```
Both forms have the same Avro physical encoding: an integer day count. The difference is the Avro schema logical type annotation. This can still affect interoperability because some readers validate manifest schemas strictly.
This issue is to document the ambiguity and raise a discussion about the intended canonical behavior.
---
## Quick comparison
| Implementation | Internal result type for `day(...)` | Manifest Avro schema written | Accepts plain Avro `int`? | Accepts Avro `int` + `logicalType: date`? |
|---|---|---|---:|---:|
| Spec literal interpretation | `int` | plain Avro `int` | N/A | N/A |
| Java Iceberg, `apache/iceberg` | `DateType` | Avro logical `date` | Yes | Yes |
| PyIceberg, `apache/iceberg-python` | `DateType` | Avro logical `date` | Generally yes in normal decode path; projection caveat | Yes |
| Go, `apache/iceberg-go` current `main` | `Int32Type` | plain Avro `int` | Yes | Not clearly guaranteed on merged `main`; open PR proposes logical `date` |
| Rust, `apache/iceberg-rust` current `main` | `Date` | Avro logical `date` | Not clearly guaranteed in normal manifest reader | Yes |
---
## Spec references and interpretation
### Partition transform result type
The partition transform table says:
> `day` Extract a date or timestamp day, as days from 1970-01-01 `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` `int`
Source: https://iceberg.apache.org/spec/#partition-transforms
This explicitly lists the result type for `day` as `int`, not `date`.
### Manifest partition field schema
The manifest section says:
> A manifest is an immutable Avro file that lists data files or delete files, along with each file's partition data tuple, metrics, and tracking information.
For the `partition` field in a manifest entry, the spec says:
> Partition data tuple, schema based on the partition spec output using partition field ids for the struct field ids
Source: https://iceberg.apache.org/spec/#manifests
This means the Avro schema for the manifest `partition` struct is derived from the partition spec output. Therefore, the transform result type matters for the Avro schema used in the manifest.
### Avro type mapping
Appendix A says:
> Values should be stored in Avro using the Avro types and logical type annotations in the table below.
The Avro mapping table distinguishes Iceberg `int` from Iceberg `date`:
| Iceberg type | Avro representation | Note |
|---|---|---|
| `int` | `int` | |
| `date` | `{ "type": "int", "logicalType": "date" }` | Stores days from 1970-01-01. |
Source: https://iceberg.apache.org/spec/#appendix-a-format-specific-requirements
### Literal spec interpretation
Because `day(...)` has result type `int`, and Iceberg `int` maps to plain Avro `int`, the literal spec interpretation appears to be:
```json
"type": "int"
```
not:
```json
{
"type": "int",
"logicalType": "date"
}
```
The spec uses the word `should` for the Avro mapping, so this may not be an absolute `MUST`-style requirement. But it is unclear whether the intended canonical manifest schema for `day(...)` partition fields is plain `int` or logical `date`.
---
## Implementation behavior
## 1. Java Iceberg: `apache/iceberg`
### What Java writes
Java Iceberg models `day(...)` as `DateType`, not `IntegerType`.
Relevant code paths:
- `Dates.getResultType(...)`: https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/transforms/Dates.java
- `Timestamps.getResultType(...)`: https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/transforms/Timestamps.java
- `TypeToSchema.DATE_SCHEMA`: https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/avro/TypeToSchema.java
For day granularity, both date and timestamp transforms return `Types.DateType.get()`. Java's Avro conversion writes `DateType` as Avro `int` annotated with logical type `date`.
So Java writes manifest partition fields for `day(...)` as:
```json
{
"type": "int",
"logicalType": "date"
}
```
### What Java accepts
Java Iceberg accepts both forms.
| Manifest Avro schema form | Accepted? | Behavior |
|---|---:|---|
| Plain Avro `int` | Yes | Reads as raw integer day count |
| Avro `int` with `logicalType: date` | Yes | Reads as raw integer day count |
Relevant reader path:
- `InternalReader.primitive(...)`: https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/avro/InternalReader.java
The Java Avro reader maps Avro logical type `date` to `ValueReaders.ints()`. Plain Avro `INT` also maps to `ValueReaders.ints()`, except for the generic int-to-long widening case.
### Java summary
Java writes the de facto implementation convention:
```json
{
"type": "int",
"logicalType": "date"
}
```
but reads both the Java-style annotated form and the spec-literal plain `int` form.
---
## 2. PyIceberg: `apache/iceberg-python`
### What PyIceberg writes
PyIceberg also models `day(...)` as `DateType`.
Relevant code paths:
- `DayTransform.result_type`: https://github.com/apache/iceberg-python/blob/main/pyiceberg/transforms.py
- `PartitionSpec.partition_type(...)`: https://github.com/apache/iceberg-python/blob/main/pyiceberg/partitioning.py
- Avro schema conversion: https://github.com/apache/iceberg-python/blob/main/pyiceberg/utils/schema_conversion.py
The `DayTransform.result_type` implementation returns `DateType()`. The code comment explains the intent: the physical representation still conforms because `DateType` is internally converted to an int, while returning `DateType` gives a more human-readable display representation.
PyIceberg's Avro conversion maps:
| PyIceberg type | Avro schema |
|---|---|
| `IntegerType` | `"int"` |
| `DateType` | `{ "type": "int", "logicalType": "date" }` |
So PyIceberg writes manifest partition fields for `day(...)` as:
```json
{
"type": "int",
"logicalType": "date"
}
```
### What PyIceberg accepts
PyIceberg recognizes both schema shapes at schema-conversion level.
| Manifest Avro schema form | Converted PyIceberg type | Practical read behavior |
|---|---|---|
| Plain Avro `int` | `IntegerType` | Integer Avro encoding is readable in normal decode path |
| Avro `int` with `logicalType: date` | `DateType` | Integer Avro encoding is readable as date logical type |
Relevant reader path:
- Avro reader implementation: https://github.com/apache/iceberg-python/blob/main/pyiceberg/avro/reader.py
At the primitive reader level, `DateReader` subclasses `IntegerReader`, so both forms use the same integer Avro encoding.
### PyIceberg caveat
PyIceberg is less explicitly tolerant than Java when a caller forces a projected/read schema expecting `DateType` while the file schema says plain `IntegerType`. The normal file-schema decode path can handle both integer-encoded forms, but explicit int-to-date projection compatibility is not as clear as Java's tolerant reader branch.
### PyIceberg summary
PyIceberg writes the same de facto convention as Java:
```json
{
"type": "int",
"logicalType": "date"
}
```
It can read both integer-encoded forms in the normal Avro decode path, with the projection caveat above.
---
## 3. Go: `apache/iceberg-go`
### What Go writes on current `main`
Current `iceberg-go` appears to follow the literal spec interpretation.
Relevant code paths:
- `DayTransform.ResultType(...)`: https://github.com/apache/iceberg-go/blob/main/transforms.go
- `PartitionSpec.PartitionType(...)`: https://github.com/apache/iceberg-go/blob/main/partitions.go
- Avro schema conversion: https://github.com/apache/iceberg-go/blob/main/schema_conversions.go
`DayTransform.ResultType(...)` returns `PrimitiveTypes.Int32`. `PartitionSpec.PartitionType(...)` uses the transform result type when constructing the partition struct. The Avro conversion maps `Int32Type` to Avro `IntNode`, while `DateType` maps to Avro `DateNode`.
So current Go writes manifest partition fields for `day(...)` as:
```json
"type": "int"
```
### What Go accepts
| Manifest Avro schema form | Current `iceberg-go` status |
|---|---|
| Plain Avro `int` | Confirmed: this is what it writes and expects |
| Avro `int` with `logicalType: date` | Not clearly guaranteed on merged `main` |
The Go manifest reader opens the Avro OCF file, uses the file schema, and builds logical-type metadata from the actual Avro schema, including nested partition fields. However, I would not call Java-style "accepts both" behavior confirmed without a merged explicit compatibility test/path.
Relevant reader path:
- Manifest code: https://github.com/apache/iceberg-go/blob/main/manifest.go
### Open Go PR changing this behavior
There is an open Go PR proposing to encode `day(...)` partition fields with Avro logical `date` for interoperability with engines that expect the Java-style manifest schema:
- https://github.com/apache/iceberg-go/pull/915
The PR description says current Go encoded all `Int32` partition fields as plain Avro `int`, and that this caused interoperability issues with engines expecting the logical `date` annotation.
As of this writing, that PR is open.
### Go summary
Current Go writes the spec-literal form:
```json
"type": "int"
```
The open PR proposes moving Go toward the Java/PyIceberg/Rust convention:
```json
{
"type": "int",
"logicalType": "date"
}
```
---
## 4. Rust: `apache/iceberg-rust`
### What Rust writes
Current `iceberg-rust` follows the Java/PyIceberg convention.
Relevant code paths:
- `Transform::Day.result_type(...)`: https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/spec/transform.rs
- Avro schema conversion: https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/avro/schema.rs
- Manifest writer: https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/spec/manifest/writer.rs
`Transform::Day.result_type(...)` maps timestamp/date source types to `PrimitiveType::Date`. Rust's Avro schema conversion maps `PrimitiveType::Date` to Avro date, while `PrimitiveType::Int` maps to plain Avro int. The manifest writer builds the manifest Avro schema from the partition type.
So Rust writes manifest partition fields for `day(...)` as:
```json
{
"type": "int",
"logicalType": "date"
}
```
### Why Rust changed
Rust originally mapped `Transform::Day` to `Int`. That caused manifest parsing failures for Java-written manifests whose day partition field used Avro logical `date`.
Related issue and PR:
- Issue: https://github.com/apache/iceberg-rust/issues/478
- PR: https://github.com/apache/iceberg-rust/pull/479
PR #479 changed `Transform::Day` to map partition types to `Date` rather than `Int` for consistency with the Java/reference implementation.
### What Rust accepts
| Manifest Avro schema form | Current `iceberg-rust` status |
|---|---|
| Avro `int` with `logicalType: date` | Confirmed: this is what current Rust writes and expects |
| Plain Avro `int` | Not clearly guaranteed in the normal manifest reader |
The normal Rust manifest reader reconstructs the expected manifest schema from table metadata and the partition spec, then reads with that expected schema. The historical bug report shows that a mismatch between the expected schema and the manifest file schema caused parsing failure. Therefore, Rust should not be assumed to accept plain Avro `int` for `day(...)` partition fields unless an explicit compatibility path or test is added.
Relevant reader path:
- Manifest reader code: https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/spec/manifest/mod.rs
### Rust summary
Rust writes the de facto implementation convention:
```json
{
"type": "int",
"logicalType": "date"
}
```
Unlike Java, Rust should not currently be assumed to accept both schema forms in all manifest reading paths.
---
## Why this matters
Although both schemas encode the same integer day count, the Avro schema annotation can matter for readers that construct an expected manifest schema and use strict Avro schema resolution.
This has led to divergent implementation behavior:
- Java/PyIceberg/Rust write logical `date`.
- Go currently writes plain `int`.
- Java accepts both forms.
- Rust and Go may not accept both forms in all manifest reading paths.
This creates uncertainty for new implementations and for existing implementations trying to decide whether to prioritize literal spec compliance or compatibility with the dominant Java/Spark convention.
---
## Questions for clarification
Could the Iceberg spec clarify the intended canonical Avro schema for manifest partition fields produced by the `day(...)` transform?
Specifically:
1. Should the canonical transform result type for `day(...)` remain `int`, as currently documented?
2. If the result type remains `int`, should manifest partition fields for `day(...)` be written as plain Avro `int`?
```json
"type": "int"
```
3. Or should `day(...)` be considered a special case where the logical result is date-like and writers should use Avro logical `date`?
```json
{
"type": "int",
"logicalType": "date"
}
```
4. Should readers be required or strongly recommended to accept both schema forms for `day(...)` partition fields?
5. If both forms are accepted, should one form be documented as canonical for writers?
6. Should the partition transform result type table be changed from `int` to `date` for `day(...)`, or should the Avro manifest encoding guidance be changed to explicitly allow/use logical `date` for `day(...)` despite the transform table saying `int`?
---
## Possible resolutions
### Option A: Keep the spec-literal interpretation
Clarify that `day(...)` result type is `int` and writers should use plain Avro `int` for manifest partition fields.
Pros:
- Matches the current partition transform table.
- Matches Appendix A's mapping for Iceberg `int`.
- Matches current `iceberg-go` behavior.
Cons:
- Diverges from current Java/PyIceberg/Rust behavior.
- May require existing implementations to change writer behavior or document a compatibility exception.
- Could cause compatibility problems with engines that expect logical `date`.
### Option B: Change or clarify `day(...)` result type to `date`
Clarify that `day(...)` returns a date-like value and should be written as Avro logical `date`.
Pros:
- Matches Java/PyIceberg/Rust behavior.
- Matches the de facto ecosystem convention.
- Provides more human-readable metadata/table output.
Cons:
- Requires changing or qualifying the current transform result type table.
- The transform table currently describes the result type as `int`.
- Could make plain-`int` writers look non-compliant despite following the current spec text.
### Option C: Keep `int` result type but explicitly allow logical `date` in manifests
Clarify that the physical value of `day(...)` is still an integer day count, but manifest Avro schemas may use either:
```json
"type": "int"
```
or:
```json
{
"type": "int",
"logicalType": "date"
}
```
and readers should accept both.
Pros:
- Acknowledges the existing ecosystem split.
- Avoids declaring current Java/PyIceberg/Rust or Go behavior invalid.
- Encourages interoperability.
Cons:
- Leaves two writer encodings unless a canonical writer form is also chosen.
- Requires readers to implement compatibility handling.
---
## Suggested clarification
A possible clarification could be:
> For partition fields produced by the `day` transform, the stored value is an integer number of days from `1970-01-01`. Manifest readers should accept both plain Avro `int` and Avro `int` annotated with `logicalType: date` for these fields. Writers should use [chosen canonical form].
The main clarification needed is the final sentence: which form should writers use?
---
## Related issues and PRs
- Java issue asking about this mismatch: https://github.com/apache/iceberg/issues/10616
- Older Java issue discussing timestamp/day returning date for display: https://github.com/apache/iceberg/issues/279
- Rust issue caused by Java date-logical-type manifests: https://github.com/apache/iceberg-rust/issues/478
- Rust PR changing `Transform::Day` to `Date`: https://github.com/apache/iceberg-rust/pull/479
- Go PR proposing logical `date` for day-transform partition fields: https://github.com/apache/iceberg-go/pull/915
---
## Requested outcome
Please clarify the intended spec behavior for `day(...)` partition fields in manifest Avro schemas:
- Is plain Avro `int` the canonical writer form?
- Is Avro logical `date` the canonical writer form?
- Are both forms allowed?
- Should readers be required or strongly recommended to accept both?
This would help align Java, PyIceberg, Go, Rust, and other implementations around one documented compatibility expectation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spec ambiguity: Avro schema for `day` partition transform fields in manifests [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #16414:
URL: https://github.com/apache/iceberg/issues/16414#issuecomment-4510315302
Linking to previous discussion, https://github.com/apache/iceberg/issues/10616#issuecomment-2200191427
looks like we went back and forth on this a couple of times already 😄
lets figure this out once and for all 🤞
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spec ambiguity: Avro schema for `day` partition transform fields in manifests [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #16414:
URL: https://github.com/apache/iceberg/issues/16414#issuecomment-4696712203
Changed the table spec to document this in #16446
Hope we don't need to re-litigate this in the future 🤞
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spec ambiguity: Avro schema for `day` partition transform fields in manifests [iceberg]
Posted by "zhjwpku (via GitHub)" <gi...@apache.org>.
zhjwpku commented on issue #16414:
URL: https://github.com/apache/iceberg/issues/16414#issuecomment-4639558799
FWIW, iceberg-cpp also changed the result type of DayTransform from `int` to `date`[1]
[1] https://github.com/apache/iceberg-cpp/pull/285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spec ambiguity: Avro schema for `day` partition transform fields in manifests [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu closed issue #16414: Spec ambiguity: Avro schema for `day` partition transform fields in manifests
URL: https://github.com/apache/iceberg/issues/16414
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org