You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "HaoXuAI (via GitHub)" <gi...@apache.org> on 2026/08/24 20:19:36 UTC
[I] equality_ids written as list, but spec and all other implementations use list [iceberg-python]
HaoXuAI opened a new issue, #3840:
URL: https://github.com/apache/iceberg-python/issues/3840
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
`equality_ids` is declared as a list of **long** in the manifest-entry Avro schema, but the Iceberg spec, the Java reference implementation, and iceberg-cpp all specify **int**. pyiceberg is the only implementation that disagrees, so manifests it writes are rejected outright by spec-conformant readers.
## Evidence
| source | element type of field 136 |
|---|---|
| Iceberg spec, `format/spec.md` ("Manifests" table) | `list<136: int>` |
| Java, `api/src/main/java/org/apache/iceberg/DataFile.java` | `ListType.ofRequired(136, IntegerType.get())` |
| iceberg-cpp, `src/iceberg/manifest_entry.h` | `iceberg::int32()` |
| **pyiceberg, `pyiceberg/manifest.py:298` and `:393`** | **`LongType()`** |
Both the v1 and v2 manifest-entry schemas are affected. Note that pyiceberg's own type hint already says `int`:
```python
# pyiceberg/manifest.py:528
def equality_ids(self) -> list[int] | None:
```
So only the Avro schema disagrees; the Python side already treats these as ints.
## Impact
A spec-conformant reader cannot read a pyiceberg-written manifest at all. iceberg-cpp fails with:
```
Cannot read Iceberg type: int from Avro type: "long"
```
because its `kInt` case accepts only `AVRO_INT` (Avro permits `int` → `long` promotion, not the reverse). Any Java or C++ consumer of pyiceberg-written metadata hits the same wall.
This is not limited to equality-delete tables: the schema is written into every manifest, so an ordinary append-only table with no delete files is equally unreadable.
## Reproducer
```python
import fastavro
# any table written by pyiceberg
with open("<some>-m0.avro", "rb") as f:
schema = fastavro.reader(f).writer_schema
# -> equality_ids: ["null", {"element-id": 136, "type": "array", "items": "long"}]
# spec says items should be "int"
```
## The obvious fix breaks backwards compatibility
Changing `LongType()` → `IntegerType()` in those two places makes pyiceberg unable to read **every manifest it has previously written**:
```
ResolveError: Cannot promote long to int
```
Two reasons compound here:
1. `pyiceberg/schema.py` registers `promote` for `IntegerType` (int → long is legal) but has no `LongType` registration, so long → int falls through to the generic case and raises (`schema.py:1660`).
2. `MANIFEST_ENTRY_SCHEMAS` is used for **both** directions — the writer takes `file_schema` from it and the reader takes `record_schema` from it — so changing the constant changes the read path too.
Verified empirically: patching field 135's element type to `IntegerType` and then reading a manifest written by the current release raises the above.
## Possible directions
A correct fix has to write `int` while continuing to read `long`. Options, roughly:
- **Split the write and read schemas** so the writer emits the spec type while the reader keeps accepting the historical one. Probably the cleanest, and it changes no promotion rules.
- **Allow long → int narrowing** for this field specifically. Smaller diff, but it widens the resolver in a direction the Avro and Iceberg promotion rules deliberately exclude.
Deliberately not proposing a patch — which trade-off is acceptable seems like a maintainer call, and there may be a preferred pattern for schema-compatibility shims that I am not aware of.
## Environment
- pyiceberg 0.11.1 (current release) and `main`, both affected
- Cross-checked against the Iceberg spec, the Java reference implementation, and iceberg-cpp
- Found while making iceberg-cpp read a pyiceberg-written table
## Status checks
**Still present on `main`.** Verified against `main` on 2026-08-24: `manifest.py:298` and `:393` both still declare `LongType()`. Separately, no commit in the repository's history mentions `equality_ids` at all, and the recent commits touching `manifest.py` are unrelated (manifest cache configuration, min sequence number, `ManifestEntry.snapshot_id` setter, lint rules).
**No prior report found**, though this is the weaker of the two claims. Searching this repository for `equality_ids` returns 13 issues and PRs, all of which mention it incidentally (equality-delete tests, manifest entry status, the files metadata table) rather than discussing its declared type. Searches for `"Cannot promote long to int"` and `"Cannot read Iceberg type"` return nothing. It is entirely possible this has been raised somewhere I did not look — the dev mailing list or Slack, or phrased in a way my queries missed.
**Related but distinct:** #3744 (*Metadata inspection APIs fail with `struct.error` after int→long / float→double type promotion*) also concerns int/long handling, but it is about promotion of **user table columns** breaking `inspect.*` on pre-promotion data files. It does not involve the manifest-entry schema or `equality_ids`, so the two should not be conflated.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] equality_ids written as list, but spec and all other implementations use list [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #3840:
URL: https://github.com/apache/iceberg-python/issues/3840#issuecomment-5418352174
Thanks for reporting the issue! This is definitely a serious bug. Lets see how we can best fix forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] equality_ids written as list, but spec and all other implementations use list [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on issue #3840:
URL: https://github.com/apache/iceberg-python/issues/3840#issuecomment-5544046227
Yeah, I think that'll be good to avoid writing these in the future. I think we should still give a deprecation warning on manifests with longs for equality_ids and eventually deprecate the functionality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] equality_ids written as list, but spec and all other implementations use list [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu closed issue #3840: equality_ids written as list<long>, but spec and all other implementations use list<int>
URL: https://github.com/apache/iceberg-python/issues/3840
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] equality_ids written as list, but spec and all other implementations use list [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #3840:
URL: https://github.com/apache/iceberg-python/issues/3840#issuecomment-5641343113
last step is to implement the `rewrite_manifest` procedure, i'll take a look at #3631
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] equality_ids written as list, but spec and all other implementations use list [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #3840:
URL: https://github.com/apache/iceberg-python/issues/3840#issuecomment-5530047313
Thanks @haoxu0 for reporting this, and it is a serious issue indeed. I think the approach that @kevinjqliu is suggesting is the right direction. When we coerce int into long, we should check for overflows (and fail).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] equality_ids written as list, but spec and all other implementations use list [iceberg-python]
Posted by "haoxu0 (via GitHub)" <gi...@apache.org>.
haoxu0 commented on issue #3840:
URL: https://github.com/apache/iceberg-python/issues/3840#issuecomment-5530172127
@kevinjqliu @Fokko Thanks. Happy to help with the fix — just let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] equality_ids written as list, but spec and all other implementations use list [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #3840:
URL: https://github.com/apache/iceberg-python/issues/3840#issuecomment-5544739474
I was back and forth about whether to emit deprecation warning. I think the emitting a warning might be too noisy since all existing avro manifests from pyiceberg will trigger it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] equality_ids written as list, but spec and all other implementations use list [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #3840:
URL: https://github.com/apache/iceberg-python/issues/3840#issuecomment-5544743500
btw we're iterating on a fix in #3842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] equality_ids written as list, but spec and all other implementations use list [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #3840:
URL: https://github.com/apache/iceberg-python/issues/3840#issuecomment-5481182323
I took a look at this in depth. We are in fact using `long` type for `equality_ids` which contradicts the spec. The good thing is, we dont write `equality_ids` yet, so only the schema is wrong. And it looks like the java implementation can tolerate this difference in schema (which is why we didnt catch it in the first place). Cpp's validation is stricter so this was caught.
I think the best way to fix forward is
1. change `equality_ids` to `int` so we stop producing wrong schema
2. allow pyiceberg to tolerate `equality_ids` with `long`. this will allow us to read back avro files already written
3. implement `rewrite_manifest` so that users can correct the avro files. this will write the correct schema and allow Cpp reader to work
I considered adding this quirk to the spec's implementation note, but i think its best if we just handle this internally in pyiceberg and not burden every implementation with this caveat.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org