You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "123digits (via GitHub)" <gi...@apache.org> on 2026/04/24 14:10:04 UTC
[I] fix: Manifest partition-spec serialization/deserialization is not interoperable with iceberg-java / iceberg-cpp / pyiceberg [iceberg-rust]
123digits opened a new issue, #2365:
URL: https://github.com/apache/iceberg-rust/issues/2365
### Apache Iceberg Rust version
None
### Describe the bug
# Manifest partition-spec serialization/deserialization is not interoperable with iceberg-java / iceberg-cpp / pyiceberg
## Problem
`ManifestWriter` in iceberg-rust writes the `partition-spec` entry of the Avro manifest user metadata as a bare JSON array of fields, and `ManifestMetadata::try_from_avro_bytes` only accepts that same bare-array shape. Every other Iceberg implementation (`iceberg-java`, `iceberg-cpp`, `pyiceberg`) writes (and requires) the spec-compliant serialized `PartitionSpec` object: `{"spec-id": N, "fields": [...]}`.
Consequences:
- **iceberg-rust fails on commit** when the table has any snapshot committed by a non-rust writer. During `fast_append`, iceberg-rust loads the parent snapshot's manifests for the duplicate-file check (`transaction/snapshot.rs`), and deserialization blows up with:
```
DataInvalid => Fail to parse partition spec in manifest metadata,
source: invalid type: map, expected a sequence at line 1 column 0
```
- **Other Iceberg engines fail to read iceberg-rust manifests**, because they expect the `{"spec-id":N,"fields":[...]}` object.
Net result: iceberg-rust can't share a table with any other implementation in either direction.
## Reproduce
1. Partition a table with any partition spec (using iceberg-java, iceberg-cpp, or pyiceberg).
2. Commit at least one snapshot via that non-rust writer.
3. Attempt a `fast_append` commit from iceberg-rust on the same table.
Error surfaces from `crates/iceberg/src/spec/manifest/metadata.rs:83` via `serde_json::from_slice::<Vec<PartitionField>>(bs)`.
Observed on `0.8.0`, `0.9.0`, and `main` (commit `4b0b3525`) as of 2026-04-24.
## Expected behavior
iceberg-rust should:
1. Write `partition-spec` as the spec-compliant object form so other implementations can read our manifests.
2. Accept both shapes on read — the spec-compliant object AND the historical bare-array form — so existing rust-written manifests continue to deserialize.
## Proposed fix
Two small changes in `crates/iceberg/src/spec/manifest/`:
**`writer.rs`** — serialize the full `PartitionSpec`:
```rust
to_vec(&self.metadata.partition_spec).map_err(...) // was: to_vec(&...partition_spec.fields())
```
**`metadata.rs`** — accept either shape on read, preferring the bare-array path for backwards compatibility:
```rust
serde_json::from_slice::<Vec<PartitionField>>(bs)
.or_else(|_| {
#[derive(serde::Deserialize)]
struct PartitionSpecJson { fields: Vec<PartitionField> }
serde_json::from_slice::<PartitionSpecJson>(bs).map(|s| s.fields)
})
.map_err(...)
```
Existing 17 manifest unit tests (including `writer::tests::test_add_delete_existing` and `test_v3_delete_manifest_delete_file_roundtrip`) pass unchanged with both patches applied. Diff total: 29 insertions, 8 deletions across the two files.
I have a local patch working against a table with mixed rust/cpp writers. Working on putting up a PR for this shortly.
### To Reproduce
_No response_
### Expected behavior
_No response_
### Willingness to contribute
I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] fix: Manifest partition-spec serialization/deserialization is not interoperable with iceberg-java / iceberg-cpp / pyiceberg [iceberg-rust]
Posted by "123digits (via GitHub)" <gi...@apache.org>.
123digits commented on issue #2365:
URL: https://github.com/apache/iceberg-rust/issues/2365#issuecomment-4313860983
Created PR: https://github.com/apache/iceberg-rust/pull/2366
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] fix: Manifest partition-spec serialization/deserialization is not interoperable with iceberg-java / iceberg-cpp / pyiceberg [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2365:
URL: https://github.com/apache/iceberg-rust/issues/2365#issuecomment-6329584325
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org