You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "fm100 (via GitHub)" <gi...@apache.org> on 2026/07/09 19:27:50 UTC
[PR] fix(spec): tolerate DuckDB manifest entry schemas [iceberg-rust]
fm100 opened a new pull request, #2791:
URL: https://github.com/apache/iceberg-rust/pull/2791
## Which issue does this PR close?
- Closes #.
## What changes are included in this PR?
This PR makes manifest metadata parsing more tolerant of manifests written by affected versions of the DuckDB Iceberg extension.
According to the Iceberg spec, the Avro manifest metadata `schema` key should contain the table schema at the time the manifest was written. A DuckDB Iceberg extension bug wrote the manifest entry schema into that key instead of the table schema. DuckDB fixed this in duckdb/duckdb-iceberg#1020, but that fix has not been released yet.
Even after a fixed DuckDB Iceberg extension release is available, tables created by the buggy versions can still contain manifests with the bad metadata. Reopening those tables with a newer writer does not automatically rewrite existing manifest files, so readers need to tolerate the malformed metadata to read those existing tables.
For unpartitioned manifests, Iceberg Rust can still read the manifest entries without the table schema. This PR falls back to an empty schema when:
- the manifest metadata `schema` value cannot be parsed as an Iceberg table schema
- the manifest partition spec is empty
The fallback remains limited to unpartitioned manifests because partitioned manifests need the table schema to bind partition source fields.
## Are these changes tested?
Yes. This PR adds a unit test covering manifest metadata with a DuckDB-style manifest entry schema for an unpartitioned table.
Tested with:
```bash
cargo test -p iceberg test_manifest_metadata_with_manifest_entry_schema_for_unpartitioned_table
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(spec): tolerate DuckDB manifest entry schemas [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #2791:
URL: https://github.com/apache/iceberg-rust/pull/2791#issuecomment-4929016896
In general, I think its a good idea for the parsing logic to be more tolerable, it allows us to fix forward malformed metadata. We should still enforce spec conformance somewhere in the stack π
i'll take a look at the PR when i get some time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(spec): tolerate DuckDB manifest entry schemas [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #2791: fix(spec): tolerate DuckDB manifest entry schemas
URL: https://github.com/apache/iceberg-rust/pull/2791
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(spec): tolerate DuckDB manifest entry schemas [iceberg-rust]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #2791:
URL: https://github.com/apache/iceberg-rust/pull/2791#discussion_r3959287129
##########
crates/iceberg/src/spec/manifest/metadata.rs:
##########
@@ -72,22 +57,39 @@ impl ManifestMetadata {
})
.transpose()?
.unwrap_or(0);
+ let raw_schema = meta.get("schema").ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ "schema is required in manifest metadata but not found",
+ )
+ })?;
+ let partition_fields = {
+ let bs = meta.get("partition-spec").ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ "partition-spec is required in manifest metadata but not found",
+ )
+ })?;
+ serde_json::from_slice::<Vec<PartitionField>>(bs).map_err(|err| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ "Fail to parse partition spec in manifest metadata",
+ )
+ .with_source(err)
+ })?
+ };
+ let schema = Arc::new(parse_manifest_table_schema(raw_schema).or_else(|err| {
Review Comment:
I think the fallback quietly misses its own target case. It only fires when `parse_manifest_table_schema` returns Err, which here only happens because DuckDB's payload carries the Avro `array`+`items` shape that `SerdeType` can't parse.
A DuckDB manifest-entry schema with only scalar/struct fields β `{status: int, snapshot_id: long, data_file: struct{...}}` β parses cleanly as a valid Iceberg schema, so the `or_else` never runs and we silently keep the wrong schema instead of blanking it. That's worse than the fallback. Fingerprinting the shape before parsing (see the sibling comment) closes this one too β wdyt?
##########
crates/iceberg/src/spec/manifest/metadata.rs:
##########
@@ -72,22 +57,39 @@ impl ManifestMetadata {
})
.transpose()?
.unwrap_or(0);
+ let raw_schema = meta.get("schema").ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ "schema is required in manifest metadata but not found",
+ )
+ })?;
+ let partition_fields = {
Review Comment:
minor one, but moving the partition-spec parse ahead of the schema parse changes which error surfaces. A manifest with both an unparseable `schema` and a missing `partition-spec` used to report "Fail to parse schema"; now it reports "partition-spec is required" first and the schema error is never reached. If anything downstream matches on these strings it'll point at the wrong root cause β I'd keep the schema parse first, or leave a note that the reorder is intentional.
##########
crates/iceberg/src/spec/manifest/metadata.rs:
##########
@@ -72,22 +57,39 @@ impl ManifestMetadata {
})
.transpose()?
.unwrap_or(0);
+ let raw_schema = meta.get("schema").ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ "schema is required in manifest metadata but not found",
+ )
+ })?;
+ let partition_fields = {
+ let bs = meta.get("partition-spec").ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ "partition-spec is required in manifest metadata but not found",
+ )
+ })?;
+ serde_json::from_slice::<Vec<PartitionField>>(bs).map_err(|err| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ "Fail to parse partition spec in manifest metadata",
+ )
+ .with_source(err)
+ })?
+ };
+ let schema = Arc::new(parse_manifest_table_schema(raw_schema).or_else(|err| {
+ if partition_fields.is_empty() {
+ // Some writers store the manifest-entry schema under the
+ // `schema` metadata key. For unpartitioned manifests we can
+ // still plan file scans; column bounds that require the table
+ // schema are ignored when their field ids cannot be resolved.
+ Schema::builder().with_schema_id(schema_id).build()
Review Comment:
Even once this is fingerprinted, I'd `tracing::warn!` with the original `err` before falling back β the module already pulls in `tracing`. Right now `err` is dropped, so a manifest silently degrades to no column stats with zero signal in the logs until someone notices pruning stopped working.
##########
crates/iceberg/src/spec/manifest/metadata.rs:
##########
@@ -72,22 +57,39 @@ impl ManifestMetadata {
})
.transpose()?
.unwrap_or(0);
+ let raw_schema = meta.get("schema").ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ "schema is required in manifest metadata but not found",
+ )
+ })?;
+ let partition_fields = {
+ let bs = meta.get("partition-spec").ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ "partition-spec is required in manifest metadata but not found",
+ )
+ })?;
+ serde_json::from_slice::<Vec<PartitionField>>(bs).map_err(|err| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ "Fail to parse partition spec in manifest metadata",
+ )
+ .with_source(err)
+ })?
+ };
+ let schema = Arc::new(parse_manifest_table_schema(raw_schema).or_else(|err| {
+ if partition_fields.is_empty() {
Review Comment:
The guard here is just `partition_fields.is_empty()`, which is true for every unpartitioned manifest β not just DuckDB's. So a genuinely corrupt or truncated `schema`, or one from some other buggy writer, now silently resolves to an empty schema instead of a `DataInvalid`, and every column bound on that manifest gets dropped downstream.
I'd gate the fallback on a positive fingerprint of the manifest-entry shape β parse the raw bytes as a `Value`, confirm the root struct has `status`/`snapshot_id`/`data_file`, and only then substitute; otherwise re-raise `err`. That keeps corruption loud and scopes the tolerance to the actual bug.
##########
crates/iceberg/src/spec/manifest/mod.rs:
##########
@@ -169,6 +169,62 @@ mod tests {
use crate::io::FileIO;
use crate::spec::{Literal, NestedField, PrimitiveType, Struct, Transform, Type};
+ #[test]
+ fn test_manifest_metadata_with_manifest_entry_schema_for_unpartitioned_table() {
+ let mut meta = HashMap::new();
+ meta.insert("schema-id".to_string(), b"0".to_vec());
+ meta.insert("partition-spec".to_string(), b"[]".to_vec());
+ meta.insert("partition-spec-id".to_string(), b"0".to_vec());
+ meta.insert("format-version".to_string(), b"2".to_vec());
+ meta.insert("content".to_string(), b"data".to_vec());
+ meta.insert(
+ "schema".to_string(),
+ br#"{
+ "type": "struct",
+ "schema-id": 0,
+ "fields": [
+ {"id": 0, "name": "status", "required": true, "type": "int"},
+ {"id": 1, "name": "snapshot_id", "required": false, "type": "long"},
+ {
+ "id": 2,
+ "name": "data_file",
+ "required": true,
+ "type": {
+ "type": "struct",
+ "fields": [
+ {"id": 100, "name": "file_path", "required": true, "type": "string"},
+ {
+ "id": 125,
+ "name": "lower_bounds",
+ "required": false,
+ "type": {
+ "type": "array",
+ "items": {
+ "type": "record",
+ "name": "k126_k127",
+ "fields": [
+ {"name": "key", "type": "int", "id": 126},
+ {"name": "value", "type": "binary", "id": 127}
+ ]
+ }
+ }
+ }
+ ]
+ }
+ }
+ ]
+ }"#
+ .to_vec(),
+ );
+
+ let metadata = ManifestMetadata::parse(&meta).unwrap();
+ assert_eq!(metadata.schema_id(), 0);
+ assert!(metadata.schema().as_struct().fields().is_empty());
Review Comment:
This assertion passes whether the fallback fired or the schema simply parsed into an empty struct β it doesn't actually prove we took the fallback path. I'd assert against the helper directly (that this payload returns `Err` from `parse_manifest_table_schema`), or add a contrasting case, so the test pins the behavior it's meant to.
##########
crates/iceberg/src/spec/manifest/mod.rs:
##########
@@ -169,6 +169,62 @@ mod tests {
use crate::io::FileIO;
use crate::spec::{Literal, NestedField, PrimitiveType, Struct, Transform, Type};
+ #[test]
+ fn test_manifest_metadata_with_manifest_entry_schema_for_unpartitioned_table() {
Review Comment:
Two gaps I'd want covered before this lands. There's no negative test β a partitioned manifest carrying this same payload should hit the `else` branch and return `.is_err()`; as it stands the `partition_fields.is_empty()` guard could be inverted or deleted and nothing here would fail.
And there's no regression test that a normal, valid table schema still parses into non-empty fields β if the fallback ever fired unconditionally, no test would catch it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(spec): tolerate DuckDB manifest entry schemas [iceberg-rust]
Posted by "fm100 (via GitHub)" <gi...@apache.org>.
fm100 commented on PR #2791:
URL: https://github.com/apache/iceberg-rust/pull/2791#issuecomment-5516457931
Leaving comment for preventing it from being closed π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(spec): tolerate DuckDB manifest entry schemas [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2791:
URL: https://github.com/apache/iceberg-rust/pull/2791#issuecomment-6227273146
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(spec): tolerate DuckDB manifest entry schemas [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2791:
URL: https://github.com/apache/iceberg-rust/pull/2791#issuecomment-6071749553
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatβs incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(spec): tolerate DuckDB manifest entry schemas [iceberg-rust]
Posted by "fm100 (via GitHub)" <gi...@apache.org>.
fm100 commented on PR #2791:
URL: https://github.com/apache/iceberg-rust/pull/2791#issuecomment-5074950296
@kevinjqliu sorry to bother you, but may I ask your attention please? π Not sure what's the release cadance but hope this to be merged to the next version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(spec): tolerate DuckDB manifest entry schemas [iceberg-rust]
Posted by "fm100 (via GitHub)" <gi...@apache.org>.
fm100 commented on PR #2791:
URL: https://github.com/apache/iceberg-rust/pull/2791#issuecomment-5107854824
> Thanks for opening this PR, @fm100.
>
> I'm not super fond of this change - I am concerned that it is introducing complexity and uncertainty into the spec module of iceberg-rust, for a specific edge case. With this change, we add (albeit relatively small) edge cases and blank out the schema _only_ for unpartitioned tables where DuckDB wrote the manifests erroneously before it was fixed (release TBD).
>
> I'd rather entertain discussions on how either DuckDB or the Iceberg project can facilitate operations that recover the table in this scenario. For instance, why does the DuckDB project not own providing a procedure to rewrite these manifests tolerating bad schema bytes in the manifests' metadata? I'm aware that I'm pushing work back on to the DuckDB community, although I do question if it is a better fit where they can own the bug in terms of both the report, fix, and the mitigation for manifests written with this malformed metadata. They can also decide when to no longer maintain code required to address the bug.
>
> What do you think, @fm100 @kevinjqliu? Happy to be pushed back on. I know there's other circumstances we'd rather tolerate smaller inconsistencies (such as the Decimal(S, P) serialization), so there's a balance to be found here.
Thank you very much for the opinion. I think DuckDB might be able to make a fix on their side, particularly to rewrite metadata that is broken, but I still prefer the reader side tolerance regardless of that. As iceberg is an open table format, users have a freedom of choice of the engine. It is highly likely possible that two or more users using different version of duckdb from their terminal, and I don't think it's ideal to expect them to be aware of this issue and rewrite when things broken. The owner of the table won't see an issue because they read and write with duckdb, but the problem will happen for someone else using DataFusion backed by iceberg-rust. From my perspective, the top four popular query engines are Spark, DuckDB, polars, and DataFusion, and the Spark and DuckDB are already tolerant for this somehow (I haven't tested polars). It will give the users impression that simply iceberg-datafusion integration is unstable rather than having the fix with DuckDB. So, Duck
DB community making a fix is one way to push of course, but being tolerant from the iceberg-rust side also super important.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(spec): tolerate DuckDB manifest entry schemas [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2791:
URL: https://github.com/apache/iceberg-rust/pull/2791#issuecomment-5459135324
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatβs incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org