You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "plusplusjiajia (via GitHub)" <gi...@apache.org> on 2026/07/22 05:42:47 UTC
[PR] [iceberg] Support the v3 variant type with row lineage and REST catalog [paimon]
plusplusjiajia opened a new pull request, #8795:
URL: https://github.com/apache/paimon/pull/8795
### Purpose
Add Iceberg v3 `variant` type support to Paimon's Iceberg compatibility layer, with the row lineage and REST catalog upgrade it requires. The three are interdependent: v3 `variant` metadata without row lineage is spec-invalid (Iceberg 1.10+ won't parse metadata lacking `next-row-id`/`first-row-id`), and row lineage without the REST upgrade still leaves `rest-catalog` storage creating v2 tables that reject v3 features.
- **Variant type**: `IcebergDataField` maps Paimon `VARIANT` ↔ Iceberg `"variant"` both ways. Before a snapshot is published, `IcebergCommitCallback` rejects types that cannot produce readable metadata: `variant` when `format-version < 3` (and as a partition key, unconditionally), and nanosecond timestamps on every version (Paimon writes them as Parquet `INT96`, which Iceberg cannot read as `timestamp_ns`). Validation runs before publish and before a tag is written.
- **Row lineage**: `IcebergManifestFileMeta`/`IcebergDataFileMeta` gain `first_row_id`; ranges are assigned when writing the manifest list, mirroring Iceberg's `ManifestListWriter`. Tables upgraded from v2 get their surviving manifests assigned on the first v3 commit; regeneration continues past the highest published id so ids are never reused; rewritten/merged manifests keep unchanged rows' `_row_id`.
- **REST upgrade**: bump to Iceberg 1.10.1 (1.8.1 can't parse `variant`). Creation passes the configured `format-version`; a populated table whose version differs is dropped and recreated. Fallout: Hadoop 3.3.6 for `FileSystem.openFile`, jetty/commons-lang3 test-classpath alignment, `RemoveSnapshot` → `RemoveSnapshots`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [iceberg] Support the variant type in Iceberg schema type conversion [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8795:
URL: https://github.com/apache/paimon/pull/8795
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [iceberg] Support the variant type in Iceberg schema type conversion [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8795:
URL: https://github.com/apache/paimon/pull/8795#issuecomment-5082629635
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [iceberg] Support the variant type in Iceberg schema type conversion [paimon]
Posted by "vbabenkoru (via GitHub)" <gi...@apache.org>.
vbabenkoru commented on PR #8795:
URL: https://github.com/apache/paimon/pull/8795#issuecomment-5174470742
@plusplusjiajia Why was row lineage removed from the original PR? I'm hitting the issue of the Iceberg metadata not being really readable by external readers because it doesn't implement row lineage which is required by v3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [iceberg] Support the variant type in Iceberg schema type conversion [paimon]
Posted by "plusplusjiajia (via GitHub)" <gi...@apache.org>.
plusplusjiajia commented on PR #8795:
URL: https://github.com/apache/paimon/pull/8795#issuecomment-5174653028
@vbabenkoru Split out, not dropped — I moved the type-conversion part into its own PR so it could land first. What's merged doesn't change row-lineage behaviour either way; the gap predates it:`metadata.iceberg.format-version` accepts `3`, but `IcebergCommitCallback` never writes `first-row-id` / `next-row-id`, so Paimon's v3 metadata is incomplete for external readers.
I have that working from the original PR. What kept it out is concurrency: the Iceberg metadata is generated in a post-commit callback outside the snapshot lock, so two writers can regenerate from a stale watermark and assign overlapping row ids. Didn't want to land it half-done.
I can open a follow-up PR for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org