[I] [Feature] Support custom table properties when syncing to Iceberg REST catalog [paimon]

["JTaky (via GitHub)" <gi...@apache.org>]

JTaky opened a new issue, #9226:
URL: https://github.com/apache/paimon/issues/9226

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   
   ### Motivation
   
   When Paimon syncs a table to an Iceberg REST catalog (metadata.iceberg.storage = rest-catalog), the only properties ever written to the Iceberg table are the two internal retention knobs (write.metadata.previous-versions-max, write.metadata.delete-after-commit.enabled) set in IcebergRestMetadataCommitter. Any other table option a user sets is silently dropped and never reaches the Iceberg table's properties.
   
   This matters for us because we have downstream tooling and conventions (table categorization tags, ownership/color-coding metadata, etc.) that rely on custom Iceberg table properties, and today there's no way to set them on a Paimon-produced table without hand-editing the table after the fact outside Paimon.
   
   Paimon already supports this pattern for its own native catalogs — HiveCatalog forwards table options into Hive/JDBC metastore table parameters by default (CatalogOptions.SYNC_ALL_PROPERTIES, defaulting to true). The Iceberg REST path just never got the equivalent treatment.
   
   ### Solution
   
   Add an explicit opt-in prefix, e.g. _metadata.iceberg.table-properties.<key> → <key>_, mirroring the existing metadata.iceberg.rest.* (IcebergOptions.REST_CONFIG_PREFIX) pattern already used to configure the REST catalog client. On commit, strip the prefix and merge the resulting keys into the Iceberg table's properties, alongside the two existing hardcoded ones.
   
   Two things worth getting right:
   - Filter out Iceberg's reserved property keys (TableProperties.RESERVED_PROPERTIES) before merging, since Iceberg throws if they're present.
   - Apply the properties on both the create path and the drop-and-recreate path (the REST committer recreates the table on metadata-base mismatch), not just the update-in-place path — otherwise custom properties would silently disappear whenever that happens.
   
   An explicit opt-in prefix (rather than forwarding all Paimon table options) avoids leaking Paimon-internal config (bucket, merge-engine, storage credentials, etc.).
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Support custom table properties when syncing to Iceberg REST catalog [paimon]

["JTaky (via GitHub)" <gi...@apache.org>]

JTaky closed issue #9226: [Feature] Support custom table properties when syncing to Iceberg REST catalog
URL: https://github.com/apache/paimon/issues/9226


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Support custom table properties when syncing to Iceberg REST catalog [paimon]

["JTaky (via GitHub)" <gi...@apache.org>]

JTaky commented on issue #9226:
URL: https://github.com/apache/paimon/issues/9226#issuecomment-5353822169

   was done as append only properties in https://github.com/apache/paimon/pull/9268


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org