You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "sugarcrm-jgminder (via GitHub)" <gi...@apache.org> on 2025/07/09 21:43:36 UTC

[I] [Bug] Iceberg compatibility does not set the snapshot parent_id field [paimon]

sugarcrm-jgminder opened a new issue, #5860:
URL: https://github.com/apache/paimon/issues/5860

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   
   ### Paimon version
   
   When using Paimon version 1.2.1, if you create a Paimon table using Iceberg compatibility such as here: https://paimon.apache.org/docs/1.1/migration/iceberg-compatibility/ the `parent_id` field of snapshots is not set.
   
   Is there a reason the `parent_id` is not set when Paimon generates the table versus when the Iceberg libraries create the table?
   
   ### Compute Engine
   
   I used Flink 1.20.1 as the compute engine and provided a FlinkSQL example.
   
   ### Minimal reproduce step
   
   ```
   CREATE CATALOG paimon_catalog WITH (
       'type' = 'paimon',
       'warehouse' = 's3://paimon'
   );
   
   CREATE TABLE paimon_catalog.`default`.cities (
       country STRING,
       name STRING
   ) WITH (
       'metadata.iceberg.storage' = 'hadoop-catalog'
   );
   
   INSERT INTO paimon_catalog.`default`.cities VALUES ('usa', 'new york'), ('germany', 'berlin'), ('usa', 'chicago'), ('germany', 'hamburg');
   INSERT INTO iceberg_catalog.`default`.cities VALUES ('usa', 'orlando');
   INSERT INTO iceberg_catalog.`default`.cities VALUES ('usa', 'houston'), ('germany', 'munich');
   
   CREATE CATALOG iceberg_catalog WITH (
       'type' = 'iceberg',
       'catalog-type' = 'hadoop',
        'warehouse' = 's3://paimon/iceberg',
        'cache-enabled' = 'false' -- disable iceberg catalog caching to quickly see the result
   );
   
   CREATE TABLE iceberg_catalog.`default`.cities_iceberg (
       country STRING,
       name STRING
   );
   
   INSERT INTO paimon_catalog.`default`.cities_iceberg VALUES ('usa', 'new york'), ('germany', 'berlin'), ('usa', 'chicago'), ('germany', 'hamburg');
   INSERT INTO iceberg_catalog.`default`.cities_iceberg VALUES ('usa', 'orlando');
   INSERT INTO iceberg_catalog.`default`.cities_iceberg VALUES ('usa', 'houston'), ('germany', 'munich');
   
   SELECT * FROM iceberg_catalog.`default`.cities$snapshots;
   +----------------------------+-------------+-----------+-----------+--------------------------------+---------+
   |               committed_at | snapshot_id | parent_id | operation |                  manifest_list | summary |
   +----------------------------+-------------+-----------+-----------+--------------------------------+---------+
   | 2025-07-09 19:30:25.728000 |           1 |    <NULL> |    append | s3://paimon/iceberg/default... |      {} |
   | 2025-07-09 19:33:27.470000 |           2 |    <NULL> |    append | s3://paimon/iceberg/default... |      {} |
   | 2025-07-09 19:46:32.233000 |           3 |    <NULL> |    append | s3://paimon/iceberg/default... |      {} |
   +----------------------------+-------------+-----------+-----------+--------------------------------+---------+
   
   SELECT * FROM iceberg_catalog.`default`.cities_iceberg$snapshots;
   +----------------------------+---------------------+---------------------+-----------+--------------------------------+--------------------------------+
   |               committed_at |         snapshot_id |           parent_id | operation |                  manifest_list |                        summary |
   +----------------------------+---------------------+---------------------+-----------+--------------------------------+--------------------------------+
   | 2025-07-09 19:48:50.299000 | 8358828322173939955 |              <NULL> |    append | s3://paimon/iceberg/default... | {flink.operator-id=90bea66d... |
   | 2025-07-09 19:49:16.963000 |  123304806091236739 | 8358828322173939955 |    append | s3://paimon/iceberg/default... | {flink.operator-id=90bea66d... |
   | 2025-07-09 19:49:45.906000 | 6003285977049337526 |  123304806091236739 |    append | s3://paimon/iceberg/default... | {flink.operator-id=90bea66d... |
   +----------------------------+---------------------+---------------------+-----------+--------------------------------+--------------------------------+
   ```
   
   Here you can see that the Paimon generated Iceberg table lacks the `parent_id` whereas the one generated with Iceberg libraries has this field populated.
   
   ### What doesn't meet your expectations?
   
   The `parent_id` should be populated when querying the snapshot data, otherwise, if a user wanted to call something like `create_changelog_view` it wouldn't have the necessary data for the procedure call.
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Iceberg compatibility does not set the snapshot parent_id field [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #5860: [Bug] Iceberg compatibility does not set the snapshot parent_id field
URL: https://github.com/apache/paimon/issues/5860


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Iceberg compatibility does not set the snapshot parent_id field [paimon]

Posted by "nickdelnano (via GitHub)" <gi...@apache.org>.
nickdelnano commented on issue #5860:
URL: https://github.com/apache/paimon/issues/5860#issuecomment-3146671286

   @sugarcrm-jgminder can you explain your use case in more detail? The intended usage here would be to use Paimon to derive changelogs (not Iceberg metadata of Paimon table).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Iceberg compatibility does not set the snapshot parent_id field [paimon]

Posted by "brunsgaard (via GitHub)" <gi...@apache.org>.
brunsgaard commented on issue #5860:
URL: https://github.com/apache/paimon/issues/5860#issuecomment-3877580956

   https://github.com/apache/paimon/pull/6354 seems to resolve the issue, the fix is currently in main and not released as of v1.3.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org