You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "wForget (via GitHub)" <gi...@apache.org> on 2024/04/15 07:06:08 UTC

[I] Iceberg Spark Extensions conflict with Paimon [iceberg]

wForget opened a new issue, #10143:
URL: https://github.com/apache/iceberg/issues/10143

   ### Feature Request / Improvement
   
   The `Call` syntax is defined in both Iceberg and Paimon, which may cause conflicts when I introduce their SparkSessionExtensions at the same time.
   
   Reproduce:
   
   ```
   spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
   
   -- create iceberg table
   CREATE TABLE iceberg_catalog.sample.iceberg_t1 (
       user_id BIGINT,
       item_id BIGINT,
       behavior STRING,
       dt STRING,
       hh STRING
   ) using iceberg;
   
   -- create paimon table
   CREATE TABLE paimon_catalog.sample.paimon_t1 (
       user_id BIGINT,
       item_id BIGINT,
       behavior STRING,
       dt STRING,
       hh STRING
   ) TBLPROPERTIES (
       'primary-key' = 'dt,hh,user_id'
   );
   
   -- Successed
   CALL iceberg_catalog.system.remove_orphan_files(table => "sample.iceberg_t1");
   
   -- Failed, use iceberg ResolveProcedures
   CALL paimon_catalog.sys.remove_orphan_files(table => "sample.paimon_t1");
   ```
   
   One idea:
   If `currentCatalog` is not Spark Session catalog and Iceberg Spark Catalog, we first use `delegate`  catalog to parse sqlText in `IcebergSparkSqlExtensionsParser#parsePlan`.
   
   ### Query engine
   
   Spark


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Iceberg Spark Extensions conflict with Paimon [iceberg]

Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #10143:
URL: https://github.com/apache/iceberg/issues/10143#issuecomment-2060201859

   > Never mind. I think what you mean it can be fixed either by Iceberg or by paimon?
   
   Yes, I'm not sure if my idea is acceptable, so I submitted issues on both sides hoping to get more suggestions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Iceberg Spark Extensions conflict with Paimon [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on issue #10143:
URL: https://github.com/apache/iceberg/issues/10143#issuecomment-2058746453

   can you close this if it is a duplicate of https://github.com/apache/paimon/issues/3212 ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org