You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "wForget (via GitHub)" <gi...@apache.org> on 2024/04/15 07:12:02 UTC
[I] [Feature] Paimon Spark Extensions conflict with Iceberg [paimon]
wForget opened a new issue, #3212:
URL: https://github.com/apache/paimon/issues/3212
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
The `Call` syntax is defined in both Iceberg and Paimon, which may cause conflicts when I introduce their SparkSessionExtensions at the same time.
Reproduce:
```
spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
-- create iceberg table
CREATE TABLE iceberg_catalog.sample.iceberg_t1 (
user_id BIGINT,
item_id BIGINT,
behavior STRING,
dt STRING,
hh STRING
) using iceberg;
-- create paimon table
CREATE TABLE paimon_catalog.sample.paimon_t1 (
user_id BIGINT,
item_id BIGINT,
behavior STRING,
dt STRING,
hh STRING
) TBLPROPERTIES (
'primary-key' = 'dt,hh,user_id'
);
-- Successed
CALL iceberg_catalog.system.remove_orphan_files(table => "sample.iceberg_t1");
-- Failed, use iceberg ResolveProcedures
CALL paimon_catalog.sys.remove_orphan_files(table => "sample.paimon_t1");
```
duplicate: https://github.com/apache/iceberg/issues/10143
### Solution
If `currentCatalog` is not Spark Session catalog and Paimon Spark Catalog, we first use `delegate` catalog to parse sqlText in `PaimonSparkSqlExtensionsParser#parsePlan`.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Paimon Spark Extensions conflict with Iceberg [paimon]
Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on issue #3212:
URL: https://github.com/apache/paimon/issues/3212#issuecomment-2146795461
> I determine which spark extensions to use in the front-end sql gateway service.
@wForget So, you didn't combine these two extension functionalities, but used them separately, and routed the queries between them, is my understanding correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Paimon Spark Extensions conflict with Iceberg [paimon]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #3212:
URL: https://github.com/apache/paimon/issues/3212#issuecomment-2146679044
> Hey @wForget, I came across the same issue, have you got this fixed?
I determine which spark extensions to use in the front-end sql gateway service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Paimon Spark Extensions conflict with Iceberg [paimon]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget closed issue #3212: [Feature] Paimon Spark Extensions conflict with Iceberg
URL: https://github.com/apache/paimon/issues/3212
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Paimon Spark Extensions conflict with Iceberg [paimon]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #3212:
URL: https://github.com/apache/paimon/issues/3212#issuecomment-2146811019
> > I determine which spark extensions to use in the front-end sql gateway service.
>
> @wForget So, you didn't combine these two extension functionalities, but used them separately, and routed the queries between them, is my understanding correct?
Yes, we currently have no need to mix them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Paimon Spark Extensions conflict with Iceberg [paimon]
Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on issue #3212:
URL: https://github.com/apache/paimon/issues/3212#issuecomment-2146580323
Hey @wForget, I came across the same issue, have you got this fixed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Paimon Spark Extensions conflict with Iceberg [paimon]
Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on issue #3212:
URL: https://github.com/apache/paimon/issues/3212#issuecomment-2146816053
@wForget Thanks for replying
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org