You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "salahdjb (via GitHub)" <gi...@apache.org> on 2024/06/17 16:44:04 UTC
[I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]
salahdjb opened a new issue, #10524:
URL: https://github.com/apache/iceberg/issues/10524
### Query engine
- Spark 3.5.0
- Iceberg 1.5.2
```
iceberg-spark-runtime-3.5_2.12-1.5.2.jar
iceberg-aws-bundle-1.5.2.jar
```
### Question
I successfully opened a spark sql shell from which I can query iceberg tables created on aws glue/athena.
I can `show databases`, `show tables` and query data of iceberg tables.
**However** I can't see **Non** iceberg tables (other formats).
Is that normal?
- My goal is to query non iceberg tables and insert query output into an iceberg table. And/or perform sql join with iceberg and non iceberg tables.
Here is my config:
```
spark.sql.catalog.AwsDataCatalog=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.AwsDataCatalog.type=glue
spark.sql.catalog.AwsDataCatalog.glue.id=...
spark.sql.catalog.AwsDataCatalog.client.factory=org.apache.iceberg.aws.AssumeRoleAwsClientFactory
spark.sql.catalog.AwsDataCatalog.client.assume-role.arn=...
spark.sql.catalog.AwsDataCatalog.client.assume-role.region=...
spark.sql.catalog.AwsDataCatalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
```
Am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]
Posted by "salah-djb (via GitHub)" <gi...@apache.org>.
salah-djb commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2662621838
We implemented this class AWSCredentialsProviderFactory
but I guess there might be better solutions now..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]
Posted by "joshua-yeung-mox (via GitHub)" <gi...@apache.org>.
joshua-yeung-mox commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2655917729
> Finally, I noticed the problem was the session catalog (the default one) was not configured to connect to glue catalog. So it was just an empty local "default" catalog. What must be done at first (before trying to setup anything related to iceberg) is to setup spark catalog using hive dependencies to connect to glue catalog and query non iceberg tables. Once this catalog is setup you can add iceberg configuration with a new catalog based on iceberg tables. Hope this help.
@salah-djb , can you share how to configure the session catalog to connect to glue catalog?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]
Posted by "salahdjb (via GitHub)" <gi...@apache.org>.
salahdjb commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2175328324
Still same error, table is not found ....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]
Posted by "salah-djb (via GitHub)" <gi...@apache.org>.
salah-djb commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2655977524
Hi guys,
To read hive tables from glue catalog you have to patch/build the catalog client like [explained here](https://github.com/awslabs/aws-glue-data-catalog-client-for-apache-hive-metastore?tab=readme-ov-file#patching-apache-hive-and-installing-it-locally)
and define spark configuration for this tables (credentials, catalog id etc) something like:
```
spark.hadoop.aws.catalog.credentials.provider.factory.class=*****
spark.hadoop.aws.glue.assumed.role.arn=*****
spark.hadoop.aws.glue.endpoint=*****
spark.hadoop.aws.region=*****
spark.hadoop.fs.s3a.assumed.role.arn=*****
spark.hadoop.fs.s3a.assumed.role.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider
spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
spark.hadoop.fs.s3a.endpoint=*****
spark.hadoop.hive.imetastoreclient.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
spark.hadoop.hive.metastore.glue.catalogid=*****
```
On top of that if you want to read/write iceberg tables, you must add dependencies for iceberg and add the following configurations (when assuming aws role, otherwise you can also pass pair of access/secret keys)
```
spark.sql.catalog.spark_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
spark.sql.catalog.spark_catalog.client.assume-role.arn=*****
spark.sql.catalog.spark_catalog.client.assume-role.region=*****
spark.sql.catalog.spark_catalog.client.factory=org.apache.iceberg.aws.AssumeRoleAwsClientFactory
spark.sql.catalog.spark_catalog.glue.id=*****
spark.sql.catalog.spark_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.spark_catalog.metrics-reporter-impl=*****
spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.defaultCatalog=spark_catalog
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
```
Hope it helps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]
Posted by "salahdjb (via GitHub)" <gi...@apache.org>.
salahdjb commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2174018221
@amogh-jahagirdar thanks for your reply, I switched to SparkSessionCatalog as you suggested,
I'm getting the following error:
`Caused by: java.lang.NullPointerException: Delegated SessionCatalog is missing. Please make sure your are replacing Spark's default catalog, named 'spark_catalog'.
`
Looking at the [documentation](https://iceberg.apache.org/docs/nightly/spark-configuration/#replacing-the-session-catalog), I also tried to add this conf to my initial configuration:
`spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog`
but I get _TABLE_OR_VIEW_NOT_FOUND_ error.
Do I need to define 2 catalogs? (Since I only have one glue catalog with iceberg and other formats tables.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]
Posted by "boaz-g (via GitHub)" <gi...@apache.org>.
boaz-g commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2620056733
Hello Salah, i am facing the same config challenge .
Can you please post the final conf for the spark session - that supports iceberg and non-iceberg tables in my glue catalog?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]
Posted by "joshua-yeung-mox (via GitHub)" <gi...@apache.org>.
joshua-yeung-mox commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2661813598
@salah-djb Thanks for your explanation. I also want to know which spark.hadoop.aws.catalog.credentials.provider.factory.class should I use if I want to use assumed role for authentication?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2173880721
@salahdjb Using the SparkSessionCatalog instead of just the SparkCatalog should work. SparkSessionCatalog will delegate to the built in catalog for non-iceberg tables. Try the following:
```
spark.sql.catalog.AwsDataCatalog=org.apache.iceberg.spark.SparkSessionCatalog
```
Let me know if that works!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2174734872
Ah sorry I forgot with the SessionCatalog you need to actually call your catalog name "spark_catalog". Please replace AwsDataCatalog in your catalog name in all your configurations with spark_catalog. Also configure spark.sql.defaultCatalog=spark_catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2174739993
Something like
```
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
spark.sql.catalog.spark_catalog.type=glue \
spark.sql.catalog.AwsDataCatalog=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.AwsDataCatalog.type=glue
spark.sql.catalog.AwsDataCatalog.glue.id=...
spark.sql.catalog.AwsDataCatalog.client.factory=org.apache.iceberg.aws.AssumeRoleAwsClientFactory
spark.sql.catalog.AwsDataCatalog.client.assume-role.arn=...
spark.sql.catalog.AwsDataCatalog.client.assume-role.region=...
spark.sql.catalog.AwsDataCatalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]
Posted by "amoghjaha-db (via GitHub)" <gi...@apache.org>.
amoghjaha-db commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2174733735
Ah sorry I forgot with the SessionCatalog you need to actually call your catalog name "spark_catalog". Please replace AwsDataCatalog in your catalog name in all your configurations with `spark_catalog`. Also configure spark.sql.defaultCatalog=spark_catalog. I think that should do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org