You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "salahdjb (via GitHub)" <gi...@apache.org> on 2024/06/17 16:44:04 UTC

[I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

salahdjb opened a new issue, #10524:
URL: https://github.com/apache/iceberg/issues/10524

   ### Query engine
   
   - Spark 3.5.0
   - Iceberg 1.5.2
   
   ```
   iceberg-spark-runtime-3.5_2.12-1.5.2.jar
   iceberg-aws-bundle-1.5.2.jar
   ```
   
   ### Question
   
   I successfully opened a spark sql shell from which I can query iceberg tables created on aws glue/athena.
   I can `show databases`, `show tables` and query data of iceberg tables.
   **However** I can't see **Non** iceberg tables (other formats).
   
   Is that normal?
   - My goal is to query non iceberg tables and insert query output into an iceberg table. And/or perform sql join with iceberg and non iceberg tables.
   
   Here is my config: 
   ```
   spark.sql.catalog.AwsDataCatalog=org.apache.iceberg.spark.SparkCatalog
   spark.sql.catalog.AwsDataCatalog.type=glue
   spark.sql.catalog.AwsDataCatalog.glue.id=...
   spark.sql.catalog.AwsDataCatalog.client.factory=org.apache.iceberg.aws.AssumeRoleAwsClientFactory
   spark.sql.catalog.AwsDataCatalog.client.assume-role.arn=...
   spark.sql.catalog.AwsDataCatalog.client.assume-role.region=...
   spark.sql.catalog.AwsDataCatalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
   ```
   
   Am I missing something?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

Posted by "salah-djb (via GitHub)" <gi...@apache.org>.
salah-djb commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2662621838

   We implemented this class AWSCredentialsProviderFactory
   but I guess there might be better solutions now..


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

Posted by "joshua-yeung-mox (via GitHub)" <gi...@apache.org>.
joshua-yeung-mox commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2655917729

   > Finally, I noticed the problem was the session catalog (the default one) was not configured to connect to glue catalog. So it was just an empty local "default" catalog. What must be done at first (before trying to setup anything related to iceberg) is to setup spark catalog using hive dependencies to connect to glue catalog and query non iceberg tables. Once this catalog is setup you can add iceberg configuration with a new catalog based on iceberg tables. Hope this help.
   
   @salah-djb , can you share how to configure the session catalog to connect to glue catalog?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

Posted by "salahdjb (via GitHub)" <gi...@apache.org>.
salahdjb commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2175328324

   Still same error, table is not found ....


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

Posted by "salah-djb (via GitHub)" <gi...@apache.org>.
salah-djb commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2655977524

   Hi guys, 
   To read hive tables from glue catalog you have to patch/build the catalog client like [explained here](https://github.com/awslabs/aws-glue-data-catalog-client-for-apache-hive-metastore?tab=readme-ov-file#patching-apache-hive-and-installing-it-locally)
   and define spark configuration for this tables (credentials, catalog id etc) something like:
   
   ```
   spark.hadoop.aws.catalog.credentials.provider.factory.class=*****
   spark.hadoop.aws.glue.assumed.role.arn=*****
   spark.hadoop.aws.glue.endpoint=*****
   spark.hadoop.aws.region=*****
   spark.hadoop.fs.s3a.assumed.role.arn=*****
   spark.hadoop.fs.s3a.assumed.role.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider
   spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
   spark.hadoop.fs.s3a.endpoint=*****
   spark.hadoop.hive.imetastoreclient.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
   spark.hadoop.hive.metastore.glue.catalogid=*****
   
   ```
   
   On top of that if you want to read/write iceberg tables, you must add dependencies for iceberg and add the following configurations (when assuming aws role, otherwise you can also pass pair of access/secret keys)
   ```
   spark.sql.catalog.spark_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
   spark.sql.catalog.spark_catalog.client.assume-role.arn=*****
   spark.sql.catalog.spark_catalog.client.assume-role.region=*****
   spark.sql.catalog.spark_catalog.client.factory=org.apache.iceberg.aws.AssumeRoleAwsClientFactory
   spark.sql.catalog.spark_catalog.glue.id=*****
   spark.sql.catalog.spark_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
   spark.sql.catalog.spark_catalog.metrics-reporter-impl=*****
   spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog
   spark.sql.defaultCatalog=spark_catalog
   spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
   
   ```
   Hope it helps


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

Posted by "salahdjb (via GitHub)" <gi...@apache.org>.
salahdjb commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2174018221

   @amogh-jahagirdar thanks for your reply, I switched to SparkSessionCatalog as you suggested, 
   I'm getting the following error:
   
   `Caused by: java.lang.NullPointerException: Delegated SessionCatalog is missing. Please make sure your are replacing Spark's default catalog, named 'spark_catalog'.
   `
   
   
   Looking at the [documentation](https://iceberg.apache.org/docs/nightly/spark-configuration/#replacing-the-session-catalog), I also tried to add this conf to my initial configuration:
   
   `spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog` 
   
    but I get _TABLE_OR_VIEW_NOT_FOUND_ error.
   
   Do I need to define 2 catalogs? (Since I only have one glue catalog with iceberg and other formats tables.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

Posted by "boaz-g (via GitHub)" <gi...@apache.org>.
boaz-g commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2620056733

   Hello Salah, i am facing the same config challenge .
   Can you please post the final conf for the spark session - that supports iceberg and non-iceberg tables in my glue catalog?
   Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

Posted by "joshua-yeung-mox (via GitHub)" <gi...@apache.org>.
joshua-yeung-mox commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2661813598

   @salah-djb Thanks for your explanation. I also want to know which spark.hadoop.aws.catalog.credentials.provider.factory.class should I use if I want to use assumed role for authentication?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2173880721

   @salahdjb Using the SparkSessionCatalog instead of just the SparkCatalog should work. SparkSessionCatalog will delegate to the built in catalog for non-iceberg tables. Try the following:
   ```
   spark.sql.catalog.AwsDataCatalog=org.apache.iceberg.spark.SparkSessionCatalog
   ```
   
   Let me know if that works! 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2174734872

   Ah sorry I forgot with the SessionCatalog you need to actually call your catalog name "spark_catalog". Please replace AwsDataCatalog in your catalog name in all your configurations with spark_catalog. Also configure spark.sql.defaultCatalog=spark_catalog. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2174739993

   Something like 
   
   ```
   spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
   spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
   spark.sql.catalog.spark_catalog.type=glue \
   spark.sql.catalog.AwsDataCatalog=org.apache.iceberg.spark.SparkCatalog
   spark.sql.catalog.AwsDataCatalog.type=glue
   spark.sql.catalog.AwsDataCatalog.glue.id=...
   spark.sql.catalog.AwsDataCatalog.client.factory=org.apache.iceberg.aws.AssumeRoleAwsClientFactory
   spark.sql.catalog.AwsDataCatalog.client.assume-role.arn=...
   spark.sql.catalog.AwsDataCatalog.client.assume-role.region=...
   spark.sql.catalog.AwsDataCatalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

Posted by "amoghjaha-db (via GitHub)" <gi...@apache.org>.
amoghjaha-db commented on issue #10524:
URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2174733735

   Ah sorry I forgot with the SessionCatalog you need to actually call your catalog name "spark_catalog". Please replace AwsDataCatalog in your catalog name in all your configurations with `spark_catalog`. Also configure spark.sql.defaultCatalog=spark_catalog. I think that should do it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org