[GitHub] [iceberg] lgbaeza commented on issue #5369: Unable to query Iceberg table from PySpark script in AWS Glue

[GitBox <gi...@apache.org>]

lgbaeza commented on issue #5369:
URL: https://github.com/apache/iceberg/issues/5369#issuecomment-1217304300

   > Updated the config to this (based on iceberg table configuration):
   > 
   > `spark = SparkSession.builder.config("spark.driver.memory", "25g").config("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog").config("spark.sql.catalog.spark_catalog.type", "hive").appName(app_name).getOrCreate()`
   > 
   > I'm getting this new error - `An error occurred while calling o87.sql. Cannot find catalog plugin class for catalog 'spark_catalog': org.apache.iceberg.spark.SparkSessionCatalog`
   
   That error seems to be due to not having the dependencies for iceberg. Have you configured your glue job with the iceberg connector from marketplace? Here you can find how https://aws.amazon.com/es/blogs/big-data/implement-a-cdc-based-upsert-in-a-data-lake-using-apache-iceberg-and-aws-glue/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org