[GitHub] [iceberg] rajarshisarkar commented on issue #4804: Iceberg Streaming Job Can't run on Fargate on EMR on EKS

[GitBox <gi...@apache.org>]

rajarshisarkar commented on issue #4804:
URL: https://github.com/apache/iceberg/issues/4804#issuecomment-1131501318

   Seems like there is an issue to load the credentials. Is the AWS SDK v2 library in the classpath? Possibly the `--jars` path is breaking at one place before the AWS SDK v2 library due to `'s3-data-bucket'`:
   
   ```
   "sparkSubmitParameters": "--jars .....,s3://'s3-data-bucket'/spark-sql-kinesis_2.12-1.2.1_spark-3.0-SNAPSHOT.jar,https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.15.40/bundle-2.15.40.jar,https://repo1.maven.org/maven2/software/amazon/awssdk/url-connection-client/2.15.40/url-connection-client-2.15.40.jar"
   ```
   
   If it is in the classpath, can you please explicitly try providing the access key specified either via environment variable (`AWS_ACCESS_KEY_ID`) or system property (`aws.accessKeyId`) for testing purpose.
   
   Also, I couldn't find the Dynamo DB lock manager Spark configuration: `--conf spark.sql.catalog.my_catalog.lock-impl=org.apache.iceberg.aws.glue.DynamoLockManager`. Refer: [DynamoDB for Commit Locking](https://iceberg.apache.org/docs/latest/aws/#dynamodb-for-commit-locking)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org