You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/05/26 11:58:01 UTC
[GitHub] [iceberg] tiborkiss opened a new issue #2636: [AWS] AssumeRoleAwsClientFactory with Flink
tiborkiss opened a new issue #2636:
URL: https://github.com/apache/iceberg/issues/2636
We have assume role based AWS setup and I would like to try Iceberg with Flink 1.11 . Unfortunately I cannot figure out the setup in case of Flink.
In the [documentation](https://iceberg.apache.org/aws) there is a section **Cross-Account and Cross-Region Access**. That in case of Spark gives an example how to pass
```
--conf spark.sql.catalog.my_catalog.client.factory=org.apache.iceberg.aws.AssumeRoleAwsClientFactory \
--conf spark.sql.catalog.my_catalog.client.assume-role.arn=arn:aws:iam::123456789:role/myRoleToAssume \
--conf spark.sql.catalog.my_catalog.client.assume-role.region=ap-northeast-1
```
I cannot figure out how to apply that for Flink.
Under flink there is a `conf/sql-client-defaults.yaml` which has a section
```
catalogs:
- name: tiko-dev
type: iceberg
warehouse: s3://<mybucket>/<warehouse>
catalog-impl: org.apache.iceberg.aws.glue.GlueCatalog
io-impl: org.apache.iceberg.aws.s3.S3FileIO
lock-impl: org.apache.iceberg.aws.glue.DynamoLockManager
lock.table: myGlueLockTable
```
Unfortunately it cannot parse if I add
```
client.factory=org.apache.iceberg.aws.AssumeRoleAwsClientFactory
client.assume-role.arn=arn:aws:iam::1212121121212:role/rolename
client.assume-role.region=eu-central-1
```
fails with
```
No default environment specified.
Searching for '/home/tibor/flink/flink-1.11.3/conf/sql-client-defaults.yaml'...found.
Reading default environment from: file:/home/tibor/flink/flink-1.11.3/conf/sql-client-defaults.yaml
Exception in thread "main" org.apache.flink.table.client.SqlClientException: Could not parse environment file. Cause: while scanning a simple key
in 'reader', line 85, column 5:
client.factory=org.apache.iceber ...
^
could not find expected ':'
in 'reader', line 86, column 5:
client.assume-role.arn=arn:aws:i ...
^
at [Source: (URL); line: 84, column: 19] (through reference chain: org.apache.flink.table.client.config.Environment["catalogs"]->java.util.ArrayList[0])
at org.apache.flink.table.client.config.Environment.parse(Environment.java:215)
at org.apache.flink.table.client.gateway.local.LocalExecutor.<init>(LocalExecutor.java:163)
at org.apache.flink.table.client.SqlClient.start(SqlClient.java:94)
at org.apache.flink.table.client.SqlClient.main(SqlClient.java:201)
```
Maybe someone knows how to configure AWS + Flink + Iceberg with AssumeRoleAwsClientFactory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] tiborkiss commented on issue #2636: [AWS] AssumeRoleAwsClientFactory with Flink
Posted by GitBox <gi...@apache.org>.
tiborkiss commented on issue #2636:
URL: https://github.com/apache/iceberg/issues/2636#issuecomment-854490837
I close because we had a non-standard aws client configuration, which caused the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] tiborkiss closed issue #2636: [AWS] AssumeRoleAwsClientFactory with Flink
Posted by GitBox <gi...@apache.org>.
tiborkiss closed issue #2636:
URL: https://github.com/apache/iceberg/issues/2636
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org