You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/10/05 19:56:07 UTC

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #5902: Docs: Add doc for AWS assume role session name and several http client configurations

jackye1995 commented on code in PR #5902:
URL: https://github.com/apache/iceberg/pull/5902#discussion_r985267316


##########
docs/aws.md:
##########
@@ -552,12 +552,13 @@ This also serves as an example for users who would like to implement their own A
 
 This client factory has the following configurable catalog properties:
 
-| Property                          | Default                                  | Description                                            |
-| --------------------------------- | ---------------------------------------- | ------------------------------------------------------ |
-| client.assume-role.arn            | null, requires user input                | ARN of the role to assume, e.g. arn:aws:iam::123456789:role/myRoleToAssume  |
-| client.assume-role.region         | null, requires user input                | All AWS clients except the STS client will use the given region instead of the default region chain  |
-| client.assume-role.external-id    | null                                     | An optional [external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html)  |
-| client.assume-role.timeout-sec    | 1 hour                                   | Timeout of each assume role session. At the end of the timeout, a new set of role session credentials will be fetched through a STS client.  |
+| Property                          | Default                   | Description                                                                                                                                 |

Review Comment:
   no need to change all the lines, just need to add one more line for the new config added



##########
docs/aws.md:
##########
@@ -573,6 +574,55 @@ spark-sql --packages org.apache.iceberg:iceberg-spark3-runtime:{{% icebergVersio
     --conf spark.sql.catalog.my_catalog.client.assume-role.region=ap-northeast-1
 ```
 
+### URL Connection HTTP Client
+In default, AWS clients use the [URL Connection HTTP Client](https://mvnrepository.com/artifact/software.amazon.awssdk/url-connection-client) for HTTP connection management.
+
+This HTTP Client has the following configurable properties:
+
+| Property                                        | Default | Description                                                                                                                                                                                                      |
+|-------------------------------------------------|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| http-client.urlconnection.socket-timeout-ms     | null    | An optional [socket timeout](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/http/urlconnection/UrlConnectionHttpClient.Builder.html#socketTimeout(java.time.Duration)) in milliseconds         |
+| http-client.urlconnection.connection-timeout-ms | null    | An optional [connection timeout](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/http/urlconnection/UrlConnectionHttpClient.Builder.html#connectionTimeout(java.time.Duration)) in milliseconds |
+
+Here is an example to start Spark shell with URL Connection HTTP client and some optional settings:
+```shell
+spark-sql --packages org.apache.iceberg:iceberg-spark3-runtime:{{% icebergVersion %}},software.amazon.awssdk:bundle:2.17.257 \
+    --conf spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog \
+    --conf spark.sql.catalog.my_catalog.warehouse=s3://my-bucket/my/key/prefix \    
+    --conf spark.sql.catalog.my_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \
+    --conf spark.sql.catalog.my_catalog.http-client.urlconnection.socket-timeout-ms=80 \
+    --conf spark.sql.catalog.my_catalog.http-client.urlconnection.connection-timeout-ms=90
+```
+
+### Apache HTTP Client

Review Comment:
   I think we want a bit more levels, 1 general level for HTTP client configuration, an intro paragraph about choosing the type of HTTP client, and 2 sub sections for configuring UrlConnection and Apache type clients.



##########
docs/aws.md:
##########
@@ -552,12 +552,13 @@ This also serves as an example for users who would like to implement their own A
 
 This client factory has the following configurable catalog properties:
 
-| Property                          | Default                                  | Description                                            |
-| --------------------------------- | ---------------------------------------- | ------------------------------------------------------ |
-| client.assume-role.arn            | null, requires user input                | ARN of the role to assume, e.g. arn:aws:iam::123456789:role/myRoleToAssume  |
-| client.assume-role.region         | null, requires user input                | All AWS clients except the STS client will use the given region instead of the default region chain  |
-| client.assume-role.external-id    | null                                     | An optional [external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html)  |
-| client.assume-role.timeout-sec    | 1 hour                                   | Timeout of each assume role session. At the end of the timeout, a new set of role session credentials will be fetched through a STS client.  |
+| Property                          | Default                   | Description                                                                                                                                 |
+|-----------------------------------|---------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+| client.assume-role.arn            | null, requires user input | ARN of the role to assume, e.g. arn:aws:iam::123456789:role/myRoleToAssume                                                                  |
+| client.assume-role.region         | null, requires user input | All AWS clients except the STS client will use the given region instead of the default region chain                                         |
+| client.assume-role.external-id    | null                      | An optional [external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html)                        |
+| client.assume-role.timeout-sec    | 1 hour                    | Timeout of each assume role session. At the end of the timeout, a new set of role session credentials will be fetched through a STS client. |
+| client.assume-role.session-name   | null                      | An optional [session name](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_iam-condition-keys.html#ck_rolesessionname)  |

Review Comment:
   I know you recently added both changes to assume role factory and http client, but they are kind of unrelated. Can we make this only focused on the features in http client? that is already a good amount of changes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org