You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "SEZ9 (via GitHub)" <gi...@apache.org> on 2025/08/03 03:01:42 UTC

[PR] [Feature][Connector-Iceberg]Support iceberg rest catalog integration and support AWS S3 tables [seatunnel]

SEZ9 opened a new pull request, #9654:
URL: https://github.com/apache/seatunnel/pull/9654

   <!--
   
   Thank you for contributing to SeaTunnel! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   ## Contribution Checklist
     - Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/seatunnel/issues).
     - Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
     - Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
   -->
   
   ### Purpose of this pull request
   
   <!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
   
   
   ### Does this PR introduce _any_ user-facing change?
   
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released SeaTunnel versions or within the unreleased branches such as dev.
   If no, write 'No'.
   If you are adding/modifying connector documents, please follow our new specifications: https://github.com/apache/seatunnel/issues/4544.
   -->
   
   
   ### How was this patch tested?
   
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If you are adding E2E test cases, maybe refer to https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/resources/mysqlcdc_to_mysql.conf, here is a good example.
   -->
   
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-Iceberg]Support iceberg rest catalog integration and support AWS S3 tables [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9654:
URL: https://github.com/apache/seatunnel/pull/9654#discussion_r2250312138


##########
seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/IcebergCommonOptions.java:
##########
@@ -87,4 +87,35 @@ public class IcebergCommonOptions {
                     .stringType()
                     .noDefaultValue()
                     .withDescription("When using kerberos, We should specify the keytab path");
+
+    // REST Catalog Configuration
+    public static final Option<String> REST_URI =
+            Options.key("rest.uri")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription("The URI for the REST catalog endpoint");
+
+    public static final Option<String> REST_WAREHOUSE =
+            Options.key("rest.warehouse")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription("The warehouse location for the REST catalog");
+
+    public static final Option<String> REST_AUTH_TYPE =
+            Options.key("rest.auth.type")
+                    .stringType()
+                    .defaultValue("none")
+                    .withDescription(
+                            "Authentication type for REST catalog. Supported values: none, token, aws");
+
+    public static final Option<String> REST_AUTH_TOKEN =
+            Options.key("rest.auth.token")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription(
+                            "Authentication token for REST catalog when auth.type is 'token'");

Review Comment:
   why not directly put all config into `iceberg.catalog.config`?



##########
seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/IcebergCatalogLoader.java:
##########
@@ -58,8 +60,98 @@ public IcebergCatalogLoader(IcebergCommonConfig config) {
     public Catalog loadCatalog() {
         // When using the SeaTunnel engine, set the current class loader to prevent loading failures
         Thread.currentThread().setContextClassLoader(IcebergCatalogLoader.class.getClassLoader());
+
+        // Prepare catalog properties with REST-specific configurations
+        Map<String, String> catalogProps = prepareCatalogProperties();
+
         return CatalogUtil.buildIcebergCatalog(
-                config.getCatalogName(), config.getCatalogProps(), loadHadoopConfig(config));
+                config.getCatalogName(), catalogProps, loadHadoopConfig(config));
+    }
+
+    private Map<String, String> prepareCatalogProperties() {
+        Map<String, String> catalogProps = new HashMap<>();
+        log.info("Preparing catalog properties for REST catalog");
+
+        // Add base catalog properties if provided
+        if (config.getCatalogProps() != null) {
+            catalogProps.putAll(config.getCatalogProps());
+            log.info("Added base catalog properties: {}", config.getCatalogProps());
+        }
+
+        // Add REST catalog specific properties
+        if (config.getRestUri() != null) {
+            catalogProps.put("uri", config.getRestUri());
+            log.info("REST URI configured: {}", config.getRestUri());
+        } else {
+            log.warn("REST URI is not configured");
+        }
+
+        if (config.getRestWarehouse() != null) {
+            catalogProps.put("warehouse", config.getRestWarehouse());
+            log.info("REST warehouse configured: {}", config.getRestWarehouse());
+        } else {
+            log.warn("REST warehouse is not configured");
+        }
+
+        // Handle authentication based on auth type
+        String authType = config.getRestAuthType();
+        log.info("REST authentication type: {}", authType);
+
+        if ("aws".equals(authType)) {
+            log.info("Setting up AWS authentication");
+            setupAwsAuthentication(catalogProps);
+        } else if ("token".equals(authType) && config.getRestAuthToken() != null) {
+            catalogProps.put("token", config.getRestAuthToken());
+            log.info("Token authentication configured");
+        } else if ("none".equals(authType)) {
+            log.info("No authentication configured");
+        } else {
+            log.warn("Unknown or unsupported authentication type: {}", authType);
+        }
+
+        log.info(
+                "Final catalog properties (excluding secrets): {}",
+                catalogProps.entrySet().stream()
+                        .filter(
+                                entry ->
+                                        !entry.getKey().contains("secret")
+                                                && !entry.getKey().contains("token"))
+                        .collect(
+                                java.util.stream.Collectors.toMap(
+                                        java.util.Map.Entry::getKey,
+                                        java.util.Map.Entry::getValue)));
+
+        return catalogProps;
+    }
+
+    private void setupAwsAuthentication(Map<String, String> catalogProps) {
+        log.info("Setting up AWS authentication for REST catalog");
+
+        // Enable SigV4 signing for AWS REST catalog
+        catalogProps.put("rest.sigv4-enabled", "true");
+        log.info("Enabled SigV4 signing");
+
+        // AWS credentials will be resolved through environment variables:
+        // AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN (optional)
+        // AWS_REGION or AWS_DEFAULT_REGION
+        log.info("AWS credentials will be resolved through environment variables");
+
+        // Determine signing service based on REST URI
+        String restUri = config.getRestUri();
+        if (restUri != null) {
+            if (restUri.contains("s3tables")) {
+                catalogProps.put("rest.signing-name", "s3tables");
+                log.info("Signing service set to: s3tables for URI: {}", restUri);
+            } else {
+                catalogProps.put("rest.signing-name", "glue");
+                log.info("Signing service set to: glue for URI: {}", restUri);
+            }
+        } else {
+            catalogProps.put("rest.signing-name", "glue");
+            log.warn("REST URI is null, defaulting signing service to: glue");
+        }

Review Comment:
   Are our existing parameters already capable of reading AWS's rest catalog? Is this PR just a parameter conversion? I prefer to just add examples to the documentation rather than add a new config key.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-Iceberg]Support iceberg rest catalog integration and support AWS S3 tables [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on code in PR #9654:
URL: https://github.com/apache/seatunnel/pull/9654#discussion_r2250321041


##########
seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/IcebergCommonOptions.java:
##########
@@ -87,4 +87,35 @@ public class IcebergCommonOptions {
                     .stringType()
                     .noDefaultValue()
                     .withDescription("When using kerberos, We should specify the keytab path");
+
+    // REST Catalog Configuration
+    public static final Option<String> REST_URI =
+            Options.key("rest.uri")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription("The URI for the REST catalog endpoint");
+
+    public static final Option<String> REST_WAREHOUSE =
+            Options.key("rest.warehouse")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription("The warehouse location for the REST catalog");
+
+    public static final Option<String> REST_AUTH_TYPE =
+            Options.key("rest.auth.type")
+                    .stringType()
+                    .defaultValue("none")
+                    .withDescription(
+                            "Authentication type for REST catalog. Supported values: none, token, aws");
+
+    public static final Option<String> REST_AUTH_TOKEN =
+            Options.key("rest.auth.token")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription(
+                            "Authentication token for REST catalog when auth.type is 'token'");

Review Comment:
   Ok



##########
seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/IcebergCatalogLoader.java:
##########
@@ -58,8 +60,98 @@ public IcebergCatalogLoader(IcebergCommonConfig config) {
     public Catalog loadCatalog() {
         // When using the SeaTunnel engine, set the current class loader to prevent loading failures
         Thread.currentThread().setContextClassLoader(IcebergCatalogLoader.class.getClassLoader());
+
+        // Prepare catalog properties with REST-specific configurations
+        Map<String, String> catalogProps = prepareCatalogProperties();
+
         return CatalogUtil.buildIcebergCatalog(
-                config.getCatalogName(), config.getCatalogProps(), loadHadoopConfig(config));
+                config.getCatalogName(), catalogProps, loadHadoopConfig(config));
+    }
+
+    private Map<String, String> prepareCatalogProperties() {
+        Map<String, String> catalogProps = new HashMap<>();
+        log.info("Preparing catalog properties for REST catalog");
+
+        // Add base catalog properties if provided
+        if (config.getCatalogProps() != null) {
+            catalogProps.putAll(config.getCatalogProps());
+            log.info("Added base catalog properties: {}", config.getCatalogProps());
+        }
+
+        // Add REST catalog specific properties
+        if (config.getRestUri() != null) {
+            catalogProps.put("uri", config.getRestUri());
+            log.info("REST URI configured: {}", config.getRestUri());
+        } else {
+            log.warn("REST URI is not configured");
+        }
+
+        if (config.getRestWarehouse() != null) {
+            catalogProps.put("warehouse", config.getRestWarehouse());
+            log.info("REST warehouse configured: {}", config.getRestWarehouse());
+        } else {
+            log.warn("REST warehouse is not configured");
+        }
+
+        // Handle authentication based on auth type
+        String authType = config.getRestAuthType();
+        log.info("REST authentication type: {}", authType);
+
+        if ("aws".equals(authType)) {
+            log.info("Setting up AWS authentication");
+            setupAwsAuthentication(catalogProps);
+        } else if ("token".equals(authType) && config.getRestAuthToken() != null) {
+            catalogProps.put("token", config.getRestAuthToken());
+            log.info("Token authentication configured");
+        } else if ("none".equals(authType)) {
+            log.info("No authentication configured");
+        } else {
+            log.warn("Unknown or unsupported authentication type: {}", authType);
+        }
+
+        log.info(
+                "Final catalog properties (excluding secrets): {}",
+                catalogProps.entrySet().stream()
+                        .filter(
+                                entry ->
+                                        !entry.getKey().contains("secret")
+                                                && !entry.getKey().contains("token"))
+                        .collect(
+                                java.util.stream.Collectors.toMap(
+                                        java.util.Map.Entry::getKey,
+                                        java.util.Map.Entry::getValue)));
+
+        return catalogProps;
+    }
+
+    private void setupAwsAuthentication(Map<String, String> catalogProps) {
+        log.info("Setting up AWS authentication for REST catalog");
+
+        // Enable SigV4 signing for AWS REST catalog
+        catalogProps.put("rest.sigv4-enabled", "true");
+        log.info("Enabled SigV4 signing");
+
+        // AWS credentials will be resolved through environment variables:
+        // AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN (optional)
+        // AWS_REGION or AWS_DEFAULT_REGION
+        log.info("AWS credentials will be resolved through environment variables");
+
+        // Determine signing service based on REST URI
+        String restUri = config.getRestUri();
+        if (restUri != null) {
+            if (restUri.contains("s3tables")) {
+                catalogProps.put("rest.signing-name", "s3tables");
+                log.info("Signing service set to: s3tables for URI: {}", restUri);
+            } else {
+                catalogProps.put("rest.signing-name", "glue");
+                log.info("Signing service set to: glue for URI: {}", restUri);
+            }
+        } else {
+            catalogProps.put("rest.signing-name", "glue");
+            log.warn("REST URI is null, defaulting signing service to: glue");
+        }

Review Comment:
   No, currently ST's Iceberg catalog only supports Hive and Hadoop, and does not support Rest.  S3 tables is only a special Rest,and need to specify parameters for compatibility.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-Iceberg]Support iceberg rest catalog integration and support AWS S3 tables [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9654:
URL: https://github.com/apache/seatunnel/pull/9654#discussion_r2250325021


##########
seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/IcebergCatalogLoader.java:
##########
@@ -58,8 +60,98 @@ public IcebergCatalogLoader(IcebergCommonConfig config) {
     public Catalog loadCatalog() {
         // When using the SeaTunnel engine, set the current class loader to prevent loading failures
         Thread.currentThread().setContextClassLoader(IcebergCatalogLoader.class.getClassLoader());
+
+        // Prepare catalog properties with REST-specific configurations
+        Map<String, String> catalogProps = prepareCatalogProperties();
+
         return CatalogUtil.buildIcebergCatalog(
-                config.getCatalogName(), config.getCatalogProps(), loadHadoopConfig(config));
+                config.getCatalogName(), catalogProps, loadHadoopConfig(config));
+    }
+
+    private Map<String, String> prepareCatalogProperties() {
+        Map<String, String> catalogProps = new HashMap<>();
+        log.info("Preparing catalog properties for REST catalog");
+
+        // Add base catalog properties if provided
+        if (config.getCatalogProps() != null) {
+            catalogProps.putAll(config.getCatalogProps());
+            log.info("Added base catalog properties: {}", config.getCatalogProps());
+        }
+
+        // Add REST catalog specific properties
+        if (config.getRestUri() != null) {
+            catalogProps.put("uri", config.getRestUri());
+            log.info("REST URI configured: {}", config.getRestUri());
+        } else {
+            log.warn("REST URI is not configured");
+        }
+
+        if (config.getRestWarehouse() != null) {
+            catalogProps.put("warehouse", config.getRestWarehouse());
+            log.info("REST warehouse configured: {}", config.getRestWarehouse());
+        } else {
+            log.warn("REST warehouse is not configured");
+        }
+
+        // Handle authentication based on auth type
+        String authType = config.getRestAuthType();
+        log.info("REST authentication type: {}", authType);
+
+        if ("aws".equals(authType)) {
+            log.info("Setting up AWS authentication");
+            setupAwsAuthentication(catalogProps);
+        } else if ("token".equals(authType) && config.getRestAuthToken() != null) {
+            catalogProps.put("token", config.getRestAuthToken());
+            log.info("Token authentication configured");
+        } else if ("none".equals(authType)) {
+            log.info("No authentication configured");
+        } else {
+            log.warn("Unknown or unsupported authentication type: {}", authType);
+        }
+
+        log.info(
+                "Final catalog properties (excluding secrets): {}",
+                catalogProps.entrySet().stream()
+                        .filter(
+                                entry ->
+                                        !entry.getKey().contains("secret")
+                                                && !entry.getKey().contains("token"))
+                        .collect(
+                                java.util.stream.Collectors.toMap(
+                                        java.util.Map.Entry::getKey,
+                                        java.util.Map.Entry::getValue)));
+
+        return catalogProps;
+    }
+
+    private void setupAwsAuthentication(Map<String, String> catalogProps) {
+        log.info("Setting up AWS authentication for REST catalog");
+
+        // Enable SigV4 signing for AWS REST catalog
+        catalogProps.put("rest.sigv4-enabled", "true");
+        log.info("Enabled SigV4 signing");
+
+        // AWS credentials will be resolved through environment variables:
+        // AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN (optional)
+        // AWS_REGION or AWS_DEFAULT_REGION
+        log.info("AWS credentials will be resolved through environment variables");
+
+        // Determine signing service based on REST URI
+        String restUri = config.getRestUri();
+        if (restUri != null) {
+            if (restUri.contains("s3tables")) {
+                catalogProps.put("rest.signing-name", "s3tables");
+                log.info("Signing service set to: s3tables for URI: {}", restUri);
+            } else {
+                catalogProps.put("rest.signing-name", "glue");
+                log.info("Signing service set to: glue for URI: {}", restUri);
+            }
+        } else {
+            catalogProps.put("rest.signing-name", "glue");
+            log.warn("REST URI is null, defaulting signing service to: glue");
+        }

Review Comment:
   This is a demo:
   ```
   iceberg.catalog.config = {
   type=rest
   uri="http://192.168.124.199:8080/catalog/"
   warehouse=demo2
   "s3.endpoint"="http://192.168.124.199:9000"
   "s3.access-key-id"="******"
   "s3.secret-access-key"="******"
   io-impl="org.apache.iceberg.aws.s3.S3FileIO"
   
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-Iceberg]Support iceberg rest catalog integration and support AWS S3 tables [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9654:
URL: https://github.com/apache/seatunnel/pull/9654#discussion_r2250323959


##########
seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/IcebergCatalogLoader.java:
##########
@@ -58,8 +60,98 @@ public IcebergCatalogLoader(IcebergCommonConfig config) {
     public Catalog loadCatalog() {
         // When using the SeaTunnel engine, set the current class loader to prevent loading failures
         Thread.currentThread().setContextClassLoader(IcebergCatalogLoader.class.getClassLoader());
+
+        // Prepare catalog properties with REST-specific configurations
+        Map<String, String> catalogProps = prepareCatalogProperties();
+
         return CatalogUtil.buildIcebergCatalog(
-                config.getCatalogName(), config.getCatalogProps(), loadHadoopConfig(config));
+                config.getCatalogName(), catalogProps, loadHadoopConfig(config));
+    }
+
+    private Map<String, String> prepareCatalogProperties() {
+        Map<String, String> catalogProps = new HashMap<>();
+        log.info("Preparing catalog properties for REST catalog");
+
+        // Add base catalog properties if provided
+        if (config.getCatalogProps() != null) {
+            catalogProps.putAll(config.getCatalogProps());
+            log.info("Added base catalog properties: {}", config.getCatalogProps());
+        }
+
+        // Add REST catalog specific properties
+        if (config.getRestUri() != null) {
+            catalogProps.put("uri", config.getRestUri());
+            log.info("REST URI configured: {}", config.getRestUri());
+        } else {
+            log.warn("REST URI is not configured");
+        }
+
+        if (config.getRestWarehouse() != null) {
+            catalogProps.put("warehouse", config.getRestWarehouse());
+            log.info("REST warehouse configured: {}", config.getRestWarehouse());
+        } else {
+            log.warn("REST warehouse is not configured");
+        }
+
+        // Handle authentication based on auth type
+        String authType = config.getRestAuthType();
+        log.info("REST authentication type: {}", authType);
+
+        if ("aws".equals(authType)) {
+            log.info("Setting up AWS authentication");
+            setupAwsAuthentication(catalogProps);
+        } else if ("token".equals(authType) && config.getRestAuthToken() != null) {
+            catalogProps.put("token", config.getRestAuthToken());
+            log.info("Token authentication configured");
+        } else if ("none".equals(authType)) {
+            log.info("No authentication configured");
+        } else {
+            log.warn("Unknown or unsupported authentication type: {}", authType);
+        }
+
+        log.info(
+                "Final catalog properties (excluding secrets): {}",
+                catalogProps.entrySet().stream()
+                        .filter(
+                                entry ->
+                                        !entry.getKey().contains("secret")
+                                                && !entry.getKey().contains("token"))
+                        .collect(
+                                java.util.stream.Collectors.toMap(
+                                        java.util.Map.Entry::getKey,
+                                        java.util.Map.Entry::getValue)));
+
+        return catalogProps;
+    }
+
+    private void setupAwsAuthentication(Map<String, String> catalogProps) {
+        log.info("Setting up AWS authentication for REST catalog");
+
+        // Enable SigV4 signing for AWS REST catalog
+        catalogProps.put("rest.sigv4-enabled", "true");
+        log.info("Enabled SigV4 signing");
+
+        // AWS credentials will be resolved through environment variables:
+        // AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN (optional)
+        // AWS_REGION or AWS_DEFAULT_REGION
+        log.info("AWS credentials will be resolved through environment variables");
+
+        // Determine signing service based on REST URI
+        String restUri = config.getRestUri();
+        if (restUri != null) {
+            if (restUri.contains("s3tables")) {
+                catalogProps.put("rest.signing-name", "s3tables");
+                log.info("Signing service set to: s3tables for URI: {}", restUri);
+            } else {
+                catalogProps.put("rest.signing-name", "glue");
+                log.info("Signing service set to: glue for URI: {}", restUri);
+            }
+        } else {
+            catalogProps.put("rest.signing-name", "glue");
+            log.warn("REST URI is null, defaulting signing service to: glue");
+        }

Review Comment:
   [IcebergCatalogType.java](https://github.com/apache/seatunnel/pull/9654/files#diff-803e06e71e5893c50bfdff1fe7940910fc53d03461e0152f8601f3afa0346305) only worked for test case. So I think we support rest catalog. We have also accessed the rest catalog normally through the current version code in our environment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-Iceberg]Support iceberg rest catalog integration and support AWS S3 tables [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on code in PR #9654:
URL: https://github.com/apache/seatunnel/pull/9654#discussion_r2250337069


##########
seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/IcebergCatalogLoader.java:
##########
@@ -58,8 +60,98 @@ public IcebergCatalogLoader(IcebergCommonConfig config) {
     public Catalog loadCatalog() {
         // When using the SeaTunnel engine, set the current class loader to prevent loading failures
         Thread.currentThread().setContextClassLoader(IcebergCatalogLoader.class.getClassLoader());
+
+        // Prepare catalog properties with REST-specific configurations
+        Map<String, String> catalogProps = prepareCatalogProperties();
+
         return CatalogUtil.buildIcebergCatalog(
-                config.getCatalogName(), config.getCatalogProps(), loadHadoopConfig(config));
+                config.getCatalogName(), catalogProps, loadHadoopConfig(config));
+    }
+
+    private Map<String, String> prepareCatalogProperties() {
+        Map<String, String> catalogProps = new HashMap<>();
+        log.info("Preparing catalog properties for REST catalog");
+
+        // Add base catalog properties if provided
+        if (config.getCatalogProps() != null) {
+            catalogProps.putAll(config.getCatalogProps());
+            log.info("Added base catalog properties: {}", config.getCatalogProps());
+        }
+
+        // Add REST catalog specific properties
+        if (config.getRestUri() != null) {
+            catalogProps.put("uri", config.getRestUri());
+            log.info("REST URI configured: {}", config.getRestUri());
+        } else {
+            log.warn("REST URI is not configured");
+        }
+
+        if (config.getRestWarehouse() != null) {
+            catalogProps.put("warehouse", config.getRestWarehouse());
+            log.info("REST warehouse configured: {}", config.getRestWarehouse());
+        } else {
+            log.warn("REST warehouse is not configured");
+        }
+
+        // Handle authentication based on auth type
+        String authType = config.getRestAuthType();
+        log.info("REST authentication type: {}", authType);
+
+        if ("aws".equals(authType)) {
+            log.info("Setting up AWS authentication");
+            setupAwsAuthentication(catalogProps);
+        } else if ("token".equals(authType) && config.getRestAuthToken() != null) {
+            catalogProps.put("token", config.getRestAuthToken());
+            log.info("Token authentication configured");
+        } else if ("none".equals(authType)) {
+            log.info("No authentication configured");
+        } else {
+            log.warn("Unknown or unsupported authentication type: {}", authType);
+        }
+
+        log.info(
+                "Final catalog properties (excluding secrets): {}",
+                catalogProps.entrySet().stream()
+                        .filter(
+                                entry ->
+                                        !entry.getKey().contains("secret")
+                                                && !entry.getKey().contains("token"))
+                        .collect(
+                                java.util.stream.Collectors.toMap(
+                                        java.util.Map.Entry::getKey,
+                                        java.util.Map.Entry::getValue)));
+
+        return catalogProps;
+    }
+
+    private void setupAwsAuthentication(Map<String, String> catalogProps) {
+        log.info("Setting up AWS authentication for REST catalog");
+
+        // Enable SigV4 signing for AWS REST catalog
+        catalogProps.put("rest.sigv4-enabled", "true");
+        log.info("Enabled SigV4 signing");
+
+        // AWS credentials will be resolved through environment variables:
+        // AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN (optional)
+        // AWS_REGION or AWS_DEFAULT_REGION
+        log.info("AWS credentials will be resolved through environment variables");
+
+        // Determine signing service based on REST URI
+        String restUri = config.getRestUri();
+        if (restUri != null) {
+            if (restUri.contains("s3tables")) {
+                catalogProps.put("rest.signing-name", "s3tables");
+                log.info("Signing service set to: s3tables for URI: {}", restUri);
+            } else {
+                catalogProps.put("rest.signing-name", "glue");
+                log.info("Signing service set to: glue for URI: {}", restUri);
+            }
+        } else {
+            catalogProps.put("rest.signing-name", "glue");
+            log.warn("REST URI is null, defaulting signing service to: glue");
+        }

Review Comment:
   let me test 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-Iceberg]Support iceberg rest catalog integration and support AWS S3 tables [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 closed pull request #9654: [Feature][Connector-Iceberg]Support iceberg rest catalog integration and support AWS S3 tables
URL: https://github.com/apache/seatunnel/pull/9654


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-Iceberg]Support iceberg rest catalog integration and support AWS S3 tables [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on code in PR #9654:
URL: https://github.com/apache/seatunnel/pull/9654#discussion_r2249534346


##########
seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/data/RowConverter.java:
##########
@@ -430,11 +429,7 @@ private OffsetDateTime convertOffsetDateTime(Object value) {
         } else if (value instanceof OffsetDateTime) {
             return (OffsetDateTime) value;
         } else if (value instanceof LocalDateTime) {
-            // Convert to OffsetDateTime using the system(jvm) default timezone
-            return ((LocalDateTime) value)
-                    .atZone(ZoneId.systemDefault())
-                    .withZoneSameInstant(ZoneOffset.UTC)
-                    .toOffsetDateTime();
+            return ((LocalDateTime) value).atOffset(ZoneOffset.UTC);

Review Comment:
   This is not right, 
     // In Beijing (UTC+8), developer writes:
     LocalDateTime meeting = LocalDateTime.of(2024, 1, 1, 14, 0, 0); // 2 PM local
   
     // Original code: Correctly converts to 6 AM UTC
     // Modified code:  Treats as 2 PM UTC (8 hours off!)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-Iceberg]Support iceberg rest catalog integration and support AWS S3 tables [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on code in PR #9654:
URL: https://github.com/apache/seatunnel/pull/9654#discussion_r2265145049


##########
seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/IcebergCatalogLoader.java:
##########
@@ -58,8 +60,98 @@ public IcebergCatalogLoader(IcebergCommonConfig config) {
     public Catalog loadCatalog() {
         // When using the SeaTunnel engine, set the current class loader to prevent loading failures
         Thread.currentThread().setContextClassLoader(IcebergCatalogLoader.class.getClassLoader());
+
+        // Prepare catalog properties with REST-specific configurations
+        Map<String, String> catalogProps = prepareCatalogProperties();
+
         return CatalogUtil.buildIcebergCatalog(
-                config.getCatalogName(), config.getCatalogProps(), loadHadoopConfig(config));
+                config.getCatalogName(), catalogProps, loadHadoopConfig(config));
+    }
+
+    private Map<String, String> prepareCatalogProperties() {
+        Map<String, String> catalogProps = new HashMap<>();
+        log.info("Preparing catalog properties for REST catalog");
+
+        // Add base catalog properties if provided
+        if (config.getCatalogProps() != null) {
+            catalogProps.putAll(config.getCatalogProps());
+            log.info("Added base catalog properties: {}", config.getCatalogProps());
+        }
+
+        // Add REST catalog specific properties
+        if (config.getRestUri() != null) {
+            catalogProps.put("uri", config.getRestUri());
+            log.info("REST URI configured: {}", config.getRestUri());
+        } else {
+            log.warn("REST URI is not configured");
+        }
+
+        if (config.getRestWarehouse() != null) {
+            catalogProps.put("warehouse", config.getRestWarehouse());
+            log.info("REST warehouse configured: {}", config.getRestWarehouse());
+        } else {
+            log.warn("REST warehouse is not configured");
+        }
+
+        // Handle authentication based on auth type
+        String authType = config.getRestAuthType();
+        log.info("REST authentication type: {}", authType);
+
+        if ("aws".equals(authType)) {
+            log.info("Setting up AWS authentication");
+            setupAwsAuthentication(catalogProps);
+        } else if ("token".equals(authType) && config.getRestAuthToken() != null) {
+            catalogProps.put("token", config.getRestAuthToken());
+            log.info("Token authentication configured");
+        } else if ("none".equals(authType)) {
+            log.info("No authentication configured");
+        } else {
+            log.warn("Unknown or unsupported authentication type: {}", authType);
+        }
+
+        log.info(
+                "Final catalog properties (excluding secrets): {}",
+                catalogProps.entrySet().stream()
+                        .filter(
+                                entry ->
+                                        !entry.getKey().contains("secret")
+                                                && !entry.getKey().contains("token"))
+                        .collect(
+                                java.util.stream.Collectors.toMap(
+                                        java.util.Map.Entry::getKey,
+                                        java.util.Map.Entry::getValue)));
+
+        return catalogProps;
+    }
+
+    private void setupAwsAuthentication(Map<String, String> catalogProps) {
+        log.info("Setting up AWS authentication for REST catalog");
+
+        // Enable SigV4 signing for AWS REST catalog
+        catalogProps.put("rest.sigv4-enabled", "true");
+        log.info("Enabled SigV4 signing");
+
+        // AWS credentials will be resolved through environment variables:
+        // AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN (optional)
+        // AWS_REGION or AWS_DEFAULT_REGION
+        log.info("AWS credentials will be resolved through environment variables");
+
+        // Determine signing service based on REST URI
+        String restUri = config.getRestUri();
+        if (restUri != null) {
+            if (restUri.contains("s3tables")) {
+                catalogProps.put("rest.signing-name", "s3tables");
+                log.info("Signing service set to: s3tables for URI: {}", restUri);
+            } else {
+                catalogProps.put("rest.signing-name", "glue");
+                log.info("Signing service set to: glue for URI: {}", restUri);
+            }
+        } else {
+            catalogProps.put("rest.signing-name", "glue");
+            log.warn("REST URI is null, defaulting signing service to: glue");
+        }

Review Comment:
   Thanks for @Hisoka-X 's suggestion. After careful consideration and testing, i notice that there's really no need to add separate parameter support for AWS S3 Tables.  I've added documentation for rest catalog integration support in another pr https://github.com/apache/seatunnel/pull/9686 , so i'll close this one . Thank you. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org