You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "mansipp (via GitHub)" <gi...@apache.org> on 2023/08/02 22:20:17 UTC

[GitHub] [hudi] mansipp opened a new pull request, #9347: Upgrade aws java sdk to v2

mansipp opened a new pull request, #9347:
URL: https://github.com/apache/hudi/pull/9347

   ### Change Logs
   
   _Describe context and summary for this change. Highlight if any code was copied._
   
   ### Impact
   
   _Describe any public API or user-facing feature change or any performance impact._
   
   ### Risk level (write none, low medium or high below)
   
   _If medium or high, explain what verification was done to mitigate the risks._
   
   ### Documentation Update
   
   _Describe any necessary documentation update if there is any new feature, config, or user-facing change_
   
   - _The config description must be updated if new configs are added or the default value of the configs are changed_
   - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
     ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
     changes to the website._
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] mansipp commented on a diff in pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

Posted by "mansipp (via GitHub)" <gi...@apache.org>.
mansipp commented on code in PR #9347:
URL: https://github.com/apache/hudi/pull/9347#discussion_r1284641431


##########
hudi-utilities/pom.xml:
##########
@@ -450,6 +450,14 @@
       <scope>test</scope>
     </dependency>
 
+    <!-- AWS Services -->
+    <!-- https://mvnrepository.com/artifact/software.amazon.awssdk/aws-java-sdk-sqs -->
+    <dependency>
+      <groupId>software.amazon.awssdk</groupId>
+      <artifactId>sqs</artifactId>
+      <version>${aws.sdk.version}</version>
+    </dependency>

Review Comment:
   We have added these 2 dependencies back to the ```hudi-utililes``` pom because of the ```TestS3EventsSource```  ```java.lang.NoClassDefFoundError: org/apache/http/impl/client/DefaultClientConnectionReuseStrategy``` failure. We still need to identify that why these dependencies are required to be specified in ```hudi-utilities``` even though ```hudi-aws``` is pulling them in ```hudi-utililes```
   ```
   <dependency>
         <groupId>org.apache.httpcomponents</groupId>
         <artifactId>httpclient</artifactId>
         <version>${aws.sdk.httpclient.version}</version>
       </dependency>
       <dependency>
         <groupId>org.apache.httpcomponents</groupId>
         <artifactId>httpcore</artifactId>
         <version>${aws.sdk.httpcore.version}</version>
       </dependency>
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #9347: Upgrade aws java sdk to v2

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9347:
URL: https://github.com/apache/hudi/pull/9347#issuecomment-1663099148

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19015",
       "triggerID" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d2360a5a7de655991202680013d20268ce325666",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d2360a5a7de655991202680013d20268ce325666",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4e17424eda9aa3bd50841ebc0f8846305b27f6d2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19015) 
   * d2360a5a7de655991202680013d20268ce325666 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] mansipp commented on a diff in pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

Posted by "mansipp (via GitHub)" <gi...@apache.org>.
mansipp commented on code in PR #9347:
URL: https://github.com/apache/hudi/pull/9347#discussion_r1283879950


##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -143,31 +142,32 @@ public void addPartitionsToTable(String tableName, List<String> partitionsToAdd)
     LOG.info("Adding " + partitionsToAdd.size() + " partition(s) in table " + tableId(databaseName, tableName));
     try {
       Table table = getTable(awsGlue, databaseName, tableName);
-      StorageDescriptor sd = table.getStorageDescriptor();
+      StorageDescriptor sd = table.storageDescriptor();
       List<PartitionInput> partitionInputs = partitionsToAdd.stream().map(partition -> {
-        StorageDescriptor partitionSd = sd.clone();
         String fullPartitionPath = FSUtils.getPartitionPath(getBasePath(), partition).toString();
         List<String> partitionValues = partitionValueExtractor.extractPartitionValuesInPath(partition);
-        partitionSd.setLocation(fullPartitionPath);
-        return new PartitionInput().withValues(partitionValues).withStorageDescriptor(partitionSd);
+        StorageDescriptor partitionSD = sd.copy(copySd -> copySd.location(fullPartitionPath));
+        return PartitionInput.builder().values(partitionValues).storageDescriptor(partitionSD).build();
       }).collect(Collectors.toList());
 
-      List<Future<BatchCreatePartitionResult>> futures = new ArrayList<>();
+      List<CompletableFuture<BatchCreatePartitionResponse>> futures = new ArrayList<>();
 
       for (List<PartitionInput> batch : CollectionUtils.batches(partitionInputs, MAX_PARTITIONS_PER_REQUEST)) {
-        BatchCreatePartitionRequest request = new BatchCreatePartitionRequest();
-        request.withDatabaseName(databaseName).withTableName(tableName).withPartitionInputList(batch);
-        futures.add(awsGlue.batchCreatePartitionAsync(request));
+        BatchCreatePartitionRequest request = BatchCreatePartitionRequest.builder()
+                .databaseName(databaseName).tableName(tableName).partitionInputList(batch).build();
+        futures.add(awsGlue.batchCreatePartition(request));
       }
 
-      for (Future<BatchCreatePartitionResult> future : futures) {
-        BatchCreatePartitionResult result = future.get();
-        if (CollectionUtils.nonEmpty(result.getErrors())) {
-          if (result.getErrors().stream().allMatch((error) -> "AlreadyExistsException".equals(error.getErrorDetail().getErrorCode()))) {
-            LOG.warn("Partitions already exist in glue: " + result.getErrors());
+      for (CompletableFuture<BatchCreatePartitionResponse> future : futures) {
+        BatchCreatePartitionResponse response = future.get();
+        if (CollectionUtils.nonEmpty(response.errors())) {
+          if (response.errors().stream()
+              .allMatch(
+                  (error) -> "AlreadyExistsException".equals(error.errorDetail().errorCode()))) {

Review Comment:
   We haven't changed anything here other than making it compatible with sdk v2. Haven't run any test to specifically test this error case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #9347: Upgrade aws java sdk to v2

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9347:
URL: https://github.com/apache/hudi/pull/9347#issuecomment-1663324203

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19015",
       "triggerID" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d2360a5a7de655991202680013d20268ce325666",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19016",
       "triggerID" : "d2360a5a7de655991202680013d20268ce325666",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d2360a5a7de655991202680013d20268ce325666 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19016) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] yihua commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #9347:
URL: https://github.com/apache/hudi/pull/9347#issuecomment-1664836797

   @mansipp could you also check the Azure CI failure?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9347:
URL: https://github.com/apache/hudi/pull/9347#issuecomment-1664955745

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19015",
       "triggerID" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d2360a5a7de655991202680013d20268ce325666",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19016",
       "triggerID" : "d2360a5a7de655991202680013d20268ce325666",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5c1391571fbed3ce391399b7848f33b629455941",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19041",
       "triggerID" : "5c1391571fbed3ce391399b7848f33b629455941",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5c1391571fbed3ce391399b7848f33b629455941",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19057",
       "triggerID" : "1664944883",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 5c1391571fbed3ce391399b7848f33b629455941 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19041) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19057) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #9347: Upgrade aws java sdk to v2

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9347:
URL: https://github.com/apache/hudi/pull/9347#issuecomment-1663065177

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19015",
       "triggerID" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4e17424eda9aa3bd50841ebc0f8846305b27f6d2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19015) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] yihua commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #9347:
URL: https://github.com/apache/hudi/pull/9347#issuecomment-1665666102

   Azure CI failures is due to a flaky test in `hudi-flink` which is irrelevant to this PR.  Merging this now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #9347: Upgrade aws java sdk to v2

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9347:
URL: https://github.com/apache/hudi/pull/9347#issuecomment-1663103718

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19015",
       "triggerID" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d2360a5a7de655991202680013d20268ce325666",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19016",
       "triggerID" : "d2360a5a7de655991202680013d20268ce325666",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4e17424eda9aa3bd50841ebc0f8846305b27f6d2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19015) 
   * d2360a5a7de655991202680013d20268ce325666 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19016) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to v2

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9347:
URL: https://github.com/apache/hudi/pull/9347#issuecomment-1664392777

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19015",
       "triggerID" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d2360a5a7de655991202680013d20268ce325666",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19016",
       "triggerID" : "d2360a5a7de655991202680013d20268ce325666",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5c1391571fbed3ce391399b7848f33b629455941",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19041",
       "triggerID" : "5c1391571fbed3ce391399b7848f33b629455941",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d2360a5a7de655991202680013d20268ce325666 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19016) 
   * 5c1391571fbed3ce391399b7848f33b629455941 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19041) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9347:
URL: https://github.com/apache/hudi/pull/9347#issuecomment-1665392297

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19015",
       "triggerID" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d2360a5a7de655991202680013d20268ce325666",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19016",
       "triggerID" : "d2360a5a7de655991202680013d20268ce325666",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5c1391571fbed3ce391399b7848f33b629455941",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19041",
       "triggerID" : "5c1391571fbed3ce391399b7848f33b629455941",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5c1391571fbed3ce391399b7848f33b629455941",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19057",
       "triggerID" : "1664944883",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c24825896bd7893fe303f240b523bd20ae1e7567",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19059",
       "triggerID" : "c24825896bd7893fe303f240b523bd20ae1e7567",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c24825896bd7893fe303f240b523bd20ae1e7567 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19059) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to v2

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9347:
URL: https://github.com/apache/hudi/pull/9347#issuecomment-1664381788

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19015",
       "triggerID" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d2360a5a7de655991202680013d20268ce325666",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19016",
       "triggerID" : "d2360a5a7de655991202680013d20268ce325666",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5c1391571fbed3ce391399b7848f33b629455941",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5c1391571fbed3ce391399b7848f33b629455941",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d2360a5a7de655991202680013d20268ce325666 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19016) 
   * 5c1391571fbed3ce391399b7848f33b629455941 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to v2

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9347:
URL: https://github.com/apache/hudi/pull/9347#issuecomment-1664723220

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19015",
       "triggerID" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d2360a5a7de655991202680013d20268ce325666",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19016",
       "triggerID" : "d2360a5a7de655991202680013d20268ce325666",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5c1391571fbed3ce391399b7848f33b629455941",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19041",
       "triggerID" : "5c1391571fbed3ce391399b7848f33b629455941",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5c1391571fbed3ce391399b7848f33b629455941 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19041) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] mansipp commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

Posted by "mansipp (via GitHub)" <gi...@apache.org>.
mansipp commented on PR #9347:
URL: https://github.com/apache/hudi/pull/9347#issuecomment-1664944883

   @hudi-bot run azure
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #9347: Upgrade aws java sdk to v2

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9347:
URL: https://github.com/apache/hudi/pull/9347#issuecomment-1663059384

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4e17424eda9aa3bd50841ebc0f8846305b27f6d2 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9347:
URL: https://github.com/apache/hudi/pull/9347#issuecomment-1664960199

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19015",
       "triggerID" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d2360a5a7de655991202680013d20268ce325666",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19016",
       "triggerID" : "d2360a5a7de655991202680013d20268ce325666",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5c1391571fbed3ce391399b7848f33b629455941",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19041",
       "triggerID" : "5c1391571fbed3ce391399b7848f33b629455941",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5c1391571fbed3ce391399b7848f33b629455941",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19057",
       "triggerID" : "1664944883",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c24825896bd7893fe303f240b523bd20ae1e7567",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c24825896bd7893fe303f240b523bd20ae1e7567",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5c1391571fbed3ce391399b7848f33b629455941 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19041) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19057) 
   * c24825896bd7893fe303f240b523bd20ae1e7567 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] yihua merged pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #9347:
URL: https://github.com/apache/hudi/pull/9347


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] mansipp commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to v2

Posted by "mansipp (via GitHub)" <gi...@apache.org>.
mansipp commented on PR #9347:
URL: https://github.com/apache/hudi/pull/9347#issuecomment-1664649658

   Manually tested s3a path using EMR cluster. 
   
   ```scala
   spark-shell \
   --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" \
   --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" \
   --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" \
   --conf "spark.sql.hive.convertMetastoreParquet=false" \
   --jars /usr/lib/hudi/hudi-aws-bundle-0.13.1-amzn-1-SNAPSHOT.jar,/usr/lib/hudi/hudi-spark3-bundle_2.12-0.13.1-amzn-1-SNAPSHOT.jar
   ```
   
   ```scala
   import org.apache.spark.sql.SaveMode
   import org.apache.spark.sql.functions._
   import org.apache.hudi.DataSourceWriteOptions
   import org.apache.hudi.DataSourceReadOptions
   import org.apache.hudi.config.HoodieWriteConfig
   import org.apache.hudi.hive.MultiPartKeysValueExtractor
   import org.apache.hudi.hive.HiveSyncConfig
   import org.apache.hudi.sync.common.HoodieSyncConfig
   
   // Create a DataFrame
   var tableName = "mansi_s3a_hudi_test"
   var tablePath = "s3a://<myBucket>/tables/" + tableName
   val inputDF = Seq(
    ("100", "2015-01-01", "2015-01-01T13:51:39.340396Z"),
    ("101", "2015-01-01", "2015-01-01T12:14:58.597216Z"),
    ("102", "2015-01-01", "2015-01-01T13:51:40.417052Z"),
    ("103", "2015-01-01", "2015-01-01T13:51:40.519832Z"),
    ("104", "2015-01-02", "2015-01-01T12:15:00.512679Z"),
    ("105", "2015-01-02", "2015-01-01T13:51:42.248818Z")
    ).toDF("id", "creation_date", "last_update_time")
   
   //Specify common DataSourceWriteOptions in the single hudiOptions variable 
   val hudiOptions = Map[String,String](
     HoodieWriteConfig.TABLE_NAME -> tableName,
     DataSourceWriteOptions.TABLE_TYPE_OPT_KEY -> "COPY_ON_WRITE", 
     DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY -> "id",
     DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY -> "creation_date",
     DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY -> "last_update_time",
     DataSourceWriteOptions.HIVE_SYNC_ENABLED_OPT_KEY -> "true",
     DataSourceWriteOptions.HIVE_TABLE_OPT_KEY -> tableName,
     DataSourceWriteOptions.HIVE_PARTITION_FIELDS_OPT_KEY -> "creation_date",
     DataSourceWriteOptions.HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY -> classOf[MultiPartKeysValueExtractor].getName
   )
   
   // Write the DataFrame as a Hudi dataset
   (inputDF.write
       .format("org.apache.hudi")
       .option(DataSourceWriteOptions.OPERATION_OPT_KEY, DataSourceWriteOptions.INSERT_OPERATION_OPT_VAL)
       .options(hudiOptions)
       .mode(SaveMode.Overwrite)
       .save(tablePath))
   
   ``` 
   ![Image](https://github.com/apache/hudi/assets/107222979/672d5057-cbe5-43f7-b927-838510b1220c)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9347:
URL: https://github.com/apache/hudi/pull/9347#issuecomment-1665035080

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19015",
       "triggerID" : "4e17424eda9aa3bd50841ebc0f8846305b27f6d2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d2360a5a7de655991202680013d20268ce325666",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19016",
       "triggerID" : "d2360a5a7de655991202680013d20268ce325666",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5c1391571fbed3ce391399b7848f33b629455941",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19041",
       "triggerID" : "5c1391571fbed3ce391399b7848f33b629455941",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5c1391571fbed3ce391399b7848f33b629455941",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19057",
       "triggerID" : "1664944883",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c24825896bd7893fe303f240b523bd20ae1e7567",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19059",
       "triggerID" : "c24825896bd7893fe303f240b523bd20ae1e7567",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5c1391571fbed3ce391399b7848f33b629455941 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19041) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19057) 
   * c24825896bd7893fe303f240b523bd20ae1e7567 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19059) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] yihua commented on a diff in pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #9347:
URL: https://github.com/apache/hudi/pull/9347#discussion_r1283849039


##########
pom.xml:
##########
@@ -130,6 +130,8 @@
     <orc.flink.version>1.5.6</orc.flink.version>
     <airlift.version>0.16</airlift.version>
     <prometheus.version>0.8.0</prometheus.version>
+    <aws.sdk.httpclient.version>4.5.13</aws.sdk.httpclient.version>
+    <aws.sdk.httpcore.version>4.4.13</aws.sdk.httpcore.version>

Review Comment:
   Any reason we pick different versions here?



##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -143,31 +142,32 @@ public void addPartitionsToTable(String tableName, List<String> partitionsToAdd)
     LOG.info("Adding " + partitionsToAdd.size() + " partition(s) in table " + tableId(databaseName, tableName));
     try {
       Table table = getTable(awsGlue, databaseName, tableName);
-      StorageDescriptor sd = table.getStorageDescriptor();
+      StorageDescriptor sd = table.storageDescriptor();
       List<PartitionInput> partitionInputs = partitionsToAdd.stream().map(partition -> {
-        StorageDescriptor partitionSd = sd.clone();
         String fullPartitionPath = FSUtils.getPartitionPath(getBasePath(), partition).toString();
         List<String> partitionValues = partitionValueExtractor.extractPartitionValuesInPath(partition);
-        partitionSd.setLocation(fullPartitionPath);
-        return new PartitionInput().withValues(partitionValues).withStorageDescriptor(partitionSd);
+        StorageDescriptor partitionSD = sd.copy(copySd -> copySd.location(fullPartitionPath));
+        return PartitionInput.builder().values(partitionValues).storageDescriptor(partitionSD).build();
       }).collect(Collectors.toList());
 
-      List<Future<BatchCreatePartitionResult>> futures = new ArrayList<>();
+      List<CompletableFuture<BatchCreatePartitionResponse>> futures = new ArrayList<>();
 
       for (List<PartitionInput> batch : CollectionUtils.batches(partitionInputs, MAX_PARTITIONS_PER_REQUEST)) {
-        BatchCreatePartitionRequest request = new BatchCreatePartitionRequest();
-        request.withDatabaseName(databaseName).withTableName(tableName).withPartitionInputList(batch);
-        futures.add(awsGlue.batchCreatePartitionAsync(request));
+        BatchCreatePartitionRequest request = BatchCreatePartitionRequest.builder()
+                .databaseName(databaseName).tableName(tableName).partitionInputList(batch).build();
+        futures.add(awsGlue.batchCreatePartition(request));
       }
 
-      for (Future<BatchCreatePartitionResult> future : futures) {
-        BatchCreatePartitionResult result = future.get();
-        if (CollectionUtils.nonEmpty(result.getErrors())) {
-          if (result.getErrors().stream().allMatch((error) -> "AlreadyExistsException".equals(error.getErrorDetail().getErrorCode()))) {
-            LOG.warn("Partitions already exist in glue: " + result.getErrors());
+      for (CompletableFuture<BatchCreatePartitionResponse> future : futures) {
+        BatchCreatePartitionResponse response = future.get();
+        if (CollectionUtils.nonEmpty(response.errors())) {
+          if (response.errors().stream()
+              .allMatch(
+                  (error) -> "AlreadyExistsException".equals(error.errorDetail().errorCode()))) {

Review Comment:
   I assume the error name is the same here and this should still work.  Is the error case tested?



##########
hudi-utilities/pom.xml:
##########
@@ -450,6 +450,14 @@
       <scope>test</scope>
     </dependency>
 
+    <!-- AWS Services -->
+    <!-- https://mvnrepository.com/artifact/software.amazon.awssdk/aws-java-sdk-sqs -->
+    <dependency>
+      <groupId>software.amazon.awssdk</groupId>
+      <artifactId>sqs</artifactId>
+      <version>${aws.sdk.version}</version>
+    </dependency>

Review Comment:
   Same question as before: https://github.com/apache/hudi/pull/8441/files#r1164644285
   Should this be moved to `hudi-aws` module (`hudi-utilities` module has already relied on `hudi-aws` module)?



##########
hudi-aws/src/main/java/org/apache/hudi/aws/transaction/lock/DynamoDBBasedLockProvider.java:
##########
@@ -153,45 +153,53 @@ public LockItem getLock() {
     return lock;
   }
 
-  private AmazonDynamoDB getDynamoDBClient() {
+  private DynamoDbClient getDynamoDBClient() {
     String region = this.dynamoDBLockConfiguration.getString(DynamoDbBasedLockConfig.DYNAMODB_LOCK_REGION);
     String endpointURL = this.dynamoDBLockConfiguration.contains(DynamoDbBasedLockConfig.DYNAMODB_ENDPOINT_URL.key())
-                          ? this.dynamoDBLockConfiguration.getString(DynamoDbBasedLockConfig.DYNAMODB_ENDPOINT_URL)
-                          : RegionUtils.getRegion(region).getServiceEndpoint(AmazonDynamoDB.ENDPOINT_PREFIX);
-    AwsClientBuilder.EndpointConfiguration dynamodbEndpoint =
-            new AwsClientBuilder.EndpointConfiguration(endpointURL, region);
-    return AmazonDynamoDBClientBuilder.standard()
-            .withEndpointConfiguration(dynamodbEndpoint)
-            .withCredentials(HoodieAWSCredentialsProviderFactory.getAwsCredentialsProvider(dynamoDBLockConfiguration.getProps()))
+            ? this.dynamoDBLockConfiguration.getString(DynamoDbBasedLockConfig.DYNAMODB_ENDPOINT_URL)
+            : DynamoDbClient.serviceMetadata().endpointFor(Region.of(region)).toString();
+
+    if (!endpointURL.startsWith("https://") || !endpointURL.startsWith("http://")) {

Review Comment:
   could the endpoint URL start without the HTTP prefix?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] mansipp commented on a diff in pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

Posted by "mansipp (via GitHub)" <gi...@apache.org>.
mansipp commented on code in PR #9347:
URL: https://github.com/apache/hudi/pull/9347#discussion_r1283877492


##########
hudi-aws/src/main/java/org/apache/hudi/aws/transaction/lock/DynamoDBBasedLockProvider.java:
##########
@@ -153,45 +153,53 @@ public LockItem getLock() {
     return lock;
   }
 
-  private AmazonDynamoDB getDynamoDBClient() {
+  private DynamoDbClient getDynamoDBClient() {
     String region = this.dynamoDBLockConfiguration.getString(DynamoDbBasedLockConfig.DYNAMODB_LOCK_REGION);
     String endpointURL = this.dynamoDBLockConfiguration.contains(DynamoDbBasedLockConfig.DYNAMODB_ENDPOINT_URL.key())
-                          ? this.dynamoDBLockConfiguration.getString(DynamoDbBasedLockConfig.DYNAMODB_ENDPOINT_URL)
-                          : RegionUtils.getRegion(region).getServiceEndpoint(AmazonDynamoDB.ENDPOINT_PREFIX);
-    AwsClientBuilder.EndpointConfiguration dynamodbEndpoint =
-            new AwsClientBuilder.EndpointConfiguration(endpointURL, region);
-    return AmazonDynamoDBClientBuilder.standard()
-            .withEndpointConfiguration(dynamodbEndpoint)
-            .withCredentials(HoodieAWSCredentialsProviderFactory.getAwsCredentialsProvider(dynamoDBLockConfiguration.getProps()))
+            ? this.dynamoDBLockConfiguration.getString(DynamoDbBasedLockConfig.DYNAMODB_ENDPOINT_URL)
+            : DynamoDbClient.serviceMetadata().endpointFor(Region.of(region)).toString();
+
+    if (!endpointURL.startsWith("https://") || !endpointURL.startsWith("http://")) {

Review Comment:
   We are getting this error if we don't specify http:// or https:// in sdk v2. Based on this doc dynamodb supports both http and https so added a check for both. https://docs.aws.amazon.com/general/latest/gr/ddb.html
   ```
   Caused by: java.lang.NullPointerException: The URI scheme of endpointOverride must not be null.
       at org.apache.hudi.software.amazon.awssdk.utils.Validate.paramNotNull(Validate.java:156)
       at org.apache.hudi.software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.endpointOverride(SdkDefaultClientBuilder.java:445)
       at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.getDynamoDBClient(DynamoDBBasedLockProvider.java:163)
       at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:87)
       at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:77)
       ... 65 more
   ``` 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org