You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "voonhous (via GitHub)" <gi...@apache.org> on 2026/09/09 08:22:33 UTC

[PR] test(aws): cover the Glue sync client, DynamoTableUtils and HoodieAWSConfig [hudi]

voonhous opened a new pull request, #19876:
URL: https://github.com/apache/hudi/pull/19876

   ### Describe the issue this Pull Request addresses
   
   `AWSGlueCatalogSyncClient` is the largest coverage gap in hudi-aws (292 of 665 lines uncovered on Codecov, master 56eae7b4), and `DynamoTableUtils` (45 lines) and most of the `HoodieAWSConfig` builder (42 lines) have no tests. Part of the effort to bring repository coverage from 79.31% to 80%.
   
   ### Summary and Changelog
   
   Test-only change, all with Mockito and no AWS access. The Glue client tests assert the request sent to Glue through an `ArgumentCaptor`, not only the return value. One shared fixture had to change: `GlueTestUtil` wrote its commit as raw JSON to `.hoodie/101.commit`, which the 1.x timeline layout never reads, so the fixture table had an empty timeline and no schema; it now writes through `DefaultCommitMetadataSerDe` into `.hoodie/timeline/` and records the schema. All pre-existing tests pass unchanged.
   
   <details>
   <summary>Tests added, by class</summary>
   
   - `TestAWSGlueSyncClient`: 39 new tests on the existing mocked `GlueAsyncClient` fixture. They cover partition-index management end to end (enable flag, stale-index drop, missing-index create, the three-index cap), `updateLastCommitTimeSynced` including the swallowed partition-index failures and the empty-timeline branch, `updateSerdeProperties` (all four exits), `getMetastoreSchema`, `getStorageFieldSchemas`, `getPartitionsFromList`, the endpoint/region branches of `buildAsyncClient` (static-mocked SDK builder), and the failure-wrapping branches of create/drop/exists/update/addPartitions. Each asserts the request sent to Glue through an `ArgumentCaptor`, not only the return value.
   - `TestDynamoTableUtils` (new, 13 tests): `waitUntilExists` / `waitUntilActive` with short explicit timeouts (first-poll success, not-found-then-found, timeout into `TableNeverTransitionedToStateException`, argument guards), `createTableIfNotExists` and `deleteTableIfExists` on both exits.
   - `TestHoodieAWSConfig` (new, 4 tests): every builder method, `build()` defaults, `fromProperties`, `fromFile`.
   - `TestS3StorageLockClient`: two branches of the upsert error handling.
   - `GlueTestUtil`: the fixture wrote its commit as raw JSON to `.hoodie/101.commit`, which the 1.x timeline layout does not read, so the fixture table had an empty timeline and no schema. It now writes through `DefaultCommitMetadataSerDe` into `.hoodie/timeline/` with a completion time and records the table schema (with docs), and gains `createTableWithoutCommits()` for the empty-timeline cases. All pre-existing tests still pass unchanged.
   
   </details>
   
   Left alone on purpose: `updateTableComments` / `setComments` (being fixed in #19488; a test here would pin the behaviour that PR changes), the `AlreadyExistsException` handlers that the async client cannot reach (`CompletableFuture.get()` always wraps them in `ExecutionException`), `S3StorageLockClient`'s real-client constructors (region lookup and `getBucketLocation` need the network), and `AwsGlueCatalogSyncTool.main` (`System.exit`).
   
   ### Impact
   
   No production change. About 10 s of additional test time in the catch-all job.
   
   ### Risk Level
   
   none
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(aws): cover the Glue sync client, DynamoTableUtils and HoodieAWSConfig [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19876:
URL: https://github.com/apache/hudi/pull/19876#issuecomment-5612868594

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4f362f6a7c6b88a24cd8407f6cf3d51d0fad5a70",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16944",
       "triggerID" : "4f362f6a7c6b88a24cd8407f6cf3d51d0fad5a70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb7fccea371553f05d5d66a51d7b2d6711c8e0df",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bb7fccea371553f05d5d66a51d7b2d6711c8e0df",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f362f6a7c6b88a24cd8407f6cf3d51d0fad5a70 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16944) 
   * bb7fccea371553f05d5d66a51d7b2d6711c8e0df UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(aws): cover the Glue sync client, DynamoTableUtils and HoodieAWSConfig [hudi]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19876:
URL: https://github.com/apache/hudi/pull/19876#issuecomment-5602012829

   ## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19876?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 79.32%. Comparing base ([`6ac2790`](https://app.codecov.io/gh/apache/hudi/commit/6ac27904f71b1393a985e7bd417d32fe55e168dc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`4f362f6`](https://app.codecov.io/gh/apache/hudi/commit/4f362f6a7c6b88a24cd8407f6cf3d51d0fad5a70?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 1 commits behind head on master.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff            @@
   ##             master   #19876   +/-   ##
   =========================================
     Coverage     79.31%   79.32%           
   - Complexity    34319    34321    +2     
   =========================================
     Files          2543     2543           
     Lines        142231   142231           
     Branches      17300    17300           
   =========================================
   + Hits         112815   112819    +4     
   + Misses        21614    21612    -2     
   + Partials       7802     7800    -2     
   ```
   
   | [Components](https://app.codecov.io/gh/apache/hudi/pull/19876/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19876/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.80% <ø> (+<0.01%)` | :arrow_up: |
   | [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19876/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.24% <ø> (+<0.01%)` | :arrow_up: |
   | [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19876/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.60% <ø> (ø)` | |
   | [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19876/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.28% <ø> (ø)` | |
   | [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19876/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.54% <ø> (ø)` | |
   | [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19876/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.96% <ø> (ø)` | |
   | [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19876/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.79% <ø> (ø)` | |
   | [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19876/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.95% <ø> (ø)` | |
   | [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19876/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.24% <ø> (ø)` | |
   | [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19876/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.74% <ø> (ø)` | |
   | [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19876/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.81% <ø> (ø)` | |
   | [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19876/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
   
   | [Flag](https://app.codecov.io/gh/apache/hudi/pull/19876/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19876/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.10% <ø> (+<0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   [see 2 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19876/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   </details>
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(aws): cover the Glue sync client, DynamoTableUtils and HoodieAWSConfig [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous merged PR #19876:
URL: https://github.com/apache/hudi/pull/19876


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(aws): cover the Glue sync client, DynamoTableUtils and HoodieAWSConfig [hudi]

Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #19876:
URL: https://github.com/apache/hudi/pull/19876#discussion_r3974925346


##########
hudi-aws/src/test/java/org/apache/hudi/aws/sync/TestAWSGlueSyncClient.java:
##########
@@ -1010,4 +1034,631 @@ void testUpdateTableSchema_issuesOneUpdateTable() {
     verify(mockAwsGlue, times(1)).updateTable(any(UpdateTableRequest.class));
     verify(mockAwsGlue, never()).batchUpdatePartition(any(BatchUpdatePartitionRequest.class));
   }
+
+  @Test
+  void testGetPartitionsFromList_returnsPartitionsKnownToGlue() {
+    String tableName = "tbl";
+    software.amazon.awssdk.services.glue.model.Partition gluePartition =
+        software.amazon.awssdk.services.glue.model.Partition.builder()
+            .values("2024-01-15")
+            .storageDescriptor(StorageDescriptor.builder().location("s3://base/2024/01/15").build())
+            .build();
+    ArgumentCaptor<BatchGetPartitionRequest> captor = ArgumentCaptor.forClass(BatchGetPartitionRequest.class);
+    when(mockAwsGlue.batchGetPartition(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(
+            BatchGetPartitionResponse.builder().partitions(gluePartition).build()));
+
+    List<Partition> result = awsGlueSyncClient.getPartitionsFromList(tableName, Arrays.asList("2024/01/15", "2024/01/16"));
+
+    assertEquals(1, result.size(), "only the partition Glue knows about is returned");
+    assertEquals(Collections.singletonList("2024-01-15"), result.get(0).getValues());
+    assertEquals("s3://base/2024/01/15", result.get(0).getStorageLocation());
+
+    BatchGetPartitionRequest sent = captor.getValue();
+    assertEquals(GlueTestUtil.DB_NAME, sent.databaseName());
+    assertEquals(tableName, sent.tableName());
+    assertEquals(Arrays.asList(Collections.singletonList("2024-01-15"), Collections.singletonList("2024-01-16")),
+        sent.partitionsToGet().stream().map(PartitionValueList::values).collect(Collectors.toList()),
+        "the requested partitions are the extracted partition values, not the storage paths");
+  }
+
+  @Test
+  void testGetPartitionsFromList_emptyListDoesNotCallGlue() {
+    assertTrue(awsGlueSyncClient.getPartitionsFromList("tbl", Collections.emptyList()).isEmpty());
+    verify(mockAwsGlue, never()).batchGetPartition(any(BatchGetPartitionRequest.class));
+  }
+
+  @Test
+  void testGetMetastoreSchema_mergesColumnsAndPartitionKeys() {
+    String tableName = "tbl";
+    List<Column> columns = Arrays.asList(GlueTestUtil.getColumn("name", "string", null),
+        GlueTestUtil.getColumn("age", "int", null));
+    List<Column> partitionKeys = Collections.singletonList(GlueTestUtil.getColumn("datestr", "string", null));
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(getTableWithDefaultProps(tableName, columns, partitionKeys));
+
+    Map<String, String> schema = awsGlueSyncClient.getMetastoreSchema(tableName);
+
+    assertEquals(3, schema.size());
+    assertEquals("STRING", schema.get("name"), "column types are upper cased");
+    assertEquals("INT", schema.get("age"));
+    assertEquals("STRING", schema.get("datestr"), "partition keys are merged into the schema");
+  }
+
+  @Test
+  void testGetMetastoreSchema_wrapsGlueFailure() {
+    when(mockAwsGlue.getTable(any(GetTableRequest.class))).thenThrow(new RuntimeException("boom"));
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class,
+        () -> awsGlueSyncClient.getMetastoreSchema("tbl"));
+    assertTrue(ex.getMessage().contains("Fail to get schema for table"));
+  }
+
+  @Test
+  void testGetLastCommitTimeSynced_readsTableParameters() {
+    Map<String, String> parameters = new HashMap<>();
+    parameters.put(HOODIE_LAST_COMMIT_TIME_SYNC, "100");
+    parameters.put(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC, "110");
+    Table withSyncTimes = tableWithParameters("synced", parameters);
+    Table withoutSyncTimes = tableWithParameters("unsynced", new HashMap<>());
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(withSyncTimes).build()))
+        .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(withoutSyncTimes).build()));
+
+    assertEquals("100", awsGlueSyncClient.getLastCommitTimeSynced("synced").get());
+    assertEquals("110", awsGlueSyncClient.getLastCommitCompletionTimeSynced("synced").get());
+    // the table is cached per name, so the second table name triggers the second stubbed response
+    assertFalse(awsGlueSyncClient.getLastCommitTimeSynced("unsynced").isPresent());
+    assertFalse(awsGlueSyncClient.getLastCommitCompletionTimeSynced("unsynced").isPresent());
+    verify(mockAwsGlue, times(2)).getTable(any(GetTableRequest.class));
+  }
+
+  @Test
+  void testGetStorageFieldSchemas_readsFieldsAndDocsFromStorage() {
+    Map<String, FieldSchema> byName = awsGlueSyncClient.getStorageFieldSchemas().stream()
+        .collect(Collectors.toMap(FieldSchema::getName, f -> f));
+
+    assertEquals("int", byName.get("id").getType());
+    assertEquals(GlueTestUtil.ID_FIELD_DOC, byName.get("id").getComment().get());
+    assertEquals("string", byName.get("name").getType());
+    assertEquals(GlueTestUtil.NAME_FIELD_DOC, byName.get("name").getComment().get());
+    assertTrue(byName.containsKey("_hoodie_commit_time"), "metadata fields are part of the storage schema");
+  }
+
+  @Test
+  void testManagePartitionIndexes_disabledDeactivatesFlagAndDropsIndexes() throws Exception {
+    String tableName = "tbl";
+    Map<String, String> parameters = new HashMap<>();
+    parameters.put(GLUE_PARTITION_INDEX_ENABLE, "true");
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, parameters)).build()));
+    ArgumentCaptor<UpdateTableRequest> updateCaptor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(updateCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(partitionIndexDescriptor("idx_one", "datestr"))
+            .build()));
+    ArgumentCaptor<DeletePartitionIndexRequest> deleteCaptor = ArgumentCaptor.forClass(DeletePartitionIndexRequest.class);
+    when(mockAwsGlue.deletePartitionIndex(deleteCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(DeletePartitionIndexResponse.builder().build()));
+
+    awsGlueSyncClient.managePartitionIndexes(tableName);
+
+    assertEquals("false", updateCaptor.getValue().tableInput().parameters().get(GLUE_PARTITION_INDEX_ENABLE),
+        "partition index usage is deactivated when the feature is off");
+    assertEquals(Collections.singletonList("idx_one"), deleteCaptor.getAllValues().stream()
+        .map(DeletePartitionIndexRequest::indexName).collect(Collectors.toList()));
+    verify(mockAwsGlue, never()).createPartitionIndex(any(CreatePartitionIndexRequest.class));
+  }
+
+  @Test
+  void testManagePartitionIndexes_enabledDropsStaleIndexesAndCreatesMissingOnes() throws Exception {
+    String tableName = "tbl";
+    TypedProperties props = GlueTestUtil.getHiveSyncConfig().getProps();
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS_ENABLE.key(), "true");
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS.key(), "datestr;hour,region");
+    awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, mockSts, new HiveSyncConfig(props), GlueTestUtil.getMetaClient());
+
+    // the table has no partition_filtering.enabled parameter, so indexing has to be activated first
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    ArgumentCaptor<UpdateTableRequest> updateCaptor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(updateCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+
+    PartitionIndexDescriptor keptIndex = partitionIndexDescriptor("kept_idx", "datestr", "hour");
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(keptIndex, partitionIndexDescriptor("stale_idx", "old_col"))
+            .build()))
+        // after a drop the index list is re-read
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(keptIndex)
+            .build()));
+    ArgumentCaptor<DeletePartitionIndexRequest> deleteCaptor = ArgumentCaptor.forClass(DeletePartitionIndexRequest.class);
+    when(mockAwsGlue.deletePartitionIndex(deleteCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(DeletePartitionIndexResponse.builder().build()));
+    ArgumentCaptor<CreatePartitionIndexRequest> createCaptor = ArgumentCaptor.forClass(CreatePartitionIndexRequest.class);
+    when(mockAwsGlue.createPartitionIndex(createCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(CreatePartitionIndexResponse.builder().build()));
+
+    awsGlueSyncClient.managePartitionIndexes(tableName);
+
+    assertEquals("true", updateCaptor.getValue().tableInput().parameters().get(GLUE_PARTITION_INDEX_ENABLE));
+    assertEquals(Collections.singletonList("stale_idx"), deleteCaptor.getAllValues().stream()
+        .map(DeletePartitionIndexRequest::indexName).collect(Collectors.toList()),
+        "only the index that is no longer configured is dropped");
+    assertEquals(1, createCaptor.getAllValues().size(), "the already existing index is not recreated");
+    PartitionIndex created = createCaptor.getValue().partitionIndex();
+    assertEquals(Collections.singletonList("region"), created.keys());
+    assertEquals("hudi_managed_[region]", created.indexName());
+    verify(mockAwsGlue, times(2)).getPartitionIndexes(any(GetPartitionIndexesRequest.class));

Review Comment:
   The enabled-path test starts from a table with no `partition_filtering.enabled` and one stale index, so the steady state every later sync hits - flag already true, index list already matching the config - never runs, and the `indexesChanges` re-read guard stays untested. Worth a third case asserting a single `getPartitionIndexes` and no `updateTable` or `deletePartitionIndex`?



##########
hudi-aws/src/test/java/org/apache/hudi/aws/utils/TestDynamoTableUtils.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.aws.utils;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import software.amazon.awssdk.core.exception.SdkClientException;
+import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
+import software.amazon.awssdk.services.dynamodb.model.CreateTableRequest;
+import software.amazon.awssdk.services.dynamodb.model.CreateTableResponse;
+import software.amazon.awssdk.services.dynamodb.model.DeleteTableRequest;
+import software.amazon.awssdk.services.dynamodb.model.DeleteTableResponse;
+import software.amazon.awssdk.services.dynamodb.model.DescribeTableRequest;
+import software.amazon.awssdk.services.dynamodb.model.DescribeTableResponse;
+import software.amazon.awssdk.services.dynamodb.model.ResourceInUseException;
+import software.amazon.awssdk.services.dynamodb.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.dynamodb.model.TableDescription;
+import software.amazon.awssdk.services.dynamodb.model.TableStatus;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.atLeast;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+/**
+ * Tests {@link DynamoTableUtils} against a mocked DynamoDB client. The polling helpers are always
+ * given an explicit, short timeout so the tests stay fast and deterministic.
+ */
+@ExtendWith(MockitoExtension.class)
+class TestDynamoTableUtils {
+
+  private static final String TABLE_NAME = "lock_table";
+  private static final int TIMEOUT_MS = 300;
+  private static final int INTERVAL_MS = 50;
+
+  @Mock
+  private DynamoDbClient dynamoDb;
+
+  @Test
+  void testWaitUntilExists_returnsOnTheFirstDescription() throws Exception {
+    when(dynamoDb.describeTable(any(DescribeTableRequest.class))).thenReturn(describeResponse(TableStatus.CREATING));
+
+    DynamoTableUtils.waitUntilExists(dynamoDb, TABLE_NAME, TIMEOUT_MS, INTERVAL_MS);
+
+    ArgumentCaptor<DescribeTableRequest> captor = ArgumentCaptor.forClass(DescribeTableRequest.class);
+    verify(dynamoDb, times(1)).describeTable(captor.capture());
+    assertEquals(TABLE_NAME, captor.getValue().tableName(),
+        "any table status is enough to prove the table exists");
+  }
+
+  @Test
+  void testWaitUntilExists_pollsUntilTheTableShowsUp() throws Exception {
+    when(dynamoDb.describeTable(any(DescribeTableRequest.class)))
+        .thenThrow(ResourceNotFoundException.builder().message("not there yet").build())
+        .thenReturn(describeResponse(TableStatus.ACTIVE));
+
+    DynamoTableUtils.waitUntilExists(dynamoDb, TABLE_NAME, TIMEOUT_MS, INTERVAL_MS);
+
+    verify(dynamoDb, times(2)).describeTable(any(DescribeTableRequest.class));
+  }
+
+  @Test
+  void testWaitUntilExists_throwsWhenTheTableNeverShowsUp() {
+    when(dynamoDb.describeTable(any(DescribeTableRequest.class)))
+        .thenThrow(ResourceNotFoundException.builder().message("not there yet").build());
+
+    SdkClientException ex = assertThrows(SdkClientException.class,
+        () -> DynamoTableUtils.waitUntilExists(dynamoDb, TABLE_NAME, TIMEOUT_MS, INTERVAL_MS));
+
+    assertTrue(ex.getMessage().contains(TABLE_NAME + " never returned a result"));
+    verify(dynamoDb, atLeast(2)).describeTable(any(DescribeTableRequest.class));
+  }
+
+  @Test
+  void testWaitUntilActive_returnsWhenTheTableIsActive() throws Exception {
+    when(dynamoDb.describeTable(any(DescribeTableRequest.class)))
+        .thenReturn(describeResponse(TableStatus.CREATING))
+        .thenReturn(describeResponse(TableStatus.ACTIVE));
+
+    DynamoTableUtils.waitUntilActive(dynamoDb, TABLE_NAME, TIMEOUT_MS, INTERVAL_MS);
+
+    verify(dynamoDb, times(2)).describeTable(any(DescribeTableRequest.class));
+  }
+
+  @Test
+  void testWaitUntilActive_throwsWhenTheTableStaysInAnotherState() {
+    when(dynamoDb.describeTable(any(DescribeTableRequest.class))).thenReturn(describeResponse(TableStatus.CREATING));
+
+    DynamoTableUtils.TableNeverTransitionedToStateException ex =
+        assertThrows(DynamoTableUtils.TableNeverTransitionedToStateException.class,
+            () -> DynamoTableUtils.waitUntilActive(dynamoDb, TABLE_NAME, TIMEOUT_MS, INTERVAL_MS));
+
+    assertTrue(ex.getMessage().contains(TABLE_NAME + " never transitioned to desired state of ACTIVE"));
+  }
+
+  @Test
+  void testWaitUntilActive_throwsWhenTheTableNeverAppears() {
+    when(dynamoDb.describeTable(any(DescribeTableRequest.class)))
+        .thenThrow(ResourceNotFoundException.builder().message("not there yet").build());
+
+    assertThrows(DynamoTableUtils.TableNeverTransitionedToStateException.class,
+        () -> DynamoTableUtils.waitUntilActive(dynamoDb, TABLE_NAME, TIMEOUT_MS, INTERVAL_MS),
+        "a table that never gets described is reported the same way as one stuck in another state");
+  }
+
+  @Test
+  void testDefaultTimeoutOverloadsReturnAsSoonAsTheTableIsReady() throws Exception {
+    when(dynamoDb.describeTable(any(DescribeTableRequest.class))).thenReturn(describeResponse(TableStatus.ACTIVE));
+
+    // both overloads poll before sleeping, so an already-ready table returns without waiting
+    DynamoTableUtils.waitUntilExists(dynamoDb, TABLE_NAME);

Review Comment:
   This calls the two-arg overloads, whose defaults are a 20 minute timeout and a 10 second interval, so the class javadoc's "always given an explicit, short timeout" does not hold here and nothing bounds the runtime if the poll and the sleep ever swapped order. Add `@Timeout` to this test and reword the javadoc?



##########
hudi-aws/src/test/java/org/apache/hudi/aws/sync/TestAWSGlueSyncClient.java:
##########
@@ -1010,4 +1034,631 @@ void testUpdateTableSchema_issuesOneUpdateTable() {
     verify(mockAwsGlue, times(1)).updateTable(any(UpdateTableRequest.class));
     verify(mockAwsGlue, never()).batchUpdatePartition(any(BatchUpdatePartitionRequest.class));
   }
+
+  @Test
+  void testGetPartitionsFromList_returnsPartitionsKnownToGlue() {
+    String tableName = "tbl";
+    software.amazon.awssdk.services.glue.model.Partition gluePartition =
+        software.amazon.awssdk.services.glue.model.Partition.builder()
+            .values("2024-01-15")
+            .storageDescriptor(StorageDescriptor.builder().location("s3://base/2024/01/15").build())
+            .build();
+    ArgumentCaptor<BatchGetPartitionRequest> captor = ArgumentCaptor.forClass(BatchGetPartitionRequest.class);
+    when(mockAwsGlue.batchGetPartition(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(
+            BatchGetPartitionResponse.builder().partitions(gluePartition).build()));
+
+    List<Partition> result = awsGlueSyncClient.getPartitionsFromList(tableName, Arrays.asList("2024/01/15", "2024/01/16"));
+
+    assertEquals(1, result.size(), "only the partition Glue knows about is returned");
+    assertEquals(Collections.singletonList("2024-01-15"), result.get(0).getValues());
+    assertEquals("s3://base/2024/01/15", result.get(0).getStorageLocation());
+
+    BatchGetPartitionRequest sent = captor.getValue();
+    assertEquals(GlueTestUtil.DB_NAME, sent.databaseName());
+    assertEquals(tableName, sent.tableName());
+    assertEquals(Arrays.asList(Collections.singletonList("2024-01-15"), Collections.singletonList("2024-01-16")),
+        sent.partitionsToGet().stream().map(PartitionValueList::values).collect(Collectors.toList()),
+        "the requested partitions are the extracted partition values, not the storage paths");
+  }
+
+  @Test
+  void testGetPartitionsFromList_emptyListDoesNotCallGlue() {
+    assertTrue(awsGlueSyncClient.getPartitionsFromList("tbl", Collections.emptyList()).isEmpty());
+    verify(mockAwsGlue, never()).batchGetPartition(any(BatchGetPartitionRequest.class));
+  }
+
+  @Test
+  void testGetMetastoreSchema_mergesColumnsAndPartitionKeys() {
+    String tableName = "tbl";
+    List<Column> columns = Arrays.asList(GlueTestUtil.getColumn("name", "string", null),
+        GlueTestUtil.getColumn("age", "int", null));
+    List<Column> partitionKeys = Collections.singletonList(GlueTestUtil.getColumn("datestr", "string", null));
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(getTableWithDefaultProps(tableName, columns, partitionKeys));
+
+    Map<String, String> schema = awsGlueSyncClient.getMetastoreSchema(tableName);
+
+    assertEquals(3, schema.size());
+    assertEquals("STRING", schema.get("name"), "column types are upper cased");
+    assertEquals("INT", schema.get("age"));
+    assertEquals("STRING", schema.get("datestr"), "partition keys are merged into the schema");
+  }
+
+  @Test
+  void testGetMetastoreSchema_wrapsGlueFailure() {
+    when(mockAwsGlue.getTable(any(GetTableRequest.class))).thenThrow(new RuntimeException("boom"));
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class,
+        () -> awsGlueSyncClient.getMetastoreSchema("tbl"));
+    assertTrue(ex.getMessage().contains("Fail to get schema for table"));
+  }
+
+  @Test
+  void testGetLastCommitTimeSynced_readsTableParameters() {
+    Map<String, String> parameters = new HashMap<>();
+    parameters.put(HOODIE_LAST_COMMIT_TIME_SYNC, "100");
+    parameters.put(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC, "110");
+    Table withSyncTimes = tableWithParameters("synced", parameters);
+    Table withoutSyncTimes = tableWithParameters("unsynced", new HashMap<>());
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(withSyncTimes).build()))
+        .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(withoutSyncTimes).build()));
+
+    assertEquals("100", awsGlueSyncClient.getLastCommitTimeSynced("synced").get());
+    assertEquals("110", awsGlueSyncClient.getLastCommitCompletionTimeSynced("synced").get());
+    // the table is cached per name, so the second table name triggers the second stubbed response
+    assertFalse(awsGlueSyncClient.getLastCommitTimeSynced("unsynced").isPresent());
+    assertFalse(awsGlueSyncClient.getLastCommitCompletionTimeSynced("unsynced").isPresent());
+    verify(mockAwsGlue, times(2)).getTable(any(GetTableRequest.class));
+  }
+
+  @Test
+  void testGetStorageFieldSchemas_readsFieldsAndDocsFromStorage() {
+    Map<String, FieldSchema> byName = awsGlueSyncClient.getStorageFieldSchemas().stream()
+        .collect(Collectors.toMap(FieldSchema::getName, f -> f));
+
+    assertEquals("int", byName.get("id").getType());
+    assertEquals(GlueTestUtil.ID_FIELD_DOC, byName.get("id").getComment().get());
+    assertEquals("string", byName.get("name").getType());
+    assertEquals(GlueTestUtil.NAME_FIELD_DOC, byName.get("name").getComment().get());
+    assertTrue(byName.containsKey("_hoodie_commit_time"), "metadata fields are part of the storage schema");
+  }
+
+  @Test
+  void testManagePartitionIndexes_disabledDeactivatesFlagAndDropsIndexes() throws Exception {
+    String tableName = "tbl";
+    Map<String, String> parameters = new HashMap<>();
+    parameters.put(GLUE_PARTITION_INDEX_ENABLE, "true");
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, parameters)).build()));
+    ArgumentCaptor<UpdateTableRequest> updateCaptor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(updateCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(partitionIndexDescriptor("idx_one", "datestr"))
+            .build()));
+    ArgumentCaptor<DeletePartitionIndexRequest> deleteCaptor = ArgumentCaptor.forClass(DeletePartitionIndexRequest.class);
+    when(mockAwsGlue.deletePartitionIndex(deleteCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(DeletePartitionIndexResponse.builder().build()));
+
+    awsGlueSyncClient.managePartitionIndexes(tableName);
+
+    assertEquals("false", updateCaptor.getValue().tableInput().parameters().get(GLUE_PARTITION_INDEX_ENABLE),
+        "partition index usage is deactivated when the feature is off");
+    assertEquals(Collections.singletonList("idx_one"), deleteCaptor.getAllValues().stream()
+        .map(DeletePartitionIndexRequest::indexName).collect(Collectors.toList()));
+    verify(mockAwsGlue, never()).createPartitionIndex(any(CreatePartitionIndexRequest.class));
+  }
+
+  @Test
+  void testManagePartitionIndexes_enabledDropsStaleIndexesAndCreatesMissingOnes() throws Exception {
+    String tableName = "tbl";
+    TypedProperties props = GlueTestUtil.getHiveSyncConfig().getProps();
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS_ENABLE.key(), "true");
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS.key(), "datestr;hour,region");
+    awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, mockSts, new HiveSyncConfig(props), GlueTestUtil.getMetaClient());
+
+    // the table has no partition_filtering.enabled parameter, so indexing has to be activated first
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    ArgumentCaptor<UpdateTableRequest> updateCaptor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(updateCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+
+    PartitionIndexDescriptor keptIndex = partitionIndexDescriptor("kept_idx", "datestr", "hour");
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(keptIndex, partitionIndexDescriptor("stale_idx", "old_col"))
+            .build()))
+        // after a drop the index list is re-read
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(keptIndex)
+            .build()));
+    ArgumentCaptor<DeletePartitionIndexRequest> deleteCaptor = ArgumentCaptor.forClass(DeletePartitionIndexRequest.class);
+    when(mockAwsGlue.deletePartitionIndex(deleteCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(DeletePartitionIndexResponse.builder().build()));
+    ArgumentCaptor<CreatePartitionIndexRequest> createCaptor = ArgumentCaptor.forClass(CreatePartitionIndexRequest.class);
+    when(mockAwsGlue.createPartitionIndex(createCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(CreatePartitionIndexResponse.builder().build()));
+
+    awsGlueSyncClient.managePartitionIndexes(tableName);
+
+    assertEquals("true", updateCaptor.getValue().tableInput().parameters().get(GLUE_PARTITION_INDEX_ENABLE));
+    assertEquals(Collections.singletonList("stale_idx"), deleteCaptor.getAllValues().stream()
+        .map(DeletePartitionIndexRequest::indexName).collect(Collectors.toList()),
+        "only the index that is no longer configured is dropped");
+    assertEquals(1, createCaptor.getAllValues().size(), "the already existing index is not recreated");
+    PartitionIndex created = createCaptor.getValue().partitionIndex();
+    assertEquals(Collections.singletonList("region"), created.keys());
+    assertEquals("hudi_managed_[region]", created.indexName());
+    verify(mockAwsGlue, times(2)).getPartitionIndexes(any(GetPartitionIndexesRequest.class));
+  }
+
+  @Test
+  void testParsePartitionsIndexConfig_keepsOnlyTheFirstThreeIndexes() {
+    TypedProperties props = GlueTestUtil.getHiveSyncConfig().getProps();
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS.key(), "a;b,c,d,e");
+    awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, mockSts, new HiveSyncConfig(props), GlueTestUtil.getMetaClient());
+
+    assertEquals(Arrays.asList(Arrays.asList("a", "b"), Collections.singletonList("c"), Collections.singletonList("d")),
+        awsGlueSyncClient.parsePartitionsIndexConfig(), "glue supports at most three partition indexes");
+  }
+
+  @Test
+  void testUpdateLastCommitTimeSynced_writesTimelineInstantToTableParameters() {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    ArgumentCaptor<UpdateTableRequest> captor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder().build()));
+
+    awsGlueSyncClient.updateLastCommitTimeSynced(tableName);
+
+    Map<String, String> parameters = captor.getValue().tableInput().parameters();
+    assertEquals(GlueTestUtil.INSTANT_TIME, parameters.get(HOODIE_LAST_COMMIT_TIME_SYNC),
+        "the last instant of the active timeline is synced");
+    assertEquals(GlueTestUtil.COMPLETION_TIME, parameters.get(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC),
+        "the completion time of that instant is synced alongside it");
+    assertTrue(captor.getValue().skipArchive(), "table archiving is skipped by default");
+  }
+
+  /**
+   * An indexation already in flight surfaces as an {@link ExecutionException}, anything else lands in the
+   * catch-all. Neither may fail the commit-time sync.
+   */
+  @ParameterizedTest
+  @ValueSource(booleans = {true, false})
+  void testUpdateLastCommitTimeSynced_partitionIndexFailureDoesNotFailTheSync(boolean asExecutionFailure) throws Exception {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    when(mockAwsGlue.updateTable(any(UpdateTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    if (asExecutionFailure) {
+      CompletableFuture<GetPartitionIndexesResponse> failed = mock(CompletableFuture.class);
+      when(failed.get()).thenThrow(new ExecutionException(new RuntimeException("indexing in progress")));
+      when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class))).thenReturn(failed);
+    } else {
+      when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class))).thenThrow(new RuntimeException("boom"));
+    }
+
+    awsGlueSyncClient.updateLastCommitTimeSynced(tableName);
+
+    verify(mockAwsGlue, times(1)).updateTable(any(UpdateTableRequest.class));

Review Comment:
   Deleting the `ExecutionException` arm this parameterization targets just drops the failure into the catch-all below it, which swallows identically, so both values of `asExecutionFailure` pass on this one assertion. Is the `true` case meant to pin that arm, or is it here as scenario documentation?



##########
hudi-aws/src/test/java/org/apache/hudi/aws/sync/TestAWSGlueSyncClient.java:
##########
@@ -1010,4 +1034,631 @@ void testUpdateTableSchema_issuesOneUpdateTable() {
     verify(mockAwsGlue, times(1)).updateTable(any(UpdateTableRequest.class));
     verify(mockAwsGlue, never()).batchUpdatePartition(any(BatchUpdatePartitionRequest.class));
   }
+
+  @Test
+  void testGetPartitionsFromList_returnsPartitionsKnownToGlue() {
+    String tableName = "tbl";
+    software.amazon.awssdk.services.glue.model.Partition gluePartition =
+        software.amazon.awssdk.services.glue.model.Partition.builder()
+            .values("2024-01-15")
+            .storageDescriptor(StorageDescriptor.builder().location("s3://base/2024/01/15").build())
+            .build();
+    ArgumentCaptor<BatchGetPartitionRequest> captor = ArgumentCaptor.forClass(BatchGetPartitionRequest.class);
+    when(mockAwsGlue.batchGetPartition(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(
+            BatchGetPartitionResponse.builder().partitions(gluePartition).build()));
+
+    List<Partition> result = awsGlueSyncClient.getPartitionsFromList(tableName, Arrays.asList("2024/01/15", "2024/01/16"));
+
+    assertEquals(1, result.size(), "only the partition Glue knows about is returned");
+    assertEquals(Collections.singletonList("2024-01-15"), result.get(0).getValues());
+    assertEquals("s3://base/2024/01/15", result.get(0).getStorageLocation());
+
+    BatchGetPartitionRequest sent = captor.getValue();
+    assertEquals(GlueTestUtil.DB_NAME, sent.databaseName());
+    assertEquals(tableName, sent.tableName());
+    assertEquals(Arrays.asList(Collections.singletonList("2024-01-15"), Collections.singletonList("2024-01-16")),
+        sent.partitionsToGet().stream().map(PartitionValueList::values).collect(Collectors.toList()),
+        "the requested partitions are the extracted partition values, not the storage paths");
+  }
+
+  @Test
+  void testGetPartitionsFromList_emptyListDoesNotCallGlue() {
+    assertTrue(awsGlueSyncClient.getPartitionsFromList("tbl", Collections.emptyList()).isEmpty());
+    verify(mockAwsGlue, never()).batchGetPartition(any(BatchGetPartitionRequest.class));
+  }
+
+  @Test
+  void testGetMetastoreSchema_mergesColumnsAndPartitionKeys() {
+    String tableName = "tbl";
+    List<Column> columns = Arrays.asList(GlueTestUtil.getColumn("name", "string", null),
+        GlueTestUtil.getColumn("age", "int", null));
+    List<Column> partitionKeys = Collections.singletonList(GlueTestUtil.getColumn("datestr", "string", null));
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(getTableWithDefaultProps(tableName, columns, partitionKeys));
+
+    Map<String, String> schema = awsGlueSyncClient.getMetastoreSchema(tableName);
+
+    assertEquals(3, schema.size());
+    assertEquals("STRING", schema.get("name"), "column types are upper cased");
+    assertEquals("INT", schema.get("age"));
+    assertEquals("STRING", schema.get("datestr"), "partition keys are merged into the schema");
+  }
+
+  @Test
+  void testGetMetastoreSchema_wrapsGlueFailure() {
+    when(mockAwsGlue.getTable(any(GetTableRequest.class))).thenThrow(new RuntimeException("boom"));
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class,
+        () -> awsGlueSyncClient.getMetastoreSchema("tbl"));
+    assertTrue(ex.getMessage().contains("Fail to get schema for table"));
+  }
+
+  @Test
+  void testGetLastCommitTimeSynced_readsTableParameters() {
+    Map<String, String> parameters = new HashMap<>();
+    parameters.put(HOODIE_LAST_COMMIT_TIME_SYNC, "100");
+    parameters.put(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC, "110");
+    Table withSyncTimes = tableWithParameters("synced", parameters);
+    Table withoutSyncTimes = tableWithParameters("unsynced", new HashMap<>());
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(withSyncTimes).build()))
+        .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(withoutSyncTimes).build()));
+
+    assertEquals("100", awsGlueSyncClient.getLastCommitTimeSynced("synced").get());
+    assertEquals("110", awsGlueSyncClient.getLastCommitCompletionTimeSynced("synced").get());
+    // the table is cached per name, so the second table name triggers the second stubbed response
+    assertFalse(awsGlueSyncClient.getLastCommitTimeSynced("unsynced").isPresent());
+    assertFalse(awsGlueSyncClient.getLastCommitCompletionTimeSynced("unsynced").isPresent());
+    verify(mockAwsGlue, times(2)).getTable(any(GetTableRequest.class));
+  }
+
+  @Test
+  void testGetStorageFieldSchemas_readsFieldsAndDocsFromStorage() {
+    Map<String, FieldSchema> byName = awsGlueSyncClient.getStorageFieldSchemas().stream()
+        .collect(Collectors.toMap(FieldSchema::getName, f -> f));
+
+    assertEquals("int", byName.get("id").getType());
+    assertEquals(GlueTestUtil.ID_FIELD_DOC, byName.get("id").getComment().get());
+    assertEquals("string", byName.get("name").getType());
+    assertEquals(GlueTestUtil.NAME_FIELD_DOC, byName.get("name").getComment().get());
+    assertTrue(byName.containsKey("_hoodie_commit_time"), "metadata fields are part of the storage schema");
+  }
+
+  @Test
+  void testManagePartitionIndexes_disabledDeactivatesFlagAndDropsIndexes() throws Exception {
+    String tableName = "tbl";
+    Map<String, String> parameters = new HashMap<>();
+    parameters.put(GLUE_PARTITION_INDEX_ENABLE, "true");
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, parameters)).build()));
+    ArgumentCaptor<UpdateTableRequest> updateCaptor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(updateCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(partitionIndexDescriptor("idx_one", "datestr"))
+            .build()));
+    ArgumentCaptor<DeletePartitionIndexRequest> deleteCaptor = ArgumentCaptor.forClass(DeletePartitionIndexRequest.class);
+    when(mockAwsGlue.deletePartitionIndex(deleteCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(DeletePartitionIndexResponse.builder().build()));
+
+    awsGlueSyncClient.managePartitionIndexes(tableName);
+
+    assertEquals("false", updateCaptor.getValue().tableInput().parameters().get(GLUE_PARTITION_INDEX_ENABLE),
+        "partition index usage is deactivated when the feature is off");
+    assertEquals(Collections.singletonList("idx_one"), deleteCaptor.getAllValues().stream()
+        .map(DeletePartitionIndexRequest::indexName).collect(Collectors.toList()));
+    verify(mockAwsGlue, never()).createPartitionIndex(any(CreatePartitionIndexRequest.class));
+  }
+
+  @Test
+  void testManagePartitionIndexes_enabledDropsStaleIndexesAndCreatesMissingOnes() throws Exception {
+    String tableName = "tbl";
+    TypedProperties props = GlueTestUtil.getHiveSyncConfig().getProps();
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS_ENABLE.key(), "true");
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS.key(), "datestr;hour,region");
+    awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, mockSts, new HiveSyncConfig(props), GlueTestUtil.getMetaClient());
+
+    // the table has no partition_filtering.enabled parameter, so indexing has to be activated first
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    ArgumentCaptor<UpdateTableRequest> updateCaptor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(updateCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+
+    PartitionIndexDescriptor keptIndex = partitionIndexDescriptor("kept_idx", "datestr", "hour");
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(keptIndex, partitionIndexDescriptor("stale_idx", "old_col"))
+            .build()))
+        // after a drop the index list is re-read
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(keptIndex)
+            .build()));
+    ArgumentCaptor<DeletePartitionIndexRequest> deleteCaptor = ArgumentCaptor.forClass(DeletePartitionIndexRequest.class);
+    when(mockAwsGlue.deletePartitionIndex(deleteCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(DeletePartitionIndexResponse.builder().build()));
+    ArgumentCaptor<CreatePartitionIndexRequest> createCaptor = ArgumentCaptor.forClass(CreatePartitionIndexRequest.class);
+    when(mockAwsGlue.createPartitionIndex(createCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(CreatePartitionIndexResponse.builder().build()));
+
+    awsGlueSyncClient.managePartitionIndexes(tableName);
+
+    assertEquals("true", updateCaptor.getValue().tableInput().parameters().get(GLUE_PARTITION_INDEX_ENABLE));
+    assertEquals(Collections.singletonList("stale_idx"), deleteCaptor.getAllValues().stream()
+        .map(DeletePartitionIndexRequest::indexName).collect(Collectors.toList()),
+        "only the index that is no longer configured is dropped");
+    assertEquals(1, createCaptor.getAllValues().size(), "the already existing index is not recreated");
+    PartitionIndex created = createCaptor.getValue().partitionIndex();
+    assertEquals(Collections.singletonList("region"), created.keys());
+    assertEquals("hudi_managed_[region]", created.indexName());
+    verify(mockAwsGlue, times(2)).getPartitionIndexes(any(GetPartitionIndexesRequest.class));
+  }
+
+  @Test
+  void testParsePartitionsIndexConfig_keepsOnlyTheFirstThreeIndexes() {
+    TypedProperties props = GlueTestUtil.getHiveSyncConfig().getProps();
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS.key(), "a;b,c,d,e");
+    awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, mockSts, new HiveSyncConfig(props), GlueTestUtil.getMetaClient());
+
+    assertEquals(Arrays.asList(Arrays.asList("a", "b"), Collections.singletonList("c"), Collections.singletonList("d")),
+        awsGlueSyncClient.parsePartitionsIndexConfig(), "glue supports at most three partition indexes");
+  }
+
+  @Test
+  void testUpdateLastCommitTimeSynced_writesTimelineInstantToTableParameters() {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    ArgumentCaptor<UpdateTableRequest> captor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder().build()));
+
+    awsGlueSyncClient.updateLastCommitTimeSynced(tableName);
+
+    Map<String, String> parameters = captor.getValue().tableInput().parameters();
+    assertEquals(GlueTestUtil.INSTANT_TIME, parameters.get(HOODIE_LAST_COMMIT_TIME_SYNC),
+        "the last instant of the active timeline is synced");
+    assertEquals(GlueTestUtil.COMPLETION_TIME, parameters.get(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC),
+        "the completion time of that instant is synced alongside it");
+    assertTrue(captor.getValue().skipArchive(), "table archiving is skipped by default");
+  }
+
+  /**
+   * An indexation already in flight surfaces as an {@link ExecutionException}, anything else lands in the
+   * catch-all. Neither may fail the commit-time sync.
+   */
+  @ParameterizedTest
+  @ValueSource(booleans = {true, false})
+  void testUpdateLastCommitTimeSynced_partitionIndexFailureDoesNotFailTheSync(boolean asExecutionFailure) throws Exception {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    when(mockAwsGlue.updateTable(any(UpdateTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    if (asExecutionFailure) {
+      CompletableFuture<GetPartitionIndexesResponse> failed = mock(CompletableFuture.class);
+      when(failed.get()).thenThrow(new ExecutionException(new RuntimeException("indexing in progress")));
+      when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class))).thenReturn(failed);
+    } else {
+      when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class))).thenThrow(new RuntimeException("boom"));
+    }
+
+    awsGlueSyncClient.updateLastCommitTimeSynced(tableName);
+
+    verify(mockAwsGlue, times(1)).updateTable(any(UpdateTableRequest.class));
+  }
+
+  @Test
+  void testUpdateLastCommitTimeSynced_wrapsGlueFailure() {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    when(mockAwsGlue.updateTable(any(UpdateTableRequest.class))).thenThrow(new RuntimeException("boom"));
+
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class,
+        () -> awsGlueSyncClient.updateLastCommitTimeSynced(tableName));
+    assertTrue(ex.getMessage().contains("Fail to update last sync commit time"));
+  }
+
+  @Test
+  void testUpdateSerdeProperties_emptyPropertiesSkipUpdate() {
+    assertFalse(awsGlueSyncClient.updateSerdeProperties("tbl", Collections.emptyMap(), false));
+    verify(mockAwsGlue, never()).updateTable(any(UpdateTableRequest.class));
+  }
+
+  @Test
+  void testUpdateSerdeProperties_unchangedPropertiesSkipUpdate() {
+    String tableName = "tbl";
+    Map<String, String> serdeProperties = new HashMap<>();
+    serdeProperties.put("serialization.format", "1");
+    serdeProperties.put("path", "s3://base");
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithSerdeProperties(tableName, serdeProperties)).build()));
+
+    assertFalse(awsGlueSyncClient.updateSerdeProperties(tableName, new HashMap<>(serdeProperties), false));
+    verify(mockAwsGlue, never()).updateTable(any(UpdateTableRequest.class));
+  }
+
+  @Test
+  void testUpdateSerdeProperties_changedPropertiesRewriteSerdeInfo() {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithSerdeProperties(tableName,
+                serdePropertiesOf("serialization.format", "1", "location", "s3://old"))).build()));
+    ArgumentCaptor<UpdateTableRequest> captor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+
+    Map<String, String> serdeProperties = new HashMap<>();
+    serdeProperties.put("path", "s3://new");
+    assertTrue(awsGlueSyncClient.updateSerdeProperties(tableName, serdeProperties, false));

Review Comment:
   All four new `updateSerdeProperties` tests pass `false` for `useRealtimeFormat`, and `AWSGlueCatalogSyncClient` never reads that parameter, while `HiveSyncTool` passes `true` when it syncs the MOR `_rt` table. Is the flag meant to be a no-op for Glue, or is a `true` case worth adding here?



##########
hudi-aws/src/test/java/org/apache/hudi/aws/sync/TestAWSGlueSyncClient.java:
##########
@@ -1010,4 +1034,631 @@ void testUpdateTableSchema_issuesOneUpdateTable() {
     verify(mockAwsGlue, times(1)).updateTable(any(UpdateTableRequest.class));
     verify(mockAwsGlue, never()).batchUpdatePartition(any(BatchUpdatePartitionRequest.class));
   }
+
+  @Test
+  void testGetPartitionsFromList_returnsPartitionsKnownToGlue() {
+    String tableName = "tbl";
+    software.amazon.awssdk.services.glue.model.Partition gluePartition =
+        software.amazon.awssdk.services.glue.model.Partition.builder()
+            .values("2024-01-15")
+            .storageDescriptor(StorageDescriptor.builder().location("s3://base/2024/01/15").build())
+            .build();
+    ArgumentCaptor<BatchGetPartitionRequest> captor = ArgumentCaptor.forClass(BatchGetPartitionRequest.class);
+    when(mockAwsGlue.batchGetPartition(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(
+            BatchGetPartitionResponse.builder().partitions(gluePartition).build()));
+
+    List<Partition> result = awsGlueSyncClient.getPartitionsFromList(tableName, Arrays.asList("2024/01/15", "2024/01/16"));
+
+    assertEquals(1, result.size(), "only the partition Glue knows about is returned");
+    assertEquals(Collections.singletonList("2024-01-15"), result.get(0).getValues());
+    assertEquals("s3://base/2024/01/15", result.get(0).getStorageLocation());
+
+    BatchGetPartitionRequest sent = captor.getValue();
+    assertEquals(GlueTestUtil.DB_NAME, sent.databaseName());
+    assertEquals(tableName, sent.tableName());
+    assertEquals(Arrays.asList(Collections.singletonList("2024-01-15"), Collections.singletonList("2024-01-16")),
+        sent.partitionsToGet().stream().map(PartitionValueList::values).collect(Collectors.toList()),
+        "the requested partitions are the extracted partition values, not the storage paths");
+  }
+
+  @Test
+  void testGetPartitionsFromList_emptyListDoesNotCallGlue() {
+    assertTrue(awsGlueSyncClient.getPartitionsFromList("tbl", Collections.emptyList()).isEmpty());
+    verify(mockAwsGlue, never()).batchGetPartition(any(BatchGetPartitionRequest.class));
+  }
+
+  @Test
+  void testGetMetastoreSchema_mergesColumnsAndPartitionKeys() {
+    String tableName = "tbl";
+    List<Column> columns = Arrays.asList(GlueTestUtil.getColumn("name", "string", null),
+        GlueTestUtil.getColumn("age", "int", null));
+    List<Column> partitionKeys = Collections.singletonList(GlueTestUtil.getColumn("datestr", "string", null));
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(getTableWithDefaultProps(tableName, columns, partitionKeys));
+
+    Map<String, String> schema = awsGlueSyncClient.getMetastoreSchema(tableName);
+
+    assertEquals(3, schema.size());
+    assertEquals("STRING", schema.get("name"), "column types are upper cased");
+    assertEquals("INT", schema.get("age"));
+    assertEquals("STRING", schema.get("datestr"), "partition keys are merged into the schema");
+  }
+
+  @Test
+  void testGetMetastoreSchema_wrapsGlueFailure() {
+    when(mockAwsGlue.getTable(any(GetTableRequest.class))).thenThrow(new RuntimeException("boom"));
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class,
+        () -> awsGlueSyncClient.getMetastoreSchema("tbl"));
+    assertTrue(ex.getMessage().contains("Fail to get schema for table"));
+  }
+
+  @Test
+  void testGetLastCommitTimeSynced_readsTableParameters() {
+    Map<String, String> parameters = new HashMap<>();
+    parameters.put(HOODIE_LAST_COMMIT_TIME_SYNC, "100");
+    parameters.put(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC, "110");
+    Table withSyncTimes = tableWithParameters("synced", parameters);
+    Table withoutSyncTimes = tableWithParameters("unsynced", new HashMap<>());
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(withSyncTimes).build()))
+        .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(withoutSyncTimes).build()));
+
+    assertEquals("100", awsGlueSyncClient.getLastCommitTimeSynced("synced").get());
+    assertEquals("110", awsGlueSyncClient.getLastCommitCompletionTimeSynced("synced").get());
+    // the table is cached per name, so the second table name triggers the second stubbed response
+    assertFalse(awsGlueSyncClient.getLastCommitTimeSynced("unsynced").isPresent());
+    assertFalse(awsGlueSyncClient.getLastCommitCompletionTimeSynced("unsynced").isPresent());
+    verify(mockAwsGlue, times(2)).getTable(any(GetTableRequest.class));
+  }
+
+  @Test
+  void testGetStorageFieldSchemas_readsFieldsAndDocsFromStorage() {
+    Map<String, FieldSchema> byName = awsGlueSyncClient.getStorageFieldSchemas().stream()
+        .collect(Collectors.toMap(FieldSchema::getName, f -> f));
+
+    assertEquals("int", byName.get("id").getType());
+    assertEquals(GlueTestUtil.ID_FIELD_DOC, byName.get("id").getComment().get());
+    assertEquals("string", byName.get("name").getType());
+    assertEquals(GlueTestUtil.NAME_FIELD_DOC, byName.get("name").getComment().get());
+    assertTrue(byName.containsKey("_hoodie_commit_time"), "metadata fields are part of the storage schema");
+  }
+
+  @Test
+  void testManagePartitionIndexes_disabledDeactivatesFlagAndDropsIndexes() throws Exception {
+    String tableName = "tbl";
+    Map<String, String> parameters = new HashMap<>();
+    parameters.put(GLUE_PARTITION_INDEX_ENABLE, "true");
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, parameters)).build()));
+    ArgumentCaptor<UpdateTableRequest> updateCaptor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(updateCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(partitionIndexDescriptor("idx_one", "datestr"))
+            .build()));
+    ArgumentCaptor<DeletePartitionIndexRequest> deleteCaptor = ArgumentCaptor.forClass(DeletePartitionIndexRequest.class);
+    when(mockAwsGlue.deletePartitionIndex(deleteCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(DeletePartitionIndexResponse.builder().build()));
+
+    awsGlueSyncClient.managePartitionIndexes(tableName);
+
+    assertEquals("false", updateCaptor.getValue().tableInput().parameters().get(GLUE_PARTITION_INDEX_ENABLE),
+        "partition index usage is deactivated when the feature is off");
+    assertEquals(Collections.singletonList("idx_one"), deleteCaptor.getAllValues().stream()
+        .map(DeletePartitionIndexRequest::indexName).collect(Collectors.toList()));
+    verify(mockAwsGlue, never()).createPartitionIndex(any(CreatePartitionIndexRequest.class));
+  }
+
+  @Test
+  void testManagePartitionIndexes_enabledDropsStaleIndexesAndCreatesMissingOnes() throws Exception {
+    String tableName = "tbl";
+    TypedProperties props = GlueTestUtil.getHiveSyncConfig().getProps();
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS_ENABLE.key(), "true");
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS.key(), "datestr;hour,region");
+    awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, mockSts, new HiveSyncConfig(props), GlueTestUtil.getMetaClient());
+
+    // the table has no partition_filtering.enabled parameter, so indexing has to be activated first
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    ArgumentCaptor<UpdateTableRequest> updateCaptor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(updateCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+
+    PartitionIndexDescriptor keptIndex = partitionIndexDescriptor("kept_idx", "datestr", "hour");
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(keptIndex, partitionIndexDescriptor("stale_idx", "old_col"))
+            .build()))
+        // after a drop the index list is re-read
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(keptIndex)
+            .build()));
+    ArgumentCaptor<DeletePartitionIndexRequest> deleteCaptor = ArgumentCaptor.forClass(DeletePartitionIndexRequest.class);
+    when(mockAwsGlue.deletePartitionIndex(deleteCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(DeletePartitionIndexResponse.builder().build()));
+    ArgumentCaptor<CreatePartitionIndexRequest> createCaptor = ArgumentCaptor.forClass(CreatePartitionIndexRequest.class);
+    when(mockAwsGlue.createPartitionIndex(createCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(CreatePartitionIndexResponse.builder().build()));
+
+    awsGlueSyncClient.managePartitionIndexes(tableName);
+
+    assertEquals("true", updateCaptor.getValue().tableInput().parameters().get(GLUE_PARTITION_INDEX_ENABLE));
+    assertEquals(Collections.singletonList("stale_idx"), deleteCaptor.getAllValues().stream()
+        .map(DeletePartitionIndexRequest::indexName).collect(Collectors.toList()),
+        "only the index that is no longer configured is dropped");
+    assertEquals(1, createCaptor.getAllValues().size(), "the already existing index is not recreated");
+    PartitionIndex created = createCaptor.getValue().partitionIndex();
+    assertEquals(Collections.singletonList("region"), created.keys());
+    assertEquals("hudi_managed_[region]", created.indexName());
+    verify(mockAwsGlue, times(2)).getPartitionIndexes(any(GetPartitionIndexesRequest.class));
+  }
+
+  @Test
+  void testParsePartitionsIndexConfig_keepsOnlyTheFirstThreeIndexes() {
+    TypedProperties props = GlueTestUtil.getHiveSyncConfig().getProps();
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS.key(), "a;b,c,d,e");
+    awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, mockSts, new HiveSyncConfig(props), GlueTestUtil.getMetaClient());
+
+    assertEquals(Arrays.asList(Arrays.asList("a", "b"), Collections.singletonList("c"), Collections.singletonList("d")),
+        awsGlueSyncClient.parsePartitionsIndexConfig(), "glue supports at most three partition indexes");
+  }
+
+  @Test
+  void testUpdateLastCommitTimeSynced_writesTimelineInstantToTableParameters() {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    ArgumentCaptor<UpdateTableRequest> captor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder().build()));
+
+    awsGlueSyncClient.updateLastCommitTimeSynced(tableName);
+
+    Map<String, String> parameters = captor.getValue().tableInput().parameters();
+    assertEquals(GlueTestUtil.INSTANT_TIME, parameters.get(HOODIE_LAST_COMMIT_TIME_SYNC),
+        "the last instant of the active timeline is synced");
+    assertEquals(GlueTestUtil.COMPLETION_TIME, parameters.get(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC),
+        "the completion time of that instant is synced alongside it");
+    assertTrue(captor.getValue().skipArchive(), "table archiving is skipped by default");
+  }
+
+  /**
+   * An indexation already in flight surfaces as an {@link ExecutionException}, anything else lands in the
+   * catch-all. Neither may fail the commit-time sync.
+   */
+  @ParameterizedTest
+  @ValueSource(booleans = {true, false})
+  void testUpdateLastCommitTimeSynced_partitionIndexFailureDoesNotFailTheSync(boolean asExecutionFailure) throws Exception {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    when(mockAwsGlue.updateTable(any(UpdateTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    if (asExecutionFailure) {
+      CompletableFuture<GetPartitionIndexesResponse> failed = mock(CompletableFuture.class);
+      when(failed.get()).thenThrow(new ExecutionException(new RuntimeException("indexing in progress")));
+      when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class))).thenReturn(failed);
+    } else {
+      when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class))).thenThrow(new RuntimeException("boom"));
+    }
+
+    awsGlueSyncClient.updateLastCommitTimeSynced(tableName);
+
+    verify(mockAwsGlue, times(1)).updateTable(any(UpdateTableRequest.class));
+  }
+
+  @Test
+  void testUpdateLastCommitTimeSynced_wrapsGlueFailure() {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    when(mockAwsGlue.updateTable(any(UpdateTableRequest.class))).thenThrow(new RuntimeException("boom"));
+
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class,
+        () -> awsGlueSyncClient.updateLastCommitTimeSynced(tableName));
+    assertTrue(ex.getMessage().contains("Fail to update last sync commit time"));
+  }
+
+  @Test
+  void testUpdateSerdeProperties_emptyPropertiesSkipUpdate() {
+    assertFalse(awsGlueSyncClient.updateSerdeProperties("tbl", Collections.emptyMap(), false));
+    verify(mockAwsGlue, never()).updateTable(any(UpdateTableRequest.class));
+  }
+
+  @Test
+  void testUpdateSerdeProperties_unchangedPropertiesSkipUpdate() {
+    String tableName = "tbl";
+    Map<String, String> serdeProperties = new HashMap<>();
+    serdeProperties.put("serialization.format", "1");
+    serdeProperties.put("path", "s3://base");
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithSerdeProperties(tableName, serdeProperties)).build()));
+
+    assertFalse(awsGlueSyncClient.updateSerdeProperties(tableName, new HashMap<>(serdeProperties), false));
+    verify(mockAwsGlue, never()).updateTable(any(UpdateTableRequest.class));
+  }
+
+  @Test
+  void testUpdateSerdeProperties_changedPropertiesRewriteSerdeInfo() {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithSerdeProperties(tableName,
+                serdePropertiesOf("serialization.format", "1", "location", "s3://old"))).build()));
+    ArgumentCaptor<UpdateTableRequest> captor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+
+    Map<String, String> serdeProperties = new HashMap<>();
+    serdeProperties.put("path", "s3://new");
+    assertTrue(awsGlueSyncClient.updateSerdeProperties(tableName, serdeProperties, false));
+
+    SerDeInfo sent = captor.getValue().tableInput().storageDescriptor().serdeInfo();
+    assertEquals("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe", sent.serializationLibrary(),
+        "the serde class is derived from the base file format");
+    assertEquals("s3://new", sent.parameters().get("path"));
+    assertEquals("1", sent.parameters().get("serialization.format"), "the serialization format is defaulted in");
+  }
+
+  @Test
+  void testUpdateSerdeProperties_wrapsGlueFailure() {
+    when(mockAwsGlue.getTable(any(GetTableRequest.class))).thenThrow(EntityNotFoundException.class);
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class,
+        () -> awsGlueSyncClient.updateSerdeProperties("tbl", new HashMap<>(Collections.singletonMap("path", "s3://new")), false));
+    assertTrue(ex.getMessage().contains("Failed to update table serde info for table"));
+  }
+
+  @Test
+  void testCreateTable_existingTableIsNotRecreated() {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(getTableWithDefaultProps(tableName, Collections.emptyList(), Collections.emptyList()));
+
+    awsGlueSyncClient.createTable(tableName, GlueTestUtil.getSimpleSchema(), "inputFormat", "outputFormat",
+        "serde", new HashMap<>(), new HashMap<>());
+
+    verify(mockAwsGlue, never()).createTable(any(CreateTableRequest.class));
+  }
+
+  @Test
+  void testTableExists_wrapsNonEntityNotFoundExecutionFailure() throws Exception {
+    CompletableFuture<GetTableResponse> failed = mock(CompletableFuture.class);
+    when(failed.get()).thenThrow(new ExecutionException(new RuntimeException("boom")));
+    when(mockAwsGlue.getTable(any(GetTableRequest.class))).thenReturn(failed);
+
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class, () -> awsGlueSyncClient.tableExists("tbl"));
+    assertTrue(ex.getMessage().contains("Fail to get table"));
+  }
+
+  @Test
+  void testTableExists_wrapsClientFailure() {
+    when(mockAwsGlue.getTable(any(GetTableRequest.class))).thenThrow(new RuntimeException("boom"));
+
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class, () -> awsGlueSyncClient.tableExists("tbl"));
+    assertTrue(ex.getMessage().contains("Fail to get table"));
+  }
+
+  @Test
+  void testDatabaseExists_wrapsClientFailure() {
+    when(mockAwsGlue.getDatabase(any(GetDatabaseRequest.class))).thenThrow(new RuntimeException("boom"));
+
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class, () -> awsGlueSyncClient.databaseExists("db"));
+    assertTrue(ex.getMessage().contains("Fail to check if database exists"));
+  }
+
+  @Test
+  void testDropTable_interruptionRestoresTheInterruptFlag() throws Exception {
+    CompletableFuture<DeleteTableResponse> failed = mock(CompletableFuture.class);
+    when(failed.get()).thenThrow(new InterruptedException("interrupted"));
+    when(mockAwsGlue.deleteTable(any(DeleteTableRequest.class))).thenReturn(failed);
+
+    assertThrows(HoodieGlueSyncException.class, () -> awsGlueSyncClient.dropTable("tbl"));
+    assertTrue(Thread.interrupted(), "the interrupt flag is restored for handlers up the stack");
+  }
+
+  @Test
+  void testBuildAsyncClient_appliesTheConfiguredEndpointAndRegion() {
+    TypedProperties props = GlueTestUtil.getHiveSyncConfig().getProps();
+    props.setProperty(HoodieAWSConfig.AWS_GLUE_ENDPOINT.key(), "https://glue.eu-west-1.amazonaws.com");
+    props.setProperty(HoodieAWSConfig.AWS_GLUE_REGION.key(), "eu-west-1");
+
+    try (MockedStatic<GlueAsyncClient> glueStatic = mockStatic(GlueAsyncClient.class);
+         MockedStatic<StsClient> stsStatic = mockStatic(StsClient.class)) {
+      GlueAsyncClientBuilder builder = mock(GlueAsyncClientBuilder.class);
+      glueStatic.when(GlueAsyncClient::builder).thenReturn(builder);
+      when(builder.credentialsProvider(any())).thenReturn(builder);
+      when(builder.endpointOverride(any(URI.class))).thenReturn(builder);
+      when(builder.region(any(Region.class))).thenReturn(builder);
+      when(builder.build()).thenReturn(mockAwsGlue);
+      stsStatic.when(StsClient::create).thenReturn(mockSts);
+
+      new AWSGlueCatalogSyncClient(new HiveSyncConfig(props), GlueTestUtil.getMetaClient());
+
+      verify(builder).endpointOverride(URI.create("https://glue.eu-west-1.amazonaws.com"));

Review Comment:
   Both `buildAsyncClient` tests set `hoodie.aws.glue.endpoint`, so the default deployment with neither endpoint nor region configured - the arm that leaves the builder untouched - is never exercised. The same static-mock harness with both unset, asserting `endpointOverride` and `region` are never called, would cover it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(aws): cover the Glue sync client, DynamoTableUtils and HoodieAWSConfig [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #19876:
URL: https://github.com/apache/hudi/pull/19876#issuecomment-5613849770

   <img width="901" height="263" alt="image" src="https://github.com/user-attachments/assets/8e293e5a-4800-4620-9ea8-93869cbd8834" />
   Source: https://app.codecov.io/gh/apache/hudi/commit/bb7fccea371553f05d5d66a51d7b2d6711c8e0df
   
   Merging this in.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(aws): cover the Glue sync client, DynamoTableUtils and HoodieAWSConfig [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19876:
URL: https://github.com/apache/hudi/pull/19876#issuecomment-5612944061

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4f362f6a7c6b88a24cd8407f6cf3d51d0fad5a70",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16944",
       "triggerID" : "4f362f6a7c6b88a24cd8407f6cf3d51d0fad5a70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb7fccea371553f05d5d66a51d7b2d6711c8e0df",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16970",
       "triggerID" : "bb7fccea371553f05d5d66a51d7b2d6711c8e0df",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f362f6a7c6b88a24cd8407f6cf3d51d0fad5a70 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16944) 
   * bb7fccea371553f05d5d66a51d7b2d6711c8e0df Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16970) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(aws): cover the Glue sync client, DynamoTableUtils and HoodieAWSConfig [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19876:
URL: https://github.com/apache/hudi/pull/19876#discussion_r3975338765


##########
hudi-aws/src/test/java/org/apache/hudi/aws/utils/TestDynamoTableUtils.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.aws.utils;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import software.amazon.awssdk.core.exception.SdkClientException;
+import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
+import software.amazon.awssdk.services.dynamodb.model.CreateTableRequest;
+import software.amazon.awssdk.services.dynamodb.model.CreateTableResponse;
+import software.amazon.awssdk.services.dynamodb.model.DeleteTableRequest;
+import software.amazon.awssdk.services.dynamodb.model.DeleteTableResponse;
+import software.amazon.awssdk.services.dynamodb.model.DescribeTableRequest;
+import software.amazon.awssdk.services.dynamodb.model.DescribeTableResponse;
+import software.amazon.awssdk.services.dynamodb.model.ResourceInUseException;
+import software.amazon.awssdk.services.dynamodb.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.dynamodb.model.TableDescription;
+import software.amazon.awssdk.services.dynamodb.model.TableStatus;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.atLeast;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+/**
+ * Tests {@link DynamoTableUtils} against a mocked DynamoDB client. The polling helpers are always
+ * given an explicit, short timeout so the tests stay fast and deterministic.
+ */
+@ExtendWith(MockitoExtension.class)
+class TestDynamoTableUtils {
+
+  private static final String TABLE_NAME = "lock_table";
+  private static final int TIMEOUT_MS = 300;
+  private static final int INTERVAL_MS = 50;
+
+  @Mock
+  private DynamoDbClient dynamoDb;
+
+  @Test
+  void testWaitUntilExists_returnsOnTheFirstDescription() throws Exception {
+    when(dynamoDb.describeTable(any(DescribeTableRequest.class))).thenReturn(describeResponse(TableStatus.CREATING));
+
+    DynamoTableUtils.waitUntilExists(dynamoDb, TABLE_NAME, TIMEOUT_MS, INTERVAL_MS);
+
+    ArgumentCaptor<DescribeTableRequest> captor = ArgumentCaptor.forClass(DescribeTableRequest.class);
+    verify(dynamoDb, times(1)).describeTable(captor.capture());
+    assertEquals(TABLE_NAME, captor.getValue().tableName(),
+        "any table status is enough to prove the table exists");
+  }
+
+  @Test
+  void testWaitUntilExists_pollsUntilTheTableShowsUp() throws Exception {
+    when(dynamoDb.describeTable(any(DescribeTableRequest.class)))
+        .thenThrow(ResourceNotFoundException.builder().message("not there yet").build())
+        .thenReturn(describeResponse(TableStatus.ACTIVE));
+
+    DynamoTableUtils.waitUntilExists(dynamoDb, TABLE_NAME, TIMEOUT_MS, INTERVAL_MS);
+
+    verify(dynamoDb, times(2)).describeTable(any(DescribeTableRequest.class));
+  }
+
+  @Test
+  void testWaitUntilExists_throwsWhenTheTableNeverShowsUp() {
+    when(dynamoDb.describeTable(any(DescribeTableRequest.class)))
+        .thenThrow(ResourceNotFoundException.builder().message("not there yet").build());
+
+    SdkClientException ex = assertThrows(SdkClientException.class,
+        () -> DynamoTableUtils.waitUntilExists(dynamoDb, TABLE_NAME, TIMEOUT_MS, INTERVAL_MS));
+
+    assertTrue(ex.getMessage().contains(TABLE_NAME + " never returned a result"));
+    verify(dynamoDb, atLeast(2)).describeTable(any(DescribeTableRequest.class));
+  }
+
+  @Test
+  void testWaitUntilActive_returnsWhenTheTableIsActive() throws Exception {
+    when(dynamoDb.describeTable(any(DescribeTableRequest.class)))
+        .thenReturn(describeResponse(TableStatus.CREATING))
+        .thenReturn(describeResponse(TableStatus.ACTIVE));
+
+    DynamoTableUtils.waitUntilActive(dynamoDb, TABLE_NAME, TIMEOUT_MS, INTERVAL_MS);
+
+    verify(dynamoDb, times(2)).describeTable(any(DescribeTableRequest.class));
+  }
+
+  @Test
+  void testWaitUntilActive_throwsWhenTheTableStaysInAnotherState() {
+    when(dynamoDb.describeTable(any(DescribeTableRequest.class))).thenReturn(describeResponse(TableStatus.CREATING));
+
+    DynamoTableUtils.TableNeverTransitionedToStateException ex =
+        assertThrows(DynamoTableUtils.TableNeverTransitionedToStateException.class,
+            () -> DynamoTableUtils.waitUntilActive(dynamoDb, TABLE_NAME, TIMEOUT_MS, INTERVAL_MS));
+
+    assertTrue(ex.getMessage().contains(TABLE_NAME + " never transitioned to desired state of ACTIVE"));
+  }
+
+  @Test
+  void testWaitUntilActive_throwsWhenTheTableNeverAppears() {
+    when(dynamoDb.describeTable(any(DescribeTableRequest.class)))
+        .thenThrow(ResourceNotFoundException.builder().message("not there yet").build());
+
+    assertThrows(DynamoTableUtils.TableNeverTransitionedToStateException.class,
+        () -> DynamoTableUtils.waitUntilActive(dynamoDb, TABLE_NAME, TIMEOUT_MS, INTERVAL_MS),
+        "a table that never gets described is reported the same way as one stuck in another state");
+  }
+
+  @Test
+  void testDefaultTimeoutOverloadsReturnAsSoonAsTheTableIsReady() throws Exception {
+    when(dynamoDb.describeTable(any(DescribeTableRequest.class))).thenReturn(describeResponse(TableStatus.ACTIVE));
+
+    // both overloads poll before sleeping, so an already-ready table returns without waiting
+    DynamoTableUtils.waitUntilExists(dynamoDb, TABLE_NAME);

Review Comment:
   Added `@Timeout(value = 10, unit = SECONDS)` to that test and reworded the class javadoc: the explicit-timeout tests bound their own runtime, the default-overload one is bounded by `@Timeout`. Done in bb7fccea3715.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(aws): cover the Glue sync client, DynamoTableUtils and HoodieAWSConfig [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19876:
URL: https://github.com/apache/hudi/pull/19876#issuecomment-5600595297

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4f362f6a7c6b88a24cd8407f6cf3d51d0fad5a70",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16944",
       "triggerID" : "4f362f6a7c6b88a24cd8407f6cf3d51d0fad5a70",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f362f6a7c6b88a24cd8407f6cf3d51d0fad5a70 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16944) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(aws): cover the Glue sync client, DynamoTableUtils and HoodieAWSConfig [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19876:
URL: https://github.com/apache/hudi/pull/19876#discussion_r3975337543


##########
hudi-aws/src/test/java/org/apache/hudi/aws/sync/TestAWSGlueSyncClient.java:
##########
@@ -1010,4 +1034,631 @@ void testUpdateTableSchema_issuesOneUpdateTable() {
     verify(mockAwsGlue, times(1)).updateTable(any(UpdateTableRequest.class));
     verify(mockAwsGlue, never()).batchUpdatePartition(any(BatchUpdatePartitionRequest.class));
   }
+
+  @Test
+  void testGetPartitionsFromList_returnsPartitionsKnownToGlue() {
+    String tableName = "tbl";
+    software.amazon.awssdk.services.glue.model.Partition gluePartition =
+        software.amazon.awssdk.services.glue.model.Partition.builder()
+            .values("2024-01-15")
+            .storageDescriptor(StorageDescriptor.builder().location("s3://base/2024/01/15").build())
+            .build();
+    ArgumentCaptor<BatchGetPartitionRequest> captor = ArgumentCaptor.forClass(BatchGetPartitionRequest.class);
+    when(mockAwsGlue.batchGetPartition(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(
+            BatchGetPartitionResponse.builder().partitions(gluePartition).build()));
+
+    List<Partition> result = awsGlueSyncClient.getPartitionsFromList(tableName, Arrays.asList("2024/01/15", "2024/01/16"));
+
+    assertEquals(1, result.size(), "only the partition Glue knows about is returned");
+    assertEquals(Collections.singletonList("2024-01-15"), result.get(0).getValues());
+    assertEquals("s3://base/2024/01/15", result.get(0).getStorageLocation());
+
+    BatchGetPartitionRequest sent = captor.getValue();
+    assertEquals(GlueTestUtil.DB_NAME, sent.databaseName());
+    assertEquals(tableName, sent.tableName());
+    assertEquals(Arrays.asList(Collections.singletonList("2024-01-15"), Collections.singletonList("2024-01-16")),
+        sent.partitionsToGet().stream().map(PartitionValueList::values).collect(Collectors.toList()),
+        "the requested partitions are the extracted partition values, not the storage paths");
+  }
+
+  @Test
+  void testGetPartitionsFromList_emptyListDoesNotCallGlue() {
+    assertTrue(awsGlueSyncClient.getPartitionsFromList("tbl", Collections.emptyList()).isEmpty());
+    verify(mockAwsGlue, never()).batchGetPartition(any(BatchGetPartitionRequest.class));
+  }
+
+  @Test
+  void testGetMetastoreSchema_mergesColumnsAndPartitionKeys() {
+    String tableName = "tbl";
+    List<Column> columns = Arrays.asList(GlueTestUtil.getColumn("name", "string", null),
+        GlueTestUtil.getColumn("age", "int", null));
+    List<Column> partitionKeys = Collections.singletonList(GlueTestUtil.getColumn("datestr", "string", null));
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(getTableWithDefaultProps(tableName, columns, partitionKeys));
+
+    Map<String, String> schema = awsGlueSyncClient.getMetastoreSchema(tableName);
+
+    assertEquals(3, schema.size());
+    assertEquals("STRING", schema.get("name"), "column types are upper cased");
+    assertEquals("INT", schema.get("age"));
+    assertEquals("STRING", schema.get("datestr"), "partition keys are merged into the schema");
+  }
+
+  @Test
+  void testGetMetastoreSchema_wrapsGlueFailure() {
+    when(mockAwsGlue.getTable(any(GetTableRequest.class))).thenThrow(new RuntimeException("boom"));
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class,
+        () -> awsGlueSyncClient.getMetastoreSchema("tbl"));
+    assertTrue(ex.getMessage().contains("Fail to get schema for table"));
+  }
+
+  @Test
+  void testGetLastCommitTimeSynced_readsTableParameters() {
+    Map<String, String> parameters = new HashMap<>();
+    parameters.put(HOODIE_LAST_COMMIT_TIME_SYNC, "100");
+    parameters.put(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC, "110");
+    Table withSyncTimes = tableWithParameters("synced", parameters);
+    Table withoutSyncTimes = tableWithParameters("unsynced", new HashMap<>());
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(withSyncTimes).build()))
+        .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(withoutSyncTimes).build()));
+
+    assertEquals("100", awsGlueSyncClient.getLastCommitTimeSynced("synced").get());
+    assertEquals("110", awsGlueSyncClient.getLastCommitCompletionTimeSynced("synced").get());
+    // the table is cached per name, so the second table name triggers the second stubbed response
+    assertFalse(awsGlueSyncClient.getLastCommitTimeSynced("unsynced").isPresent());
+    assertFalse(awsGlueSyncClient.getLastCommitCompletionTimeSynced("unsynced").isPresent());
+    verify(mockAwsGlue, times(2)).getTable(any(GetTableRequest.class));
+  }
+
+  @Test
+  void testGetStorageFieldSchemas_readsFieldsAndDocsFromStorage() {
+    Map<String, FieldSchema> byName = awsGlueSyncClient.getStorageFieldSchemas().stream()
+        .collect(Collectors.toMap(FieldSchema::getName, f -> f));
+
+    assertEquals("int", byName.get("id").getType());
+    assertEquals(GlueTestUtil.ID_FIELD_DOC, byName.get("id").getComment().get());
+    assertEquals("string", byName.get("name").getType());
+    assertEquals(GlueTestUtil.NAME_FIELD_DOC, byName.get("name").getComment().get());
+    assertTrue(byName.containsKey("_hoodie_commit_time"), "metadata fields are part of the storage schema");
+  }
+
+  @Test
+  void testManagePartitionIndexes_disabledDeactivatesFlagAndDropsIndexes() throws Exception {
+    String tableName = "tbl";
+    Map<String, String> parameters = new HashMap<>();
+    parameters.put(GLUE_PARTITION_INDEX_ENABLE, "true");
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, parameters)).build()));
+    ArgumentCaptor<UpdateTableRequest> updateCaptor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(updateCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(partitionIndexDescriptor("idx_one", "datestr"))
+            .build()));
+    ArgumentCaptor<DeletePartitionIndexRequest> deleteCaptor = ArgumentCaptor.forClass(DeletePartitionIndexRequest.class);
+    when(mockAwsGlue.deletePartitionIndex(deleteCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(DeletePartitionIndexResponse.builder().build()));
+
+    awsGlueSyncClient.managePartitionIndexes(tableName);
+
+    assertEquals("false", updateCaptor.getValue().tableInput().parameters().get(GLUE_PARTITION_INDEX_ENABLE),
+        "partition index usage is deactivated when the feature is off");
+    assertEquals(Collections.singletonList("idx_one"), deleteCaptor.getAllValues().stream()
+        .map(DeletePartitionIndexRequest::indexName).collect(Collectors.toList()));
+    verify(mockAwsGlue, never()).createPartitionIndex(any(CreatePartitionIndexRequest.class));
+  }
+
+  @Test
+  void testManagePartitionIndexes_enabledDropsStaleIndexesAndCreatesMissingOnes() throws Exception {
+    String tableName = "tbl";
+    TypedProperties props = GlueTestUtil.getHiveSyncConfig().getProps();
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS_ENABLE.key(), "true");
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS.key(), "datestr;hour,region");
+    awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, mockSts, new HiveSyncConfig(props), GlueTestUtil.getMetaClient());
+
+    // the table has no partition_filtering.enabled parameter, so indexing has to be activated first
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    ArgumentCaptor<UpdateTableRequest> updateCaptor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(updateCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+
+    PartitionIndexDescriptor keptIndex = partitionIndexDescriptor("kept_idx", "datestr", "hour");
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(keptIndex, partitionIndexDescriptor("stale_idx", "old_col"))
+            .build()))
+        // after a drop the index list is re-read
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(keptIndex)
+            .build()));
+    ArgumentCaptor<DeletePartitionIndexRequest> deleteCaptor = ArgumentCaptor.forClass(DeletePartitionIndexRequest.class);
+    when(mockAwsGlue.deletePartitionIndex(deleteCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(DeletePartitionIndexResponse.builder().build()));
+    ArgumentCaptor<CreatePartitionIndexRequest> createCaptor = ArgumentCaptor.forClass(CreatePartitionIndexRequest.class);
+    when(mockAwsGlue.createPartitionIndex(createCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(CreatePartitionIndexResponse.builder().build()));
+
+    awsGlueSyncClient.managePartitionIndexes(tableName);
+
+    assertEquals("true", updateCaptor.getValue().tableInput().parameters().get(GLUE_PARTITION_INDEX_ENABLE));
+    assertEquals(Collections.singletonList("stale_idx"), deleteCaptor.getAllValues().stream()
+        .map(DeletePartitionIndexRequest::indexName).collect(Collectors.toList()),
+        "only the index that is no longer configured is dropped");
+    assertEquals(1, createCaptor.getAllValues().size(), "the already existing index is not recreated");
+    PartitionIndex created = createCaptor.getValue().partitionIndex();
+    assertEquals(Collections.singletonList("region"), created.keys());
+    assertEquals("hudi_managed_[region]", created.indexName());
+    verify(mockAwsGlue, times(2)).getPartitionIndexes(any(GetPartitionIndexesRequest.class));
+  }
+
+  @Test
+  void testParsePartitionsIndexConfig_keepsOnlyTheFirstThreeIndexes() {
+    TypedProperties props = GlueTestUtil.getHiveSyncConfig().getProps();
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS.key(), "a;b,c,d,e");
+    awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, mockSts, new HiveSyncConfig(props), GlueTestUtil.getMetaClient());
+
+    assertEquals(Arrays.asList(Arrays.asList("a", "b"), Collections.singletonList("c"), Collections.singletonList("d")),
+        awsGlueSyncClient.parsePartitionsIndexConfig(), "glue supports at most three partition indexes");
+  }
+
+  @Test
+  void testUpdateLastCommitTimeSynced_writesTimelineInstantToTableParameters() {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    ArgumentCaptor<UpdateTableRequest> captor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder().build()));
+
+    awsGlueSyncClient.updateLastCommitTimeSynced(tableName);
+
+    Map<String, String> parameters = captor.getValue().tableInput().parameters();
+    assertEquals(GlueTestUtil.INSTANT_TIME, parameters.get(HOODIE_LAST_COMMIT_TIME_SYNC),
+        "the last instant of the active timeline is synced");
+    assertEquals(GlueTestUtil.COMPLETION_TIME, parameters.get(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC),
+        "the completion time of that instant is synced alongside it");
+    assertTrue(captor.getValue().skipArchive(), "table archiving is skipped by default");
+  }
+
+  /**
+   * An indexation already in flight surfaces as an {@link ExecutionException}, anything else lands in the
+   * catch-all. Neither may fail the commit-time sync.
+   */
+  @ParameterizedTest
+  @ValueSource(booleans = {true, false})
+  void testUpdateLastCommitTimeSynced_partitionIndexFailureDoesNotFailTheSync(boolean asExecutionFailure) throws Exception {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    when(mockAwsGlue.updateTable(any(UpdateTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    if (asExecutionFailure) {
+      CompletableFuture<GetPartitionIndexesResponse> failed = mock(CompletableFuture.class);
+      when(failed.get()).thenThrow(new ExecutionException(new RuntimeException("indexing in progress")));
+      when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class))).thenReturn(failed);
+    } else {
+      when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class))).thenThrow(new RuntimeException("boom"));
+    }
+
+    awsGlueSyncClient.updateLastCommitTimeSynced(tableName);
+
+    verify(mockAwsGlue, times(1)).updateTable(any(UpdateTableRequest.class));
+  }
+
+  @Test
+  void testUpdateLastCommitTimeSynced_wrapsGlueFailure() {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    when(mockAwsGlue.updateTable(any(UpdateTableRequest.class))).thenThrow(new RuntimeException("boom"));
+
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class,
+        () -> awsGlueSyncClient.updateLastCommitTimeSynced(tableName));
+    assertTrue(ex.getMessage().contains("Fail to update last sync commit time"));
+  }
+
+  @Test
+  void testUpdateSerdeProperties_emptyPropertiesSkipUpdate() {
+    assertFalse(awsGlueSyncClient.updateSerdeProperties("tbl", Collections.emptyMap(), false));
+    verify(mockAwsGlue, never()).updateTable(any(UpdateTableRequest.class));
+  }
+
+  @Test
+  void testUpdateSerdeProperties_unchangedPropertiesSkipUpdate() {
+    String tableName = "tbl";
+    Map<String, String> serdeProperties = new HashMap<>();
+    serdeProperties.put("serialization.format", "1");
+    serdeProperties.put("path", "s3://base");
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithSerdeProperties(tableName, serdeProperties)).build()));
+
+    assertFalse(awsGlueSyncClient.updateSerdeProperties(tableName, new HashMap<>(serdeProperties), false));
+    verify(mockAwsGlue, never()).updateTable(any(UpdateTableRequest.class));
+  }
+
+  @Test
+  void testUpdateSerdeProperties_changedPropertiesRewriteSerdeInfo() {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithSerdeProperties(tableName,
+                serdePropertiesOf("serialization.format", "1", "location", "s3://old"))).build()));
+    ArgumentCaptor<UpdateTableRequest> captor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+
+    Map<String, String> serdeProperties = new HashMap<>();
+    serdeProperties.put("path", "s3://new");
+    assertTrue(awsGlueSyncClient.updateSerdeProperties(tableName, serdeProperties, false));

Review Comment:
   It is not read anywhere in the Glue client; only `HoodieHiveSyncClient` uses it to pick the input format. The changed-properties test is now parameterized over both values and asserts the same request, with the input and output format carried over from the catalog unchanged. Whether Glue should honour the flag for `_rt` tables is a separate question, so production is untouched here. Done in bb7fccea3715.



##########
hudi-aws/src/test/java/org/apache/hudi/aws/sync/TestAWSGlueSyncClient.java:
##########
@@ -1010,4 +1034,631 @@ void testUpdateTableSchema_issuesOneUpdateTable() {
     verify(mockAwsGlue, times(1)).updateTable(any(UpdateTableRequest.class));
     verify(mockAwsGlue, never()).batchUpdatePartition(any(BatchUpdatePartitionRequest.class));
   }
+
+  @Test
+  void testGetPartitionsFromList_returnsPartitionsKnownToGlue() {
+    String tableName = "tbl";
+    software.amazon.awssdk.services.glue.model.Partition gluePartition =
+        software.amazon.awssdk.services.glue.model.Partition.builder()
+            .values("2024-01-15")
+            .storageDescriptor(StorageDescriptor.builder().location("s3://base/2024/01/15").build())
+            .build();
+    ArgumentCaptor<BatchGetPartitionRequest> captor = ArgumentCaptor.forClass(BatchGetPartitionRequest.class);
+    when(mockAwsGlue.batchGetPartition(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(
+            BatchGetPartitionResponse.builder().partitions(gluePartition).build()));
+
+    List<Partition> result = awsGlueSyncClient.getPartitionsFromList(tableName, Arrays.asList("2024/01/15", "2024/01/16"));
+
+    assertEquals(1, result.size(), "only the partition Glue knows about is returned");
+    assertEquals(Collections.singletonList("2024-01-15"), result.get(0).getValues());
+    assertEquals("s3://base/2024/01/15", result.get(0).getStorageLocation());
+
+    BatchGetPartitionRequest sent = captor.getValue();
+    assertEquals(GlueTestUtil.DB_NAME, sent.databaseName());
+    assertEquals(tableName, sent.tableName());
+    assertEquals(Arrays.asList(Collections.singletonList("2024-01-15"), Collections.singletonList("2024-01-16")),
+        sent.partitionsToGet().stream().map(PartitionValueList::values).collect(Collectors.toList()),
+        "the requested partitions are the extracted partition values, not the storage paths");
+  }
+
+  @Test
+  void testGetPartitionsFromList_emptyListDoesNotCallGlue() {
+    assertTrue(awsGlueSyncClient.getPartitionsFromList("tbl", Collections.emptyList()).isEmpty());
+    verify(mockAwsGlue, never()).batchGetPartition(any(BatchGetPartitionRequest.class));
+  }
+
+  @Test
+  void testGetMetastoreSchema_mergesColumnsAndPartitionKeys() {
+    String tableName = "tbl";
+    List<Column> columns = Arrays.asList(GlueTestUtil.getColumn("name", "string", null),
+        GlueTestUtil.getColumn("age", "int", null));
+    List<Column> partitionKeys = Collections.singletonList(GlueTestUtil.getColumn("datestr", "string", null));
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(getTableWithDefaultProps(tableName, columns, partitionKeys));
+
+    Map<String, String> schema = awsGlueSyncClient.getMetastoreSchema(tableName);
+
+    assertEquals(3, schema.size());
+    assertEquals("STRING", schema.get("name"), "column types are upper cased");
+    assertEquals("INT", schema.get("age"));
+    assertEquals("STRING", schema.get("datestr"), "partition keys are merged into the schema");
+  }
+
+  @Test
+  void testGetMetastoreSchema_wrapsGlueFailure() {
+    when(mockAwsGlue.getTable(any(GetTableRequest.class))).thenThrow(new RuntimeException("boom"));
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class,
+        () -> awsGlueSyncClient.getMetastoreSchema("tbl"));
+    assertTrue(ex.getMessage().contains("Fail to get schema for table"));
+  }
+
+  @Test
+  void testGetLastCommitTimeSynced_readsTableParameters() {
+    Map<String, String> parameters = new HashMap<>();
+    parameters.put(HOODIE_LAST_COMMIT_TIME_SYNC, "100");
+    parameters.put(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC, "110");
+    Table withSyncTimes = tableWithParameters("synced", parameters);
+    Table withoutSyncTimes = tableWithParameters("unsynced", new HashMap<>());
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(withSyncTimes).build()))
+        .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(withoutSyncTimes).build()));
+
+    assertEquals("100", awsGlueSyncClient.getLastCommitTimeSynced("synced").get());
+    assertEquals("110", awsGlueSyncClient.getLastCommitCompletionTimeSynced("synced").get());
+    // the table is cached per name, so the second table name triggers the second stubbed response
+    assertFalse(awsGlueSyncClient.getLastCommitTimeSynced("unsynced").isPresent());
+    assertFalse(awsGlueSyncClient.getLastCommitCompletionTimeSynced("unsynced").isPresent());
+    verify(mockAwsGlue, times(2)).getTable(any(GetTableRequest.class));
+  }
+
+  @Test
+  void testGetStorageFieldSchemas_readsFieldsAndDocsFromStorage() {
+    Map<String, FieldSchema> byName = awsGlueSyncClient.getStorageFieldSchemas().stream()
+        .collect(Collectors.toMap(FieldSchema::getName, f -> f));
+
+    assertEquals("int", byName.get("id").getType());
+    assertEquals(GlueTestUtil.ID_FIELD_DOC, byName.get("id").getComment().get());
+    assertEquals("string", byName.get("name").getType());
+    assertEquals(GlueTestUtil.NAME_FIELD_DOC, byName.get("name").getComment().get());
+    assertTrue(byName.containsKey("_hoodie_commit_time"), "metadata fields are part of the storage schema");
+  }
+
+  @Test
+  void testManagePartitionIndexes_disabledDeactivatesFlagAndDropsIndexes() throws Exception {
+    String tableName = "tbl";
+    Map<String, String> parameters = new HashMap<>();
+    parameters.put(GLUE_PARTITION_INDEX_ENABLE, "true");
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, parameters)).build()));
+    ArgumentCaptor<UpdateTableRequest> updateCaptor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(updateCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(partitionIndexDescriptor("idx_one", "datestr"))
+            .build()));
+    ArgumentCaptor<DeletePartitionIndexRequest> deleteCaptor = ArgumentCaptor.forClass(DeletePartitionIndexRequest.class);
+    when(mockAwsGlue.deletePartitionIndex(deleteCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(DeletePartitionIndexResponse.builder().build()));
+
+    awsGlueSyncClient.managePartitionIndexes(tableName);
+
+    assertEquals("false", updateCaptor.getValue().tableInput().parameters().get(GLUE_PARTITION_INDEX_ENABLE),
+        "partition index usage is deactivated when the feature is off");
+    assertEquals(Collections.singletonList("idx_one"), deleteCaptor.getAllValues().stream()
+        .map(DeletePartitionIndexRequest::indexName).collect(Collectors.toList()));
+    verify(mockAwsGlue, never()).createPartitionIndex(any(CreatePartitionIndexRequest.class));
+  }
+
+  @Test
+  void testManagePartitionIndexes_enabledDropsStaleIndexesAndCreatesMissingOnes() throws Exception {
+    String tableName = "tbl";
+    TypedProperties props = GlueTestUtil.getHiveSyncConfig().getProps();
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS_ENABLE.key(), "true");
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS.key(), "datestr;hour,region");
+    awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, mockSts, new HiveSyncConfig(props), GlueTestUtil.getMetaClient());
+
+    // the table has no partition_filtering.enabled parameter, so indexing has to be activated first
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    ArgumentCaptor<UpdateTableRequest> updateCaptor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(updateCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+
+    PartitionIndexDescriptor keptIndex = partitionIndexDescriptor("kept_idx", "datestr", "hour");
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(keptIndex, partitionIndexDescriptor("stale_idx", "old_col"))
+            .build()))
+        // after a drop the index list is re-read
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(keptIndex)
+            .build()));
+    ArgumentCaptor<DeletePartitionIndexRequest> deleteCaptor = ArgumentCaptor.forClass(DeletePartitionIndexRequest.class);
+    when(mockAwsGlue.deletePartitionIndex(deleteCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(DeletePartitionIndexResponse.builder().build()));
+    ArgumentCaptor<CreatePartitionIndexRequest> createCaptor = ArgumentCaptor.forClass(CreatePartitionIndexRequest.class);
+    when(mockAwsGlue.createPartitionIndex(createCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(CreatePartitionIndexResponse.builder().build()));
+
+    awsGlueSyncClient.managePartitionIndexes(tableName);
+
+    assertEquals("true", updateCaptor.getValue().tableInput().parameters().get(GLUE_PARTITION_INDEX_ENABLE));
+    assertEquals(Collections.singletonList("stale_idx"), deleteCaptor.getAllValues().stream()
+        .map(DeletePartitionIndexRequest::indexName).collect(Collectors.toList()),
+        "only the index that is no longer configured is dropped");
+    assertEquals(1, createCaptor.getAllValues().size(), "the already existing index is not recreated");
+    PartitionIndex created = createCaptor.getValue().partitionIndex();
+    assertEquals(Collections.singletonList("region"), created.keys());
+    assertEquals("hudi_managed_[region]", created.indexName());
+    verify(mockAwsGlue, times(2)).getPartitionIndexes(any(GetPartitionIndexesRequest.class));
+  }
+
+  @Test
+  void testParsePartitionsIndexConfig_keepsOnlyTheFirstThreeIndexes() {
+    TypedProperties props = GlueTestUtil.getHiveSyncConfig().getProps();
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS.key(), "a;b,c,d,e");
+    awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, mockSts, new HiveSyncConfig(props), GlueTestUtil.getMetaClient());
+
+    assertEquals(Arrays.asList(Arrays.asList("a", "b"), Collections.singletonList("c"), Collections.singletonList("d")),
+        awsGlueSyncClient.parsePartitionsIndexConfig(), "glue supports at most three partition indexes");
+  }
+
+  @Test
+  void testUpdateLastCommitTimeSynced_writesTimelineInstantToTableParameters() {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    ArgumentCaptor<UpdateTableRequest> captor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder().build()));
+
+    awsGlueSyncClient.updateLastCommitTimeSynced(tableName);
+
+    Map<String, String> parameters = captor.getValue().tableInput().parameters();
+    assertEquals(GlueTestUtil.INSTANT_TIME, parameters.get(HOODIE_LAST_COMMIT_TIME_SYNC),
+        "the last instant of the active timeline is synced");
+    assertEquals(GlueTestUtil.COMPLETION_TIME, parameters.get(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC),
+        "the completion time of that instant is synced alongside it");
+    assertTrue(captor.getValue().skipArchive(), "table archiving is skipped by default");
+  }
+
+  /**
+   * An indexation already in flight surfaces as an {@link ExecutionException}, anything else lands in the
+   * catch-all. Neither may fail the commit-time sync.
+   */
+  @ParameterizedTest
+  @ValueSource(booleans = {true, false})
+  void testUpdateLastCommitTimeSynced_partitionIndexFailureDoesNotFailTheSync(boolean asExecutionFailure) throws Exception {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    when(mockAwsGlue.updateTable(any(UpdateTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    if (asExecutionFailure) {
+      CompletableFuture<GetPartitionIndexesResponse> failed = mock(CompletableFuture.class);
+      when(failed.get()).thenThrow(new ExecutionException(new RuntimeException("indexing in progress")));
+      when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class))).thenReturn(failed);
+    } else {
+      when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class))).thenThrow(new RuntimeException("boom"));
+    }
+
+    awsGlueSyncClient.updateLastCommitTimeSynced(tableName);
+
+    verify(mockAwsGlue, times(1)).updateTable(any(UpdateTableRequest.class));
+  }
+
+  @Test
+  void testUpdateLastCommitTimeSynced_wrapsGlueFailure() {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    when(mockAwsGlue.updateTable(any(UpdateTableRequest.class))).thenThrow(new RuntimeException("boom"));
+
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class,
+        () -> awsGlueSyncClient.updateLastCommitTimeSynced(tableName));
+    assertTrue(ex.getMessage().contains("Fail to update last sync commit time"));
+  }
+
+  @Test
+  void testUpdateSerdeProperties_emptyPropertiesSkipUpdate() {
+    assertFalse(awsGlueSyncClient.updateSerdeProperties("tbl", Collections.emptyMap(), false));
+    verify(mockAwsGlue, never()).updateTable(any(UpdateTableRequest.class));
+  }
+
+  @Test
+  void testUpdateSerdeProperties_unchangedPropertiesSkipUpdate() {
+    String tableName = "tbl";
+    Map<String, String> serdeProperties = new HashMap<>();
+    serdeProperties.put("serialization.format", "1");
+    serdeProperties.put("path", "s3://base");
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithSerdeProperties(tableName, serdeProperties)).build()));
+
+    assertFalse(awsGlueSyncClient.updateSerdeProperties(tableName, new HashMap<>(serdeProperties), false));
+    verify(mockAwsGlue, never()).updateTable(any(UpdateTableRequest.class));
+  }
+
+  @Test
+  void testUpdateSerdeProperties_changedPropertiesRewriteSerdeInfo() {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithSerdeProperties(tableName,
+                serdePropertiesOf("serialization.format", "1", "location", "s3://old"))).build()));
+    ArgumentCaptor<UpdateTableRequest> captor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+
+    Map<String, String> serdeProperties = new HashMap<>();
+    serdeProperties.put("path", "s3://new");
+    assertTrue(awsGlueSyncClient.updateSerdeProperties(tableName, serdeProperties, false));
+
+    SerDeInfo sent = captor.getValue().tableInput().storageDescriptor().serdeInfo();
+    assertEquals("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe", sent.serializationLibrary(),
+        "the serde class is derived from the base file format");
+    assertEquals("s3://new", sent.parameters().get("path"));
+    assertEquals("1", sent.parameters().get("serialization.format"), "the serialization format is defaulted in");
+  }
+
+  @Test
+  void testUpdateSerdeProperties_wrapsGlueFailure() {
+    when(mockAwsGlue.getTable(any(GetTableRequest.class))).thenThrow(EntityNotFoundException.class);
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class,
+        () -> awsGlueSyncClient.updateSerdeProperties("tbl", new HashMap<>(Collections.singletonMap("path", "s3://new")), false));
+    assertTrue(ex.getMessage().contains("Failed to update table serde info for table"));
+  }
+
+  @Test
+  void testCreateTable_existingTableIsNotRecreated() {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(getTableWithDefaultProps(tableName, Collections.emptyList(), Collections.emptyList()));
+
+    awsGlueSyncClient.createTable(tableName, GlueTestUtil.getSimpleSchema(), "inputFormat", "outputFormat",
+        "serde", new HashMap<>(), new HashMap<>());
+
+    verify(mockAwsGlue, never()).createTable(any(CreateTableRequest.class));
+  }
+
+  @Test
+  void testTableExists_wrapsNonEntityNotFoundExecutionFailure() throws Exception {
+    CompletableFuture<GetTableResponse> failed = mock(CompletableFuture.class);
+    when(failed.get()).thenThrow(new ExecutionException(new RuntimeException("boom")));
+    when(mockAwsGlue.getTable(any(GetTableRequest.class))).thenReturn(failed);
+
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class, () -> awsGlueSyncClient.tableExists("tbl"));
+    assertTrue(ex.getMessage().contains("Fail to get table"));
+  }
+
+  @Test
+  void testTableExists_wrapsClientFailure() {
+    when(mockAwsGlue.getTable(any(GetTableRequest.class))).thenThrow(new RuntimeException("boom"));
+
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class, () -> awsGlueSyncClient.tableExists("tbl"));
+    assertTrue(ex.getMessage().contains("Fail to get table"));
+  }
+
+  @Test
+  void testDatabaseExists_wrapsClientFailure() {
+    when(mockAwsGlue.getDatabase(any(GetDatabaseRequest.class))).thenThrow(new RuntimeException("boom"));
+
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class, () -> awsGlueSyncClient.databaseExists("db"));
+    assertTrue(ex.getMessage().contains("Fail to check if database exists"));
+  }
+
+  @Test
+  void testDropTable_interruptionRestoresTheInterruptFlag() throws Exception {
+    CompletableFuture<DeleteTableResponse> failed = mock(CompletableFuture.class);
+    when(failed.get()).thenThrow(new InterruptedException("interrupted"));
+    when(mockAwsGlue.deleteTable(any(DeleteTableRequest.class))).thenReturn(failed);
+
+    assertThrows(HoodieGlueSyncException.class, () -> awsGlueSyncClient.dropTable("tbl"));
+    assertTrue(Thread.interrupted(), "the interrupt flag is restored for handlers up the stack");
+  }
+
+  @Test
+  void testBuildAsyncClient_appliesTheConfiguredEndpointAndRegion() {
+    TypedProperties props = GlueTestUtil.getHiveSyncConfig().getProps();
+    props.setProperty(HoodieAWSConfig.AWS_GLUE_ENDPOINT.key(), "https://glue.eu-west-1.amazonaws.com");
+    props.setProperty(HoodieAWSConfig.AWS_GLUE_REGION.key(), "eu-west-1");
+
+    try (MockedStatic<GlueAsyncClient> glueStatic = mockStatic(GlueAsyncClient.class);
+         MockedStatic<StsClient> stsStatic = mockStatic(StsClient.class)) {
+      GlueAsyncClientBuilder builder = mock(GlueAsyncClientBuilder.class);
+      glueStatic.when(GlueAsyncClient::builder).thenReturn(builder);
+      when(builder.credentialsProvider(any())).thenReturn(builder);
+      when(builder.endpointOverride(any(URI.class))).thenReturn(builder);
+      when(builder.region(any(Region.class))).thenReturn(builder);
+      when(builder.build()).thenReturn(mockAwsGlue);
+      stsStatic.when(StsClient::create).thenReturn(mockSts);
+
+      new AWSGlueCatalogSyncClient(new HiveSyncConfig(props), GlueTestUtil.getMetaClient());
+
+      verify(builder).endpointOverride(URI.create("https://glue.eu-west-1.amazonaws.com"));

Review Comment:
   Added `testBuildAsyncClient_withoutAnEndpointOrRegionKeepsTheSdkDefaults`: neither property set, `credentialsProvider` and `build` called, `endpointOverride` and `region` never. The builder stubbing moved into a shared `mockGlueClientBuilder` helper used by all three tests. Done in bb7fccea3715.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(aws): cover the Glue sync client, DynamoTableUtils and HoodieAWSConfig [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19876:
URL: https://github.com/apache/hudi/pull/19876#issuecomment-5598842589

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4f362f6a7c6b88a24cd8407f6cf3d51d0fad5a70",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4f362f6a7c6b88a24cd8407f6cf3d51d0fad5a70",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f362f6a7c6b88a24cd8407f6cf3d51d0fad5a70 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(aws): cover the Glue sync client, DynamoTableUtils and HoodieAWSConfig [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #19876:
URL: https://github.com/apache/hudi/pull/19876#issuecomment-5603001732

   @wombatu-kun Can you please help to review this. This will bring our coverage up by **0.21%** to **79.54%**.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(aws): cover the Glue sync client, DynamoTableUtils and HoodieAWSConfig [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19876:
URL: https://github.com/apache/hudi/pull/19876#issuecomment-5598932236

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4f362f6a7c6b88a24cd8407f6cf3d51d0fad5a70",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16944",
       "triggerID" : "4f362f6a7c6b88a24cd8407f6cf3d51d0fad5a70",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f362f6a7c6b88a24cd8407f6cf3d51d0fad5a70 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16944) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(aws): cover the Glue sync client, DynamoTableUtils and HoodieAWSConfig [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19876:
URL: https://github.com/apache/hudi/pull/19876#discussion_r3975336237


##########
hudi-aws/src/test/java/org/apache/hudi/aws/sync/TestAWSGlueSyncClient.java:
##########
@@ -1010,4 +1034,631 @@ void testUpdateTableSchema_issuesOneUpdateTable() {
     verify(mockAwsGlue, times(1)).updateTable(any(UpdateTableRequest.class));
     verify(mockAwsGlue, never()).batchUpdatePartition(any(BatchUpdatePartitionRequest.class));
   }
+
+  @Test
+  void testGetPartitionsFromList_returnsPartitionsKnownToGlue() {
+    String tableName = "tbl";
+    software.amazon.awssdk.services.glue.model.Partition gluePartition =
+        software.amazon.awssdk.services.glue.model.Partition.builder()
+            .values("2024-01-15")
+            .storageDescriptor(StorageDescriptor.builder().location("s3://base/2024/01/15").build())
+            .build();
+    ArgumentCaptor<BatchGetPartitionRequest> captor = ArgumentCaptor.forClass(BatchGetPartitionRequest.class);
+    when(mockAwsGlue.batchGetPartition(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(
+            BatchGetPartitionResponse.builder().partitions(gluePartition).build()));
+
+    List<Partition> result = awsGlueSyncClient.getPartitionsFromList(tableName, Arrays.asList("2024/01/15", "2024/01/16"));
+
+    assertEquals(1, result.size(), "only the partition Glue knows about is returned");
+    assertEquals(Collections.singletonList("2024-01-15"), result.get(0).getValues());
+    assertEquals("s3://base/2024/01/15", result.get(0).getStorageLocation());
+
+    BatchGetPartitionRequest sent = captor.getValue();
+    assertEquals(GlueTestUtil.DB_NAME, sent.databaseName());
+    assertEquals(tableName, sent.tableName());
+    assertEquals(Arrays.asList(Collections.singletonList("2024-01-15"), Collections.singletonList("2024-01-16")),
+        sent.partitionsToGet().stream().map(PartitionValueList::values).collect(Collectors.toList()),
+        "the requested partitions are the extracted partition values, not the storage paths");
+  }
+
+  @Test
+  void testGetPartitionsFromList_emptyListDoesNotCallGlue() {
+    assertTrue(awsGlueSyncClient.getPartitionsFromList("tbl", Collections.emptyList()).isEmpty());
+    verify(mockAwsGlue, never()).batchGetPartition(any(BatchGetPartitionRequest.class));
+  }
+
+  @Test
+  void testGetMetastoreSchema_mergesColumnsAndPartitionKeys() {
+    String tableName = "tbl";
+    List<Column> columns = Arrays.asList(GlueTestUtil.getColumn("name", "string", null),
+        GlueTestUtil.getColumn("age", "int", null));
+    List<Column> partitionKeys = Collections.singletonList(GlueTestUtil.getColumn("datestr", "string", null));
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(getTableWithDefaultProps(tableName, columns, partitionKeys));
+
+    Map<String, String> schema = awsGlueSyncClient.getMetastoreSchema(tableName);
+
+    assertEquals(3, schema.size());
+    assertEquals("STRING", schema.get("name"), "column types are upper cased");
+    assertEquals("INT", schema.get("age"));
+    assertEquals("STRING", schema.get("datestr"), "partition keys are merged into the schema");
+  }
+
+  @Test
+  void testGetMetastoreSchema_wrapsGlueFailure() {
+    when(mockAwsGlue.getTable(any(GetTableRequest.class))).thenThrow(new RuntimeException("boom"));
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class,
+        () -> awsGlueSyncClient.getMetastoreSchema("tbl"));
+    assertTrue(ex.getMessage().contains("Fail to get schema for table"));
+  }
+
+  @Test
+  void testGetLastCommitTimeSynced_readsTableParameters() {
+    Map<String, String> parameters = new HashMap<>();
+    parameters.put(HOODIE_LAST_COMMIT_TIME_SYNC, "100");
+    parameters.put(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC, "110");
+    Table withSyncTimes = tableWithParameters("synced", parameters);
+    Table withoutSyncTimes = tableWithParameters("unsynced", new HashMap<>());
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(withSyncTimes).build()))
+        .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(withoutSyncTimes).build()));
+
+    assertEquals("100", awsGlueSyncClient.getLastCommitTimeSynced("synced").get());
+    assertEquals("110", awsGlueSyncClient.getLastCommitCompletionTimeSynced("synced").get());
+    // the table is cached per name, so the second table name triggers the second stubbed response
+    assertFalse(awsGlueSyncClient.getLastCommitTimeSynced("unsynced").isPresent());
+    assertFalse(awsGlueSyncClient.getLastCommitCompletionTimeSynced("unsynced").isPresent());
+    verify(mockAwsGlue, times(2)).getTable(any(GetTableRequest.class));
+  }
+
+  @Test
+  void testGetStorageFieldSchemas_readsFieldsAndDocsFromStorage() {
+    Map<String, FieldSchema> byName = awsGlueSyncClient.getStorageFieldSchemas().stream()
+        .collect(Collectors.toMap(FieldSchema::getName, f -> f));
+
+    assertEquals("int", byName.get("id").getType());
+    assertEquals(GlueTestUtil.ID_FIELD_DOC, byName.get("id").getComment().get());
+    assertEquals("string", byName.get("name").getType());
+    assertEquals(GlueTestUtil.NAME_FIELD_DOC, byName.get("name").getComment().get());
+    assertTrue(byName.containsKey("_hoodie_commit_time"), "metadata fields are part of the storage schema");
+  }
+
+  @Test
+  void testManagePartitionIndexes_disabledDeactivatesFlagAndDropsIndexes() throws Exception {
+    String tableName = "tbl";
+    Map<String, String> parameters = new HashMap<>();
+    parameters.put(GLUE_PARTITION_INDEX_ENABLE, "true");
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, parameters)).build()));
+    ArgumentCaptor<UpdateTableRequest> updateCaptor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(updateCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(partitionIndexDescriptor("idx_one", "datestr"))
+            .build()));
+    ArgumentCaptor<DeletePartitionIndexRequest> deleteCaptor = ArgumentCaptor.forClass(DeletePartitionIndexRequest.class);
+    when(mockAwsGlue.deletePartitionIndex(deleteCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(DeletePartitionIndexResponse.builder().build()));
+
+    awsGlueSyncClient.managePartitionIndexes(tableName);
+
+    assertEquals("false", updateCaptor.getValue().tableInput().parameters().get(GLUE_PARTITION_INDEX_ENABLE),
+        "partition index usage is deactivated when the feature is off");
+    assertEquals(Collections.singletonList("idx_one"), deleteCaptor.getAllValues().stream()
+        .map(DeletePartitionIndexRequest::indexName).collect(Collectors.toList()));
+    verify(mockAwsGlue, never()).createPartitionIndex(any(CreatePartitionIndexRequest.class));
+  }
+
+  @Test
+  void testManagePartitionIndexes_enabledDropsStaleIndexesAndCreatesMissingOnes() throws Exception {
+    String tableName = "tbl";
+    TypedProperties props = GlueTestUtil.getHiveSyncConfig().getProps();
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS_ENABLE.key(), "true");
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS.key(), "datestr;hour,region");
+    awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, mockSts, new HiveSyncConfig(props), GlueTestUtil.getMetaClient());
+
+    // the table has no partition_filtering.enabled parameter, so indexing has to be activated first
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    ArgumentCaptor<UpdateTableRequest> updateCaptor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(updateCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+
+    PartitionIndexDescriptor keptIndex = partitionIndexDescriptor("kept_idx", "datestr", "hour");
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(keptIndex, partitionIndexDescriptor("stale_idx", "old_col"))
+            .build()))
+        // after a drop the index list is re-read
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(keptIndex)
+            .build()));
+    ArgumentCaptor<DeletePartitionIndexRequest> deleteCaptor = ArgumentCaptor.forClass(DeletePartitionIndexRequest.class);
+    when(mockAwsGlue.deletePartitionIndex(deleteCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(DeletePartitionIndexResponse.builder().build()));
+    ArgumentCaptor<CreatePartitionIndexRequest> createCaptor = ArgumentCaptor.forClass(CreatePartitionIndexRequest.class);
+    when(mockAwsGlue.createPartitionIndex(createCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(CreatePartitionIndexResponse.builder().build()));
+
+    awsGlueSyncClient.managePartitionIndexes(tableName);
+
+    assertEquals("true", updateCaptor.getValue().tableInput().parameters().get(GLUE_PARTITION_INDEX_ENABLE));
+    assertEquals(Collections.singletonList("stale_idx"), deleteCaptor.getAllValues().stream()
+        .map(DeletePartitionIndexRequest::indexName).collect(Collectors.toList()),
+        "only the index that is no longer configured is dropped");
+    assertEquals(1, createCaptor.getAllValues().size(), "the already existing index is not recreated");
+    PartitionIndex created = createCaptor.getValue().partitionIndex();
+    assertEquals(Collections.singletonList("region"), created.keys());
+    assertEquals("hudi_managed_[region]", created.indexName());
+    verify(mockAwsGlue, times(2)).getPartitionIndexes(any(GetPartitionIndexesRequest.class));

Review Comment:
   Added `testManagePartitionIndexes_enabledWithTheConfiguredIndexesInPlaceChangesNothing`: flag already `true`, one existing index equal to the configured fields. It asserts a single `getPartitionIndexes` and no `updateTable`, `deletePartitionIndex` or `createPartitionIndex`, so the `indexesChanges` re-read guard staying false is pinned. Done in bb7fccea3715.



##########
hudi-aws/src/test/java/org/apache/hudi/aws/sync/TestAWSGlueSyncClient.java:
##########
@@ -1010,4 +1034,631 @@ void testUpdateTableSchema_issuesOneUpdateTable() {
     verify(mockAwsGlue, times(1)).updateTable(any(UpdateTableRequest.class));
     verify(mockAwsGlue, never()).batchUpdatePartition(any(BatchUpdatePartitionRequest.class));
   }
+
+  @Test
+  void testGetPartitionsFromList_returnsPartitionsKnownToGlue() {
+    String tableName = "tbl";
+    software.amazon.awssdk.services.glue.model.Partition gluePartition =
+        software.amazon.awssdk.services.glue.model.Partition.builder()
+            .values("2024-01-15")
+            .storageDescriptor(StorageDescriptor.builder().location("s3://base/2024/01/15").build())
+            .build();
+    ArgumentCaptor<BatchGetPartitionRequest> captor = ArgumentCaptor.forClass(BatchGetPartitionRequest.class);
+    when(mockAwsGlue.batchGetPartition(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(
+            BatchGetPartitionResponse.builder().partitions(gluePartition).build()));
+
+    List<Partition> result = awsGlueSyncClient.getPartitionsFromList(tableName, Arrays.asList("2024/01/15", "2024/01/16"));
+
+    assertEquals(1, result.size(), "only the partition Glue knows about is returned");
+    assertEquals(Collections.singletonList("2024-01-15"), result.get(0).getValues());
+    assertEquals("s3://base/2024/01/15", result.get(0).getStorageLocation());
+
+    BatchGetPartitionRequest sent = captor.getValue();
+    assertEquals(GlueTestUtil.DB_NAME, sent.databaseName());
+    assertEquals(tableName, sent.tableName());
+    assertEquals(Arrays.asList(Collections.singletonList("2024-01-15"), Collections.singletonList("2024-01-16")),
+        sent.partitionsToGet().stream().map(PartitionValueList::values).collect(Collectors.toList()),
+        "the requested partitions are the extracted partition values, not the storage paths");
+  }
+
+  @Test
+  void testGetPartitionsFromList_emptyListDoesNotCallGlue() {
+    assertTrue(awsGlueSyncClient.getPartitionsFromList("tbl", Collections.emptyList()).isEmpty());
+    verify(mockAwsGlue, never()).batchGetPartition(any(BatchGetPartitionRequest.class));
+  }
+
+  @Test
+  void testGetMetastoreSchema_mergesColumnsAndPartitionKeys() {
+    String tableName = "tbl";
+    List<Column> columns = Arrays.asList(GlueTestUtil.getColumn("name", "string", null),
+        GlueTestUtil.getColumn("age", "int", null));
+    List<Column> partitionKeys = Collections.singletonList(GlueTestUtil.getColumn("datestr", "string", null));
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(getTableWithDefaultProps(tableName, columns, partitionKeys));
+
+    Map<String, String> schema = awsGlueSyncClient.getMetastoreSchema(tableName);
+
+    assertEquals(3, schema.size());
+    assertEquals("STRING", schema.get("name"), "column types are upper cased");
+    assertEquals("INT", schema.get("age"));
+    assertEquals("STRING", schema.get("datestr"), "partition keys are merged into the schema");
+  }
+
+  @Test
+  void testGetMetastoreSchema_wrapsGlueFailure() {
+    when(mockAwsGlue.getTable(any(GetTableRequest.class))).thenThrow(new RuntimeException("boom"));
+    HoodieGlueSyncException ex = assertThrows(HoodieGlueSyncException.class,
+        () -> awsGlueSyncClient.getMetastoreSchema("tbl"));
+    assertTrue(ex.getMessage().contains("Fail to get schema for table"));
+  }
+
+  @Test
+  void testGetLastCommitTimeSynced_readsTableParameters() {
+    Map<String, String> parameters = new HashMap<>();
+    parameters.put(HOODIE_LAST_COMMIT_TIME_SYNC, "100");
+    parameters.put(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC, "110");
+    Table withSyncTimes = tableWithParameters("synced", parameters);
+    Table withoutSyncTimes = tableWithParameters("unsynced", new HashMap<>());
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(withSyncTimes).build()))
+        .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(withoutSyncTimes).build()));
+
+    assertEquals("100", awsGlueSyncClient.getLastCommitTimeSynced("synced").get());
+    assertEquals("110", awsGlueSyncClient.getLastCommitCompletionTimeSynced("synced").get());
+    // the table is cached per name, so the second table name triggers the second stubbed response
+    assertFalse(awsGlueSyncClient.getLastCommitTimeSynced("unsynced").isPresent());
+    assertFalse(awsGlueSyncClient.getLastCommitCompletionTimeSynced("unsynced").isPresent());
+    verify(mockAwsGlue, times(2)).getTable(any(GetTableRequest.class));
+  }
+
+  @Test
+  void testGetStorageFieldSchemas_readsFieldsAndDocsFromStorage() {
+    Map<String, FieldSchema> byName = awsGlueSyncClient.getStorageFieldSchemas().stream()
+        .collect(Collectors.toMap(FieldSchema::getName, f -> f));
+
+    assertEquals("int", byName.get("id").getType());
+    assertEquals(GlueTestUtil.ID_FIELD_DOC, byName.get("id").getComment().get());
+    assertEquals("string", byName.get("name").getType());
+    assertEquals(GlueTestUtil.NAME_FIELD_DOC, byName.get("name").getComment().get());
+    assertTrue(byName.containsKey("_hoodie_commit_time"), "metadata fields are part of the storage schema");
+  }
+
+  @Test
+  void testManagePartitionIndexes_disabledDeactivatesFlagAndDropsIndexes() throws Exception {
+    String tableName = "tbl";
+    Map<String, String> parameters = new HashMap<>();
+    parameters.put(GLUE_PARTITION_INDEX_ENABLE, "true");
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, parameters)).build()));
+    ArgumentCaptor<UpdateTableRequest> updateCaptor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(updateCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(partitionIndexDescriptor("idx_one", "datestr"))
+            .build()));
+    ArgumentCaptor<DeletePartitionIndexRequest> deleteCaptor = ArgumentCaptor.forClass(DeletePartitionIndexRequest.class);
+    when(mockAwsGlue.deletePartitionIndex(deleteCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(DeletePartitionIndexResponse.builder().build()));
+
+    awsGlueSyncClient.managePartitionIndexes(tableName);
+
+    assertEquals("false", updateCaptor.getValue().tableInput().parameters().get(GLUE_PARTITION_INDEX_ENABLE),
+        "partition index usage is deactivated when the feature is off");
+    assertEquals(Collections.singletonList("idx_one"), deleteCaptor.getAllValues().stream()
+        .map(DeletePartitionIndexRequest::indexName).collect(Collectors.toList()));
+    verify(mockAwsGlue, never()).createPartitionIndex(any(CreatePartitionIndexRequest.class));
+  }
+
+  @Test
+  void testManagePartitionIndexes_enabledDropsStaleIndexesAndCreatesMissingOnes() throws Exception {
+    String tableName = "tbl";
+    TypedProperties props = GlueTestUtil.getHiveSyncConfig().getProps();
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS_ENABLE.key(), "true");
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS.key(), "datestr;hour,region");
+    awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, mockSts, new HiveSyncConfig(props), GlueTestUtil.getMetaClient());
+
+    // the table has no partition_filtering.enabled parameter, so indexing has to be activated first
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    ArgumentCaptor<UpdateTableRequest> updateCaptor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(updateCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+
+    PartitionIndexDescriptor keptIndex = partitionIndexDescriptor("kept_idx", "datestr", "hour");
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(keptIndex, partitionIndexDescriptor("stale_idx", "old_col"))
+            .build()))
+        // after a drop the index list is re-read
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder()
+            .partitionIndexDescriptorList(keptIndex)
+            .build()));
+    ArgumentCaptor<DeletePartitionIndexRequest> deleteCaptor = ArgumentCaptor.forClass(DeletePartitionIndexRequest.class);
+    when(mockAwsGlue.deletePartitionIndex(deleteCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(DeletePartitionIndexResponse.builder().build()));
+    ArgumentCaptor<CreatePartitionIndexRequest> createCaptor = ArgumentCaptor.forClass(CreatePartitionIndexRequest.class);
+    when(mockAwsGlue.createPartitionIndex(createCaptor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(CreatePartitionIndexResponse.builder().build()));
+
+    awsGlueSyncClient.managePartitionIndexes(tableName);
+
+    assertEquals("true", updateCaptor.getValue().tableInput().parameters().get(GLUE_PARTITION_INDEX_ENABLE));
+    assertEquals(Collections.singletonList("stale_idx"), deleteCaptor.getAllValues().stream()
+        .map(DeletePartitionIndexRequest::indexName).collect(Collectors.toList()),
+        "only the index that is no longer configured is dropped");
+    assertEquals(1, createCaptor.getAllValues().size(), "the already existing index is not recreated");
+    PartitionIndex created = createCaptor.getValue().partitionIndex();
+    assertEquals(Collections.singletonList("region"), created.keys());
+    assertEquals("hudi_managed_[region]", created.indexName());
+    verify(mockAwsGlue, times(2)).getPartitionIndexes(any(GetPartitionIndexesRequest.class));
+  }
+
+  @Test
+  void testParsePartitionsIndexConfig_keepsOnlyTheFirstThreeIndexes() {
+    TypedProperties props = GlueTestUtil.getHiveSyncConfig().getProps();
+    props.setProperty(GlueCatalogSyncClientConfig.META_SYNC_PARTITION_INDEX_FIELDS.key(), "a;b,c,d,e");
+    awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, mockSts, new HiveSyncConfig(props), GlueTestUtil.getMetaClient());
+
+    assertEquals(Arrays.asList(Arrays.asList("a", "b"), Collections.singletonList("c"), Collections.singletonList("d")),
+        awsGlueSyncClient.parsePartitionsIndexConfig(), "glue supports at most three partition indexes");
+  }
+
+  @Test
+  void testUpdateLastCommitTimeSynced_writesTimelineInstantToTableParameters() {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    ArgumentCaptor<UpdateTableRequest> captor = ArgumentCaptor.forClass(UpdateTableRequest.class);
+    when(mockAwsGlue.updateTable(captor.capture()))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(GetPartitionIndexesResponse.builder().build()));
+
+    awsGlueSyncClient.updateLastCommitTimeSynced(tableName);
+
+    Map<String, String> parameters = captor.getValue().tableInput().parameters();
+    assertEquals(GlueTestUtil.INSTANT_TIME, parameters.get(HOODIE_LAST_COMMIT_TIME_SYNC),
+        "the last instant of the active timeline is synced");
+    assertEquals(GlueTestUtil.COMPLETION_TIME, parameters.get(HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC),
+        "the completion time of that instant is synced alongside it");
+    assertTrue(captor.getValue().skipArchive(), "table archiving is skipped by default");
+  }
+
+  /**
+   * An indexation already in flight surfaces as an {@link ExecutionException}, anything else lands in the
+   * catch-all. Neither may fail the commit-time sync.
+   */
+  @ParameterizedTest
+  @ValueSource(booleans = {true, false})
+  void testUpdateLastCommitTimeSynced_partitionIndexFailureDoesNotFailTheSync(boolean asExecutionFailure) throws Exception {
+    String tableName = "tbl";
+    when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(
+            GetTableResponse.builder().table(tableWithParameters(tableName, new HashMap<>())).build()));
+    when(mockAwsGlue.updateTable(any(UpdateTableRequest.class)))
+        .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+    if (asExecutionFailure) {
+      CompletableFuture<GetPartitionIndexesResponse> failed = mock(CompletableFuture.class);
+      when(failed.get()).thenThrow(new ExecutionException(new RuntimeException("indexing in progress")));
+      when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class))).thenReturn(failed);
+    } else {
+      when(mockAwsGlue.getPartitionIndexes(any(GetPartitionIndexesRequest.class))).thenThrow(new RuntimeException("boom"));
+    }
+
+    awsGlueSyncClient.updateLastCommitTimeSynced(tableName);
+
+    verify(mockAwsGlue, times(1)).updateTable(any(UpdateTableRequest.class));

Review Comment:
   The two arms differ only in the log line (`An indexation process is currently running.` vs `Something went wrong with partition index`); neither rethrows, so there is nothing observable to tell them apart. Kept both values so each catch line runs, renamed the parameter to `throughExecutionException`, and the javadoc now says the effect is the same by design. Done in bb7fccea3715.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org