You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "niranjan-1408 (via GitHub)" <gi...@apache.org> on 2026/08/27 00:18:56 UTC
[PR] fix(sync): drop the duplicate UpdateTable in updateTableSchema [hudi]
niranjan-1408 opened a new pull request, #19762:
URL: https://github.com/apache/hudi/pull/19762
Stacked on #19761, please review that one first. This PR shows both commits until it merges.
### Describe the issue this Pull Request addresses
`updateTableSchema` issues the same `UpdateTable` request object twice. The second call sits outside the `if (cascade)` block, so it fires unconditionally, including on non-partitioned tables. Nothing between the two calls mutates the request and the cascade branch writes partitions rather than the table, so the second call is a no-op that Glue still counts as a new table version. Every schema change therefore consumes two table versions where one would do.
### Summary and Changelog
Removes the second `awsGlue.updateTable(request).get()`. The first is kept rather than the second because the cascade re-reads the table to source the columns it propagates: with the update first it sees the new columns, whereas keeping only the second would have it propagate stale ones. Adds a test asserting exactly one `UpdateTable` per schema change.
### Impact
Halves the Glue table-version churn from this method. No behaviour change beyond the removed redundant write.
### Risk Level
low
One deleted line. The added test fails against unfixed code with `TooManyActualInvocations: Wanted 1 time but was 2`.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sync): drop the duplicate UpdateTable in updateTableSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19762:
URL: https://github.com/apache/hudi/pull/19762#issuecomment-5435085243
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16603",
"triggerID" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"triggerType" : "PUSH"
}, {
"hash" : "c81929ccf8df794976279f6f469161b57f9e1455",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16609",
"triggerID" : "c81929ccf8df794976279f6f469161b57f9e1455",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c81929ccf8df794976279f6f469161b57f9e1455 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16609)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sync): drop the duplicate UpdateTable in updateTableSchema [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #19762:
URL: https://github.com/apache/hudi/pull/19762
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sync): drop the duplicate UpdateTable in updateTableSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19762:
URL: https://github.com/apache/hudi/pull/19762#issuecomment-5446863921
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16603",
"triggerID" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"triggerType" : "PUSH"
}, {
"hash" : "c81929ccf8df794976279f6f469161b57f9e1455",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16609",
"triggerID" : "c81929ccf8df794976279f6f469161b57f9e1455",
"triggerType" : "PUSH"
}, {
"hash" : "eec2dd73ee3164b46965d25900c1832e13c53e43",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16643",
"triggerID" : "eec2dd73ee3164b46965d25900c1832e13c53e43",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c81929ccf8df794976279f6f469161b57f9e1455 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16609)
* eec2dd73ee3164b46965d25900c1832e13c53e43 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16643)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sync): drop the duplicate UpdateTable in updateTableSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19762:
URL: https://github.com/apache/hudi/pull/19762#issuecomment-5446821316
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16603",
"triggerID" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"triggerType" : "PUSH"
}, {
"hash" : "c81929ccf8df794976279f6f469161b57f9e1455",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16609",
"triggerID" : "c81929ccf8df794976279f6f469161b57f9e1455",
"triggerType" : "PUSH"
}, {
"hash" : "eec2dd73ee3164b46965d25900c1832e13c53e43",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eec2dd73ee3164b46965d25900c1832e13c53e43",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c81929ccf8df794976279f6f469161b57f9e1455 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16609)
* eec2dd73ee3164b46965d25900c1832e13c53e43 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sync): drop the duplicate UpdateTable in updateTableSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19762:
URL: https://github.com/apache/hudi/pull/19762#issuecomment-5434347049
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16603",
"triggerID" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"triggerType" : "PUSH"
}, {
"hash" : "c81929ccf8df794976279f6f469161b57f9e1455",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16609",
"triggerID" : "c81929ccf8df794976279f6f469161b57f9e1455",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fdc17751d00dee27ae3169313ead62dc76394cc0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16603)
* c81929ccf8df794976279f6f469161b57f9e1455 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16609)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sync): drop the duplicate UpdateTable in updateTableSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19762:
URL: https://github.com/apache/hudi/pull/19762#issuecomment-5432750059
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16603",
"triggerID" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fdc17751d00dee27ae3169313ead62dc76394cc0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16603)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sync): drop the duplicate UpdateTable in updateTableSchema [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19762:
URL: https://github.com/apache/hudi/pull/19762#issuecomment-5439380991
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19762?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `83.33333%` with `4 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 77.80%. Comparing base ([`ad221c4`](https://app.codecov.io/gh/apache/hudi/commit/ad221c449ecf30967d21ff352b1862e1e3f8f4ef?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`c81929c`](https://app.codecov.io/gh/apache/hudi/commit/c81929ccf8df794976279f6f469161b57f9e1455?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19762?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19762?src=pr&el=tree&filepath=hudi-aws%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Faws%2Fsync%2FAWSGlueCatalogSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1hd3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvYXdzL3N5bmMvQVdTR2x1ZUNhdGFsb2dTeW5jQ2xpZW50LmphdmE=) | 83.33% | [3 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19762?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19762 +/- ##
============================================
- Coverage 77.97% 77.80% -0.17%
+ Complexity 33466 33410 -56
============================================
Files 2539 2539
Lines 140955 140979 +24
Branches 17013 17018 +5
============================================
- Hits 109911 109692 -219
- Misses 23386 23628 +242
- Partials 7658 7659 +1
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/19762/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.50% <ø> (+0.02%)` | :arrow_up: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.05% <ø> (-0.03%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.62% <ø> (-0.03%)` | :arrow_down: |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `72.38% <ø> (ø)` | |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `72.11% <ø> (-2.26%)` | :arrow_down: |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.06% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `69.23% <ø> (-0.02%)` | :arrow_down: |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.54% <ø> (-0.03%)` | :arrow_down: |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.85% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.44% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.77% <83.33%> (+1.49%)` | :arrow_up: |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19762/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19762/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.73% <ø> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19762?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19762?src=pr&el=tree&filepath=hudi-aws%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Faws%2Fsync%2FAWSGlueCatalogSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1hd3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvYXdzL3N5bmMvQVdTR2x1ZUNhdGFsb2dTeW5jQ2xpZW50LmphdmE=) | `55.93% <83.33%> (+5.03%)` | :arrow_up: |
... and [32 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19762/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sync): drop the duplicate UpdateTable in updateTableSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19762:
URL: https://github.com/apache/hudi/pull/19762#issuecomment-5433602464
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16603",
"triggerID" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fdc17751d00dee27ae3169313ead62dc76394cc0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16603)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sync): drop the duplicate UpdateTable in updateTableSchema [hudi]
Posted by "niranjan-1408 (via GitHub)" <gi...@apache.org>.
niranjan-1408 commented on code in PR #19762:
URL: https://github.com/apache/hudi/pull/19762#discussion_r3868604941
##########
hudi-aws/src/test/java/org/apache/hudi/aws/sync/TestAWSGlueSyncClient.java:
##########
@@ -909,4 +912,81 @@ private CompletableFuture<GetTableResponse> getTableWithDefaultProps(String tabl
.build();
return CompletableFuture.completedFuture(response);
}
+
+ @ParameterizedTest
+ @ValueSource(strings = {"2024-01-15", "datestr=2024-01-15", "2024/01/15"})
+ void testUpdateTableSchema_cascadePreservesGlueRecordedPartitionLocation(String partitionDir) {
+ String tableName = GlueTestUtil.TABLE_NAME;
+ String basePath = GlueTestUtil.getHiveSyncConfig().getString(META_SYNC_BASE_PATH);
+ String partitionLocation = new StoragePath(basePath, partitionDir).toString();
+
+ Table table = Table.builder()
+ .name(tableName)
+ .databaseName(GlueTestUtil.DB_NAME)
+ .storageDescriptor(StorageDescriptor.builder()
+ .location(basePath)
+ .columns(Column.builder().name("name").type("string").build())
+ .build())
+ .partitionKeys(Column.builder().name("datestr").type("string").build())
+ .build();
+ when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+ .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(table).build()));
+ when(mockAwsGlue.updateTable(any(UpdateTableRequest.class)))
+ .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+
+ software.amazon.awssdk.services.glue.model.Partition gluePartition =
+ software.amazon.awssdk.services.glue.model.Partition.builder()
+ .values("2024-01-15")
+ .storageDescriptor(StorageDescriptor.builder().location(partitionLocation).build())
+ .build();
+ when(mockAwsGlue.getPartitions(any(GetPartitionsRequest.class)))
+ .thenReturn(CompletableFuture.completedFuture(
+ GetPartitionsResponse.builder().partitions(gluePartition).nextToken(null).build()));
+
+ ArgumentCaptor<BatchUpdatePartitionRequest> captor = ArgumentCaptor.forClass(BatchUpdatePartitionRequest.class);
+ when(mockAwsGlue.batchUpdatePartition(captor.capture()))
+ .thenReturn(CompletableFuture.completedFuture(BatchUpdatePartitionResponse.builder().build()));
+
+ HoodieSchema schema = GlueTestUtil.getSimpleSchema();
+ SchemaDifference schemaDiff = SchemaDifference.newBuilder(schema, new HashMap<>())
+ .updateTableColumn("name", "string")
+ .build();
+
+ awsGlueSyncClient.updateTableSchema(tableName, schema, schemaDiff);
+
+ List<BatchUpdatePartitionRequestEntry> entries = captor.getValue().entries();
+ assertEquals(1, entries.size());
+ assertEquals(partitionLocation, entries.get(0).partitionInput().storageDescriptor().location());
+ assertEquals(Collections.singletonList("2024-01-15"), entries.get(0).partitionValueList());
+ assertEquals(table.storageDescriptor().columns(),
+ entries.get(0).partitionInput().storageDescriptor().columns());
Review Comment:
done in [0bfc756](https://github.com/apache/hudi/pull/19762/commits/0bfc756782857e6af91d7ad5b5c580294526a2c9)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sync): drop the duplicate UpdateTable in updateTableSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19762:
URL: https://github.com/apache/hudi/pull/19762#issuecomment-5447542224
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16603",
"triggerID" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"triggerType" : "PUSH"
}, {
"hash" : "c81929ccf8df794976279f6f469161b57f9e1455",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16609",
"triggerID" : "c81929ccf8df794976279f6f469161b57f9e1455",
"triggerType" : "PUSH"
}, {
"hash" : "eec2dd73ee3164b46965d25900c1832e13c53e43",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16643",
"triggerID" : "eec2dd73ee3164b46965d25900c1832e13c53e43",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eec2dd73ee3164b46965d25900c1832e13c53e43 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16643)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sync): drop the duplicate UpdateTable in updateTableSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19762:
URL: https://github.com/apache/hudi/pull/19762#issuecomment-5432710229
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fdc17751d00dee27ae3169313ead62dc76394cc0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sync): drop the duplicate UpdateTable in updateTableSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19762:
URL: https://github.com/apache/hudi/pull/19762#issuecomment-5434310529
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16603",
"triggerID" : "fdc17751d00dee27ae3169313ead62dc76394cc0",
"triggerType" : "PUSH"
}, {
"hash" : "c81929ccf8df794976279f6f469161b57f9e1455",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c81929ccf8df794976279f6f469161b57f9e1455",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fdc17751d00dee27ae3169313ead62dc76394cc0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16603)
* c81929ccf8df794976279f6f469161b57f9e1455 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sync): drop the duplicate UpdateTable in updateTableSchema [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19762:
URL: https://github.com/apache/hudi/pull/19762#discussion_r3868393932
##########
hudi-aws/src/test/java/org/apache/hudi/aws/sync/TestAWSGlueSyncClient.java:
##########
@@ -909,4 +912,81 @@ private CompletableFuture<GetTableResponse> getTableWithDefaultProps(String tabl
.build();
return CompletableFuture.completedFuture(response);
}
+
+ @ParameterizedTest
+ @ValueSource(strings = {"2024-01-15", "datestr=2024-01-15", "2024/01/15"})
+ void testUpdateTableSchema_cascadePreservesGlueRecordedPartitionLocation(String partitionDir) {
+ String tableName = GlueTestUtil.TABLE_NAME;
+ String basePath = GlueTestUtil.getHiveSyncConfig().getString(META_SYNC_BASE_PATH);
+ String partitionLocation = new StoragePath(basePath, partitionDir).toString();
+
+ Table table = Table.builder()
+ .name(tableName)
+ .databaseName(GlueTestUtil.DB_NAME)
+ .storageDescriptor(StorageDescriptor.builder()
+ .location(basePath)
+ .columns(Column.builder().name("name").type("string").build())
+ .build())
+ .partitionKeys(Column.builder().name("datestr").type("string").build())
+ .build();
+ when(mockAwsGlue.getTable(any(GetTableRequest.class)))
+ .thenReturn(CompletableFuture.completedFuture(GetTableResponse.builder().table(table).build()));
+ when(mockAwsGlue.updateTable(any(UpdateTableRequest.class)))
+ .thenReturn(CompletableFuture.completedFuture(UpdateTableResponse.builder().build()));
+
+ software.amazon.awssdk.services.glue.model.Partition gluePartition =
+ software.amazon.awssdk.services.glue.model.Partition.builder()
+ .values("2024-01-15")
+ .storageDescriptor(StorageDescriptor.builder().location(partitionLocation).build())
+ .build();
+ when(mockAwsGlue.getPartitions(any(GetPartitionsRequest.class)))
+ .thenReturn(CompletableFuture.completedFuture(
+ GetPartitionsResponse.builder().partitions(gluePartition).nextToken(null).build()));
+
+ ArgumentCaptor<BatchUpdatePartitionRequest> captor = ArgumentCaptor.forClass(BatchUpdatePartitionRequest.class);
+ when(mockAwsGlue.batchUpdatePartition(captor.capture()))
+ .thenReturn(CompletableFuture.completedFuture(BatchUpdatePartitionResponse.builder().build()));
+
+ HoodieSchema schema = GlueTestUtil.getSimpleSchema();
+ SchemaDifference schemaDiff = SchemaDifference.newBuilder(schema, new HashMap<>())
+ .updateTableColumn("name", "string")
+ .build();
+
+ awsGlueSyncClient.updateTableSchema(tableName, schema, schemaDiff);
+
+ List<BatchUpdatePartitionRequestEntry> entries = captor.getValue().entries();
+ assertEquals(1, entries.size());
+ assertEquals(partitionLocation, entries.get(0).partitionInput().storageDescriptor().location());
+ assertEquals(Collections.singletonList("2024-01-15"), entries.get(0).partitionValueList());
+ assertEquals(table.storageDescriptor().columns(),
+ entries.get(0).partitionInput().storageDescriptor().columns());
Review Comment:
[P2] Please make this assertion verify the updated columns, not the pre-update table schema. `getSimpleSchema()` contains both `id` and `name`, but the `getTable` stub always returns `table`, whose descriptor contains only `name`; consequently the cascade currently propagates that stale list and this test explicitly accepts it. This misses the ordering invariant used to justify keeping the first `UpdateTable` call. Could the first `getTable` return the original table and the cascade re-read return a table built from the captured update request (or otherwise containing the new columns), then assert those new columns here? An `InOrder` check for `updateTable` before `batchUpdatePartition` would cover the same dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org