You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "vamsikarnika (via GitHub)" <gi...@apache.org> on 2025/05/22 16:45:52 UTC
[PR] [HUDI-8449] Fix deletion of record from FILES partition on empty file… [hudi]
vamsikarnika opened a new pull request, #13346:
URL: https://github.com/apache/hudi/pull/13346
…s list
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "vamsikarnika (via GitHub)" <gi...@apache.org>.
vamsikarnika commented on code in PR #13346:
URL: https://github.com/apache/hudi/pull/13346#discussion_r2104009586
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java:
##########
@@ -1168,6 +1177,9 @@ public HoodieCleanMetadata doClean(String commitTime, Map<String, Integer> parti
testTableState = testTableState.createTestTableStateForCleaner(commitTime, entry.getKey(), entry.getValue());
deleteFilesInPartition(entry.getKey(), entry.getValue());
}
+
+ testTableState = testTableState.createTestTableStateForCleaner(commitTime, partitionsToDelete);
Review Comment:
Not exactly recreating the table state, but populating partitions to clean info to the table state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "vamsikarnika (via GitHub)" <gi...@apache.org>.
vamsikarnika commented on code in PR #13346:
URL: https://github.com/apache/hudi/pull/13346#discussion_r2107592955
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedTableMetadata.java:
##########
@@ -411,6 +411,61 @@ public void testRepeatedCleanActionsWithMetadataTableEnabled(final HoodieTableTy
assertEquals(tableVersion, finalTableVersion.versionCode());
}
+ @ParameterizedTest
+ @CsvSource({"COPY_ON_WRITE,6", "COPY_ON_WRITE,8", "MERGE_ON_READ,6", "MERGE_ON_READ,8"})
+ void testDeletePartitionKeyOnCleanPartition(final HoodieTableType tableType, int tableVersion) throws Exception {
+ initPath();
+ writeConfig = getWriteConfigBuilder(true, true, false)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder()
+ .enable(true)
+ .withMaxNumDeltaCommitsBeforeCompaction(6)
+ .build())
+ .build();
+ writeConfig.setValue(HoodieWriteConfig.WRITE_TABLE_VERSION, String.valueOf(tableVersion));
+ init(tableType, writeConfig);
+ String partition1 = "p1";
+ // Simulate two bulk insert operations adding two data files in partition "p1"
+ String instant1 = metaClient.createNewInstantTime();
+ testTable.doWriteOperation(instant1, BULK_INSERT, emptyList(), Collections.singletonList(partition1), 1);
+ String instant2 = metaClient.createNewInstantTime();
+ testTable.doWriteOperation(instant2, BULK_INSERT, emptyList(), Collections.singletonList(partition1), 1);
+
+ String partition2 = "p2";
+ testTable.doWriteOperation(
+ metaClient.createNewInstantTime(), BULK_INSERT, emptyList(), Collections.singletonList(partition2), 1);
+
+ final HoodieTableMetaClient metadataMetaClient = createMetaClient(metadataTableBasePath);
+ metadataMetaClient.reloadActiveTimeline();
+
+ assertEquals(0, getNumCompactions(metadataMetaClient));
+
+ String cleanInstant = metaClient.createNewInstantTime();
+ HoodieCleanMetadata cleanMetadata = testTable.doCleanBasedOnPartitions(cleanInstant, Collections.singletonList(partition1));
+
+ while (getNumCompactions(metadataMetaClient) == 0) {
+ // Write until the compaction happens in the metadata table
Review Comment:
During compaction, deleted records won’t be written to the compacted file, so there won’t be any entry for that key. Without compaction, the getRecordsByKeyPrefixes method will still read the deleted record from the log files — it does apply pre-combine and marks the record as deleted, but entry for the record will still be present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2919665774
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
}, {
"hash" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671",
"triggerID" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"triggerType" : "PUSH"
}, {
"hash" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5672",
"triggerID" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"triggerType" : "PUSH"
}, {
"hash" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5837",
"triggerID" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"triggerType" : "PUSH"
}, {
"hash" : "23136e2899254b6dca123b1dd93de74a32e24f4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "23136e2899254b6dca123b1dd93de74a32e24f4d",
"triggerType" : "PUSH"
}, {
"hash" : "9ce28229ee1e07a02b63700e778ade51c8bb7778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5838",
"triggerID" : "9ce28229ee1e07a02b63700e778ade51c8bb7778",
"triggerType" : "PUSH"
}, {
"hash" : "f05a8cd8e1916737a6ae24e215b3c26a4a7a3a80",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5848",
"triggerID" : "f05a8cd8e1916737a6ae24e215b3c26a4a7a3a80",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 23136e2899254b6dca123b1dd93de74a32e24f4d UNKNOWN
* f05a8cd8e1916737a6ae24e215b3c26a4a7a3a80 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5848)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2902600329
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 663dabfa3ee641537b70ee1b123a144d5d981735 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649)
* 62e625b73e1bb222c86ef95014df6bada2a72d2c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2903519934
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
}, {
"hash" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671",
"triggerID" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2918406811
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
}, {
"hash" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671",
"triggerID" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"triggerType" : "PUSH"
}, {
"hash" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5672",
"triggerID" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"triggerType" : "PUSH"
}, {
"hash" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2b90a5f54fb488a570af340ffd53e0dcfe912dc4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5672)
* 4b475f230c8d843dabd917f5225440e0f7ee178c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2902080551
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6e7823f4a3535ff065549d1c0a1a089a00fb258 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648)
* 663dabfa3ee641537b70ee1b123a144d5d981735 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "vamsikarnika (via GitHub)" <gi...@apache.org>.
vamsikarnika commented on code in PR #13346:
URL: https://github.com/apache/hudi/pull/13346#discussion_r2104005395
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java:
##########
@@ -1154,11 +1163,11 @@ public HoodieReplaceCommitMetadata doCluster(String commitTime, Map<String, List
return replaceMetadata;
}
- public HoodieCleanMetadata doClean(String commitTime, Map<String, Integer> partitionFileCountsToDelete) throws IOException {
- return doClean(commitTime, partitionFileCountsToDelete, Collections.emptyMap());
+ public HoodieCleanMetadata doClean(String commitTime, Map<String, Integer> partitionFileCountsToDelete, List<String> partitionsToBeDeleted) throws IOException {
+ return doClean(commitTime, partitionFileCountsToDelete, partitionsToBeDeleted, Collections.emptyMap());
Review Comment:
Added it as a new function
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java:
##########
@@ -1154,11 +1163,11 @@ public HoodieReplaceCommitMetadata doCluster(String commitTime, Map<String, List
return replaceMetadata;
}
- public HoodieCleanMetadata doClean(String commitTime, Map<String, Integer> partitionFileCountsToDelete) throws IOException {
- return doClean(commitTime, partitionFileCountsToDelete, Collections.emptyMap());
+ public HoodieCleanMetadata doClean(String commitTime, Map<String, Integer> partitionFileCountsToDelete, List<String> partitionsToBeDeleted) throws IOException {
+ return doClean(commitTime, partitionFileCountsToDelete, partitionsToBeDeleted, Collections.emptyMap());
}
- public HoodieCleanMetadata doClean(String commitTime, Map<String, Integer> partitionFileCountsToDelete, Map<String, String> extraMetadata) throws IOException {
+ public HoodieCleanMetadata doClean(String commitTime, Map<String, Integer> partitionFileCountsToDelete, List<String> partitionsToDelete, Map<String, String> extraMetadata) throws IOException {
Review Comment:
Fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13346:
URL: https://github.com/apache/hudi/pull/13346#discussion_r2103886397
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java:
##########
@@ -467,11 +468,19 @@ public HoodieTestTable addClean(String instantTime) throws IOException {
public Pair<HoodieCleanerPlan, HoodieCleanMetadata> getHoodieCleanMetadata(String commitTime, HoodieTestTableState testTableState) {
HoodieCleanerPlan cleanerPlan = new HoodieCleanerPlan(new HoodieActionInstant(commitTime, CLEAN_ACTION, EMPTY_STRING),
- EMPTY_STRING, EMPTY_STRING, new HashMap<>(), CleanPlanV2MigrationHandler.VERSION, new HashMap<>(), new ArrayList<>(), Collections.EMPTY_MAP);
+ EMPTY_STRING, EMPTY_STRING, new HashMap<>(), CleanPlanV2MigrationHandler.VERSION, new HashMap<>(), testTableState.getPartitionsToDeleteForCleaner(commitTime), Collections.EMPTY_MAP);
Review Comment:
should we reuse `testTableState.getPartitionsToDeleteForCleaner(commitTime)` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2902740865
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62e625b73e1bb222c86ef95014df6bada2a72d2c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2918431827
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
}, {
"hash" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671",
"triggerID" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"triggerType" : "PUSH"
}, {
"hash" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5672",
"triggerID" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"triggerType" : "PUSH"
}, {
"hash" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5837",
"triggerID" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"triggerType" : "PUSH"
}, {
"hash" : "23136e2899254b6dca123b1dd93de74a32e24f4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "23136e2899254b6dca123b1dd93de74a32e24f4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b475f230c8d843dabd917f5225440e0f7ee178c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5837)
* 23136e2899254b6dca123b1dd93de74a32e24f4d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2918528304
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
}, {
"hash" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671",
"triggerID" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"triggerType" : "PUSH"
}, {
"hash" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5672",
"triggerID" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"triggerType" : "PUSH"
}, {
"hash" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5837",
"triggerID" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"triggerType" : "PUSH"
}, {
"hash" : "23136e2899254b6dca123b1dd93de74a32e24f4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "23136e2899254b6dca123b1dd93de74a32e24f4d",
"triggerType" : "PUSH"
}, {
"hash" : "9ce28229ee1e07a02b63700e778ade51c8bb7778",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5838",
"triggerID" : "9ce28229ee1e07a02b63700e778ade51c8bb7778",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b475f230c8d843dabd917f5225440e0f7ee178c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5837)
* 23136e2899254b6dca123b1dd93de74a32e24f4d UNKNOWN
* 9ce28229ee1e07a02b63700e778ade51c8bb7778 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5838)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2919364930
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
}, {
"hash" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671",
"triggerID" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"triggerType" : "PUSH"
}, {
"hash" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5672",
"triggerID" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"triggerType" : "PUSH"
}, {
"hash" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5837",
"triggerID" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"triggerType" : "PUSH"
}, {
"hash" : "23136e2899254b6dca123b1dd93de74a32e24f4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "23136e2899254b6dca123b1dd93de74a32e24f4d",
"triggerType" : "PUSH"
}, {
"hash" : "9ce28229ee1e07a02b63700e778ade51c8bb7778",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5838",
"triggerID" : "9ce28229ee1e07a02b63700e778ade51c8bb7778",
"triggerType" : "PUSH"
}, {
"hash" : "f05a8cd8e1916737a6ae24e215b3c26a4a7a3a80",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5848",
"triggerID" : "f05a8cd8e1916737a6ae24e215b3c26a4a7a3a80",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 23136e2899254b6dca123b1dd93de74a32e24f4d UNKNOWN
* 9ce28229ee1e07a02b63700e778ade51c8bb7778 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5838)
* f05a8cd8e1916737a6ae24e215b3c26a4a7a3a80 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5848)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13346:
URL: https://github.com/apache/hudi/pull/13346#discussion_r2109772357
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedTableMetadata.java:
##########
@@ -411,6 +411,61 @@ public void testRepeatedCleanActionsWithMetadataTableEnabled(final HoodieTableTy
assertEquals(tableVersion, finalTableVersion.versionCode());
}
+ @ParameterizedTest
+ @CsvSource({"COPY_ON_WRITE,6", "COPY_ON_WRITE,8", "MERGE_ON_READ,6", "MERGE_ON_READ,8"})
+ void testDeletePartitionKeyOnCleanPartition(final HoodieTableType tableType, int tableVersion) throws Exception {
Review Comment:
we dont' need to make this run for 6 combinations. lets not try to add more parametrized test if there is no need.
this is just testing regular updates to FILES partition in MDT.
just 1 version would suffice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13346:
URL: https://github.com/apache/hudi/pull/13346#discussion_r2104984735
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedTableMetadata.java:
##########
@@ -411,6 +411,61 @@ public void testRepeatedCleanActionsWithMetadataTableEnabled(final HoodieTableTy
assertEquals(tableVersion, finalTableVersion.versionCode());
}
+ @ParameterizedTest
+ @CsvSource({"COPY_ON_WRITE,6", "COPY_ON_WRITE,8", "MERGE_ON_READ,6", "MERGE_ON_READ,8"})
+ void testDeletePartitionKeyOnCleanPartition(final HoodieTableType tableType, int tableVersion) throws Exception {
+ initPath();
+ writeConfig = getWriteConfigBuilder(true, true, false)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder()
+ .enable(true)
+ .withMaxNumDeltaCommitsBeforeCompaction(6)
+ .build())
+ .build();
+ writeConfig.setValue(HoodieWriteConfig.WRITE_TABLE_VERSION, String.valueOf(tableVersion));
+ init(tableType, writeConfig);
+ String partition1 = "p1";
+ // Simulate two bulk insert operations adding two data files in partition "p1"
+ String instant1 = metaClient.createNewInstantTime();
+ testTable.doWriteOperation(instant1, BULK_INSERT, emptyList(), Collections.singletonList(partition1), 1);
+ String instant2 = metaClient.createNewInstantTime();
+ testTable.doWriteOperation(instant2, BULK_INSERT, emptyList(), Collections.singletonList(partition1), 1);
+
+ String partition2 = "p2";
+ testTable.doWriteOperation(
+ metaClient.createNewInstantTime(), BULK_INSERT, emptyList(), Collections.singletonList(partition2), 1);
+
+ final HoodieTableMetaClient metadataMetaClient = createMetaClient(metadataTableBasePath);
+ metadataMetaClient.reloadActiveTimeline();
+
+ assertEquals(0, getNumCompactions(metadataMetaClient));
+
+ String cleanInstant = metaClient.createNewInstantTime();
+ HoodieCleanMetadata cleanMetadata = testTable.doCleanBasedOnPartitions(cleanInstant, Collections.singletonList(partition1));
+
+ while (getNumCompactions(metadataMetaClient) == 0) {
+ // Write until the compaction happens in the metadata table
Review Comment:
Why do we need a compaction operation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13346:
URL: https://github.com/apache/hudi/pull/13346#discussion_r2103890161
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java:
##########
@@ -1517,6 +1538,12 @@ static class HoodieTestTableState {
*/
Map<String, Map<String, List<Pair<String, Integer[]>>>> commitsToPartitionToLogFileInfoStats = new HashMap<>();
+ /**
+ * Map<commitTime, List<partitionPath>>
+ * Used to build commit metadata for partitions that got deleted
Review Comment:
commit metadata or clean metadata?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2903562775
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
}, {
"hash" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671",
"triggerID" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"triggerType" : "PUSH"
}, {
"hash" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5672",
"triggerID" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671)
* 2b90a5f54fb488a570af340ffd53e0dcfe912dc4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5672)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2918411098
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
}, {
"hash" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671",
"triggerID" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"triggerType" : "PUSH"
}, {
"hash" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5672",
"triggerID" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"triggerType" : "PUSH"
}, {
"hash" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5837",
"triggerID" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2b90a5f54fb488a570af340ffd53e0dcfe912dc4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5672)
* 4b475f230c8d843dabd917f5225440e0f7ee178c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5837)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "vamsikarnika (via GitHub)" <gi...@apache.org>.
vamsikarnika commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2919529912
> we have an existing functional tests to test deletePartitions. can we use that to add more validations.
Yes, wasn't aware of this existing test before. I've made changes to reuse the existing test to do the validations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2918471993
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
}, {
"hash" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671",
"triggerID" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"triggerType" : "PUSH"
}, {
"hash" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5672",
"triggerID" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"triggerType" : "PUSH"
}, {
"hash" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5837",
"triggerID" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"triggerType" : "PUSH"
}, {
"hash" : "23136e2899254b6dca123b1dd93de74a32e24f4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "23136e2899254b6dca123b1dd93de74a32e24f4d",
"triggerType" : "PUSH"
}, {
"hash" : "9ce28229ee1e07a02b63700e778ade51c8bb7778",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ce28229ee1e07a02b63700e778ade51c8bb7778",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b475f230c8d843dabd917f5225440e0f7ee178c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5837)
* 23136e2899254b6dca123b1dd93de74a32e24f4d UNKNOWN
* 9ce28229ee1e07a02b63700e778ade51c8bb7778 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2903179562
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62e625b73e1bb222c86ef95014df6bada2a72d2c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655)
* e94f974d6db090699471c794e1d24b7a52d94eb3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13346:
URL: https://github.com/apache/hudi/pull/13346#discussion_r2103893862
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java:
##########
@@ -1536,10 +1563,26 @@ HoodieTestTableState createTestTableStateForCleaner(String commitTime, String pa
return this;
}
+ HoodieTestTableState createTestTableStateForCleaner(String commitTime, List<String> partitionsToClean) {
Review Comment:
createTestTableStateForCleaner -> createTestTableStateForPartitionsToClean
createTestTableStateForCleaner -> createTestTableStateForFilesToClean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2903177555
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62e625b73e1bb222c86ef95014df6bada2a72d2c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655)
* e94f974d6db090699471c794e1d24b7a52d94eb3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "vamsikarnika (via GitHub)" <gi...@apache.org>.
vamsikarnika commented on code in PR #13346:
URL: https://github.com/apache/hudi/pull/13346#discussion_r2103751380
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -277,14 +277,25 @@ public static HoodieRecord<HoodieMetadataPayload> createPartitionListRecord(Coll
public static HoodieRecord<HoodieMetadataPayload> createPartitionFilesRecord(String partition,
Map<String, Long> filesAdded,
List<String> filesDeleted) {
+ return createPartitionFilesRecord(partition, filesAdded, filesDeleted, false);
+ }
+
+ public static HoodieRecord<HoodieMetadataPayload> createPartitionFilesRecord(String partition,
+ Map<String, Long> filesAdded,
+ List<String> filesDeleted,
+ boolean isPartitionDeleted) {
String partitionIdentifier = getPartitionIdentifierForFilesPartition(partition);
+ HoodieKey key = new HoodieKey(partitionIdentifier, MetadataPartitionType.FILES.getPartitionPath());
+ if (isPartitionDeleted) {
+ return new HoodieAvroRecord<>(key, new HoodieMetadataPayload(Option.empty()));
+ }
Review Comment:
Updated the logic to remove the recordKey from files partitions, if merge result is empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "vamsikarnika (via GitHub)" <gi...@apache.org>.
vamsikarnika commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2913174984
> looks good overall. 1 ask. Can you find one spark functional tests where we do deletePartition operation, and post that, try to query all files for the partition of interest (which should go via MDT). do not write any new test. but just add additional validation to an existing test.
Found this existing test case, which does delete partitions on the data table and validates metadata table by comparing the Filesystem backed table metadata. I think this should cover the scenario you mentioned?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2913434212
btw, did you try https://github.com/apache/hudi/blob/60eaefd1eb08bd29a0d6dd01af3ff7b9eeb5f3dc/hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestHoodieBackedMetadata.java#L771
this already tests deletePartition
So, why write a new test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "vamsikarnika (via GitHub)" <gi...@apache.org>.
vamsikarnika commented on code in PR #13346:
URL: https://github.com/apache/hudi/pull/13346#discussion_r2104007885
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java:
##########
@@ -1517,6 +1538,12 @@ static class HoodieTestTableState {
*/
Map<String, Map<String, List<Pair<String, Integer[]>>>> commitsToPartitionToLogFileInfoStats = new HashMap<>();
+ /**
+ * Map<commitTime, List<partitionPath>>
+ * Used to build commit metadata for partitions that got deleted
Review Comment:
It's clean metadata. Fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2903558080
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
}, {
"hash" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671",
"triggerID" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"triggerType" : "PUSH"
}, {
"hash" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671)
* 2b90a5f54fb488a570af340ffd53e0dcfe912dc4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2903359970
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
}, {
"hash" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e94f974d6db090699471c794e1d24b7a52d94eb3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668)
* 8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2903296946
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e94f974d6db090699471c794e1d24b7a52d94eb3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty file… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2901919075
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6e7823f4a3535ff065549d1c0a1a089a00fb258 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty file… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2901912919
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6e7823f4a3535ff065549d1c0a1a089a00fb258 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "vamsikarnika (via GitHub)" <gi...@apache.org>.
vamsikarnika commented on code in PR #13346:
URL: https://github.com/apache/hudi/pull/13346#discussion_r2107592955
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedTableMetadata.java:
##########
@@ -411,6 +411,61 @@ public void testRepeatedCleanActionsWithMetadataTableEnabled(final HoodieTableTy
assertEquals(tableVersion, finalTableVersion.versionCode());
}
+ @ParameterizedTest
+ @CsvSource({"COPY_ON_WRITE,6", "COPY_ON_WRITE,8", "MERGE_ON_READ,6", "MERGE_ON_READ,8"})
+ void testDeletePartitionKeyOnCleanPartition(final HoodieTableType tableType, int tableVersion) throws Exception {
+ initPath();
+ writeConfig = getWriteConfigBuilder(true, true, false)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder()
+ .enable(true)
+ .withMaxNumDeltaCommitsBeforeCompaction(6)
+ .build())
+ .build();
+ writeConfig.setValue(HoodieWriteConfig.WRITE_TABLE_VERSION, String.valueOf(tableVersion));
+ init(tableType, writeConfig);
+ String partition1 = "p1";
+ // Simulate two bulk insert operations adding two data files in partition "p1"
+ String instant1 = metaClient.createNewInstantTime();
+ testTable.doWriteOperation(instant1, BULK_INSERT, emptyList(), Collections.singletonList(partition1), 1);
+ String instant2 = metaClient.createNewInstantTime();
+ testTable.doWriteOperation(instant2, BULK_INSERT, emptyList(), Collections.singletonList(partition1), 1);
+
+ String partition2 = "p2";
+ testTable.doWriteOperation(
+ metaClient.createNewInstantTime(), BULK_INSERT, emptyList(), Collections.singletonList(partition2), 1);
+
+ final HoodieTableMetaClient metadataMetaClient = createMetaClient(metadataTableBasePath);
+ metadataMetaClient.reloadActiveTimeline();
+
+ assertEquals(0, getNumCompactions(metadataMetaClient));
+
+ String cleanInstant = metaClient.createNewInstantTime();
+ HoodieCleanMetadata cleanMetadata = testTable.doCleanBasedOnPartitions(cleanInstant, Collections.singletonList(partition1));
+
+ while (getNumCompactions(metadataMetaClient) == 0) {
+ // Write until the compaction happens in the metadata table
Review Comment:
During compaction, deleted records won’t be written to the compacted file, so there won’t be any entry for that key. Without compaction, the getRecordsByKeyPrefixes method will still read the deleted record from the log files — it does apply pre-combine and marks the record as deleted, but the record will still be present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2918421829
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
}, {
"hash" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671",
"triggerID" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"triggerType" : "PUSH"
}, {
"hash" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5672",
"triggerID" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"triggerType" : "PUSH"
}, {
"hash" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5837",
"triggerID" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b475f230c8d843dabd917f5225440e0f7ee178c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5837)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13346:
URL: https://github.com/apache/hudi/pull/13346#discussion_r2109772357
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedTableMetadata.java:
##########
@@ -411,6 +411,61 @@ public void testRepeatedCleanActionsWithMetadataTableEnabled(final HoodieTableTy
assertEquals(tableVersion, finalTableVersion.versionCode());
}
+ @ParameterizedTest
+ @CsvSource({"COPY_ON_WRITE,6", "COPY_ON_WRITE,8", "MERGE_ON_READ,6", "MERGE_ON_READ,8"})
+ void testDeletePartitionKeyOnCleanPartition(final HoodieTableType tableType, int tableVersion) throws Exception {
Review Comment:
we dont' need to make this run for 4 combinations. lets not try to add more parametrized test if there is no need.
this is just testing regular updates to FILES partition in MDT.
just 1 version would suffice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2919315528
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
}, {
"hash" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671",
"triggerID" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"triggerType" : "PUSH"
}, {
"hash" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5672",
"triggerID" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"triggerType" : "PUSH"
}, {
"hash" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5837",
"triggerID" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"triggerType" : "PUSH"
}, {
"hash" : "23136e2899254b6dca123b1dd93de74a32e24f4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "23136e2899254b6dca123b1dd93de74a32e24f4d",
"triggerType" : "PUSH"
}, {
"hash" : "9ce28229ee1e07a02b63700e778ade51c8bb7778",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5838",
"triggerID" : "9ce28229ee1e07a02b63700e778ade51c8bb7778",
"triggerType" : "PUSH"
}, {
"hash" : "f05a8cd8e1916737a6ae24e215b3c26a4a7a3a80",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f05a8cd8e1916737a6ae24e215b3c26a4a7a3a80",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 23136e2899254b6dca123b1dd93de74a32e24f4d UNKNOWN
* 9ce28229ee1e07a02b63700e778ade51c8bb7778 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5838)
* f05a8cd8e1916737a6ae24e215b3c26a4a7a3a80 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2901943876
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6e7823f4a3535ff065549d1c0a1a089a00fb258 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648)
* 663dabfa3ee641537b70ee1b123a144d5d981735 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2901934862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6e7823f4a3535ff065549d1c0a1a089a00fb258 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648)
* 663dabfa3ee641537b70ee1b123a144d5d981735 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13346:
URL: https://github.com/apache/hudi/pull/13346#discussion_r2103888277
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java:
##########
@@ -1154,11 +1163,11 @@ public HoodieReplaceCommitMetadata doCluster(String commitTime, Map<String, List
return replaceMetadata;
}
- public HoodieCleanMetadata doClean(String commitTime, Map<String, Integer> partitionFileCountsToDelete) throws IOException {
- return doClean(commitTime, partitionFileCountsToDelete, Collections.emptyMap());
+ public HoodieCleanMetadata doClean(String commitTime, Map<String, Integer> partitionFileCountsToDelete, List<String> partitionsToBeDeleted) throws IOException {
+ return doClean(commitTime, partitionFileCountsToDelete, partitionsToBeDeleted, Collections.emptyMap());
Review Comment:
can we keep the method unchanged and add a new one.
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java:
##########
@@ -1154,11 +1163,11 @@ public HoodieReplaceCommitMetadata doCluster(String commitTime, Map<String, List
return replaceMetadata;
}
- public HoodieCleanMetadata doClean(String commitTime, Map<String, Integer> partitionFileCountsToDelete) throws IOException {
- return doClean(commitTime, partitionFileCountsToDelete, Collections.emptyMap());
+ public HoodieCleanMetadata doClean(String commitTime, Map<String, Integer> partitionFileCountsToDelete, List<String> partitionsToBeDeleted) throws IOException {
+ return doClean(commitTime, partitionFileCountsToDelete, partitionsToBeDeleted, Collections.emptyMap());
}
- public HoodieCleanMetadata doClean(String commitTime, Map<String, Integer> partitionFileCountsToDelete, Map<String, String> extraMetadata) throws IOException {
+ public HoodieCleanMetadata doClean(String commitTime, Map<String, Integer> partitionFileCountsToDelete, List<String> partitionsToDelete, Map<String, String> extraMetadata) throws IOException {
Review Comment:
can we keep the method unchanged and add a new one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2903787989
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
}, {
"hash" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671",
"triggerID" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"triggerType" : "PUSH"
}, {
"hash" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5672",
"triggerID" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2b90a5f54fb488a570af340ffd53e0dcfe912dc4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5672)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #13346:
URL: https://github.com/apache/hudi/pull/13346
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2921244781
hey @yihua : I am good w/ the patch. once you are good, feel free to land.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2918734381
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
}, {
"hash" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671",
"triggerID" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"triggerType" : "PUSH"
}, {
"hash" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5672",
"triggerID" : "2b90a5f54fb488a570af340ffd53e0dcfe912dc4",
"triggerType" : "PUSH"
}, {
"hash" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5837",
"triggerID" : "4b475f230c8d843dabd917f5225440e0f7ee178c",
"triggerType" : "PUSH"
}, {
"hash" : "23136e2899254b6dca123b1dd93de74a32e24f4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "23136e2899254b6dca123b1dd93de74a32e24f4d",
"triggerType" : "PUSH"
}, {
"hash" : "9ce28229ee1e07a02b63700e778ade51c8bb7778",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5838",
"triggerID" : "9ce28229ee1e07a02b63700e778ade51c8bb7778",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 23136e2899254b6dca123b1dd93de74a32e24f4d UNKNOWN
* 9ce28229ee1e07a02b63700e778ade51c8bb7778 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5838)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13346:
URL: https://github.com/apache/hudi/pull/13346#discussion_r2103306164
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -277,14 +277,25 @@ public static HoodieRecord<HoodieMetadataPayload> createPartitionListRecord(Coll
public static HoodieRecord<HoodieMetadataPayload> createPartitionFilesRecord(String partition,
Map<String, Long> filesAdded,
List<String> filesDeleted) {
+ return createPartitionFilesRecord(partition, filesAdded, filesDeleted, false);
+ }
+
+ public static HoodieRecord<HoodieMetadataPayload> createPartitionFilesRecord(String partition,
+ Map<String, Long> filesAdded,
+ List<String> filesDeleted,
+ boolean isPartitionDeleted) {
String partitionIdentifier = getPartitionIdentifierForFilesPartition(partition);
+ HoodieKey key = new HoodieKey(partitionIdentifier, MetadataPartitionType.FILES.getPartitionPath());
+ if (isPartitionDeleted) {
+ return new HoodieAvroRecord<>(key, new HoodieMetadataPayload(Option.empty()));
+ }
Review Comment:
If the partition is deleted, we still need to add `filesDeleted` as -1s in the metadata payload so that they are deleted in the metadata table files partition?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2902089032
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 663dabfa3ee641537b70ee1b123a144d5d981735 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2902595958
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 663dabfa3ee641537b70ee1b123a144d5d981735 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649)
* 62e625b73e1bb222c86ef95014df6bada2a72d2c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13346:
URL: https://github.com/apache/hudi/pull/13346#issuecomment-2903363010
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5648",
"triggerID" : "c6e7823f4a3535ff065549d1c0a1a089a00fb258",
"triggerType" : "PUSH"
}, {
"hash" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5649",
"triggerID" : "663dabfa3ee641537b70ee1b123a144d5d981735",
"triggerType" : "PUSH"
}, {
"hash" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5655",
"triggerID" : "62e625b73e1bb222c86ef95014df6bada2a72d2c",
"triggerType" : "PUSH"
}, {
"hash" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668",
"triggerID" : "e94f974d6db090699471c794e1d24b7a52d94eb3",
"triggerType" : "PUSH"
}, {
"hash" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671",
"triggerID" : "8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e94f974d6db090699471c794e1d24b7a52d94eb3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5668)
* 8e665c559958f114f63d7c1d0a0b3eb3c3c2a6ac Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5671)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8449] Fix deletion of record from FILES partition on empty files list [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13346:
URL: https://github.com/apache/hudi/pull/13346#discussion_r2103888852
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java:
##########
@@ -1168,6 +1177,9 @@ public HoodieCleanMetadata doClean(String commitTime, Map<String, Integer> parti
testTableState = testTableState.createTestTableStateForCleaner(commitTime, entry.getKey(), entry.getValue());
deleteFilesInPartition(entry.getKey(), entry.getValue());
}
+
+ testTableState = testTableState.createTestTableStateForCleaner(commitTime, partitionsToDelete);
Review Comment:
why recreate the table state
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org