You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2024/12/08 16:14:44 UTC
[PR] [HUDI-8643] Fix a read bug for SI [hudi]
linliu-code opened a new pull request, #12447:
URL: https://github.com/apache/hudi/pull/12447
### Change Logs
This PR fixes a bug during SI, which causes the lost of SI records.
### Impact
Correct SI behavior.
### Risk level (write none, low medium or high below)
Medium.
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1881200836
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
+ }
+ } else {
+ // When and Only when the latest log record is non-tombstone, logRecordMap contains its recordKey.
+ logRecordsMap.remove(recordKey);
+ deletedRecordsFromLogs.add(recordKey);
+ }
+ });
+
+ // Return non-tombstone records from the log files.
+ logRecordsMap.forEach((key, value) -> recordKeyMap.put(
+ key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey())));
+ // Return non-tombstone records from the base file.
+ if (baseFileRecords != null) {
+ baseFileRecords.forEach((key, value) -> {
+ if (!deletedRecordsFromLogs.contains(key)) {
+ recordKeyMap.put(key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey()));
Review Comment:
I can use the above code; but it is more complex than needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1889114737
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
+ }
+ } else {
+ // When and Only when the latest log record is non-tombstone, logRecordMap contains its recordKey.
+ logRecordsMap.remove(recordKey);
Review Comment:
Right, if we have this, then output logic is much flexible. Otherwise, the output logic will be fuzzy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1881148569
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
+ }
+ } else {
+ // When and Only when the latest log record is non-tombstone, logRecordMap contains its recordKey.
+ logRecordsMap.remove(recordKey);
+ deletedRecordsFromLogs.add(recordKey);
+ }
+ });
+
+ // Return non-tombstone records from the log files.
+ logRecordsMap.forEach((key, value) -> recordKeyMap.put(
+ key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey())));
+ // Return non-tombstone records from the base file.
+ if (baseFileRecords != null) {
+ baseFileRecords.forEach((key, value) -> {
+ if (!deletedRecordsFromLogs.contains(key)) {
+ recordKeyMap.put(key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey()));
Review Comment:
For eg, I went to master and made edits as below. all tests you have added in this patch worked.
```
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> keySet, Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ // Map of recordKey (primaryKey) -> log record that is not deleted for all input recordKeys
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
+ deletedRecordsFromLogs.remove(recordKey); // we can check if its present and then remove if need be
+ }
+ } else {
+ deletedRecordsFromLogs.add(recordKey);
+ logRecordsMap.remove(recordKey);
+ }
+ });
+
+ Map<String, String> recordKeyMap = new HashMap<>();
+ if (baseFileRecords == null || baseFileRecords.isEmpty()) {
+ logRecordsMap.forEach((key1, value1) -> {
+ if (!value1.getData().isDeleted() && !deletedRecordsFromLogs.contains(key1)) {
+ recordKeyMap.put(key1, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value1.getRecordKey()));
+ }
+ });
+ } else {
+ // Iterate over all provided log-records, merging them into existing records
+ logRecordsMap.forEach((key1, value1) -> baseFileRecords.merge(key1, value1, (oldRecord, newRecord) -> {
+ Option<HoodieRecord<HoodieMetadataPayload>> mergedRecord = HoodieMetadataPayload.combineSecondaryIndexRecord(oldRecord, newRecord);
+ return mergedRecord.orElse(null);
+ }));
+ baseFileRecords.forEach((key, value) -> {
+ if (!deletedRecordsFromLogs.contains(key)) {
+ recordKeyMap.put(key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey()));
+ }
+ });
+ }
return recordKeyMap;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1881183568
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
Review Comment:
No, we cannot. The single usage of `deletedRecordsFromLogs` is to filter records from base file, .e.g., we have `br`, `lr1`(deleted), `lr2`(non-deleted). When we handle `lr2`, we cannot remove its `rk`, since `rk` is used to filter `br`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2557703417
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
}, {
"hash" : "179649bf511615beca40b21c5064e7cc2f868e20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335",
"triggerID" : "179649bf511615beca40b21c5064e7cc2f868e20",
"triggerType" : "PUSH"
}, {
"hash" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2466",
"triggerID" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"triggerType" : "PUSH"
}, {
"hash" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2468",
"triggerID" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"triggerType" : "PUSH"
}, {
"hash" : "f34959ac0a8ffe85115e01b940671567c5868b4e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f34959ac0a8ffe85115e01b940671567c5868b4e",
"triggerType" : "PUSH"
}, {
"hash" : "75c009240312e77dba5dac7c02ecbea3815363f8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2546",
"triggerID" : "75c009240312e77dba5dac7c02ecbea3815363f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
* 8f39643f964d9c03a08fa7c903b0031dc4a28897 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2468)
* f34959ac0a8ffe85115e01b940671567c5868b4e UNKNOWN
* 75c009240312e77dba5dac7c02ecbea3815363f8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2546)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1881147804
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
+ }
+ } else {
+ // When and Only when the latest log record is non-tombstone, logRecordMap contains its recordKey.
+ logRecordsMap.remove(recordKey);
+ deletedRecordsFromLogs.add(recordKey);
+ }
+ });
+
+ // Return non-tombstone records from the log files.
+ logRecordsMap.forEach((key, value) -> recordKeyMap.put(
+ key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey())));
+ // Return non-tombstone records from the base file.
+ if (baseFileRecords != null) {
+ baseFileRecords.forEach((key, value) -> {
+ if (!deletedRecordsFromLogs.contains(key)) {
+ recordKeyMap.put(key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey()));
Review Comment:
why we are not merging the records from base to log records? I understand in case of secondary index, a record is either created or deleted and there is no real merge, but lets keep the original code as is and just add fixes on top of that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1889133507
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
+ }
+ } else {
+ // When and Only when the latest log record is non-tombstone, logRecordMap contains its recordKey.
+ logRecordsMap.remove(recordKey);
+ deletedRecordsFromLogs.add(recordKey);
+ }
+ });
+
+ // Return non-tombstone records from the log files.
+ logRecordsMap.forEach((key, value) -> recordKeyMap.put(
+ key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey())));
+ // Return non-tombstone records from the base file.
+ if (baseFileRecords != null) {
+ baseFileRecords.forEach((key, value) -> {
+ if (!deletedRecordsFromLogs.contains(key)) {
+ recordKeyMap.put(key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey()));
Review Comment:
Yeah, I will adopt Siva's approach to avoid back-and-forth arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2549786090
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
}, {
"hash" : "179649bf511615beca40b21c5064e7cc2f868e20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335",
"triggerID" : "179649bf511615beca40b21c5064e7cc2f868e20",
"triggerType" : "PUSH"
}, {
"hash" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2466",
"triggerID" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"triggerType" : "PUSH"
}, {
"hash" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2468",
"triggerID" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
* 8f39643f964d9c03a08fa7c903b0031dc4a28897 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2468)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1881200836
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
+ }
+ } else {
+ // When and Only when the latest log record is non-tombstone, logRecordMap contains its recordKey.
+ logRecordsMap.remove(recordKey);
+ deletedRecordsFromLogs.add(recordKey);
+ }
+ });
+
+ // Return non-tombstone records from the log files.
+ logRecordsMap.forEach((key, value) -> recordKeyMap.put(
+ key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey())));
+ // Return non-tombstone records from the base file.
+ if (baseFileRecords != null) {
+ baseFileRecords.forEach((key, value) -> {
+ if (!deletedRecordsFromLogs.contains(key)) {
+ recordKeyMap.put(key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey()));
Review Comment:
Ok, will adopt the above code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1881187158
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
+ }
+ } else {
+ // When and Only when the latest log record is non-tombstone, logRecordMap contains its recordKey.
+ logRecordsMap.remove(recordKey);
+ deletedRecordsFromLogs.add(recordKey);
+ }
+ });
+
+ // Return non-tombstone records from the log files.
+ logRecordsMap.forEach((key, value) -> recordKeyMap.put(
+ key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey())));
+ // Return non-tombstone records from the base file.
+ if (baseFileRecords != null) {
+ baseFileRecords.forEach((key, value) -> {
+ if (!deletedRecordsFromLogs.contains(key)) {
+ recordKeyMap.put(key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey()));
Review Comment:
@nsivabalan , the it is the merge that cause the bug. Suppose we have `br` and a non-delete `lr`. In this case, after the merge, `lr` is returned. Now it will go through the ` if (!deletedRecordsFromLogs.contains(key))`, since `deletedRecordsFromLogs` contains its rk, `lr` is gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2557764756
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
}, {
"hash" : "179649bf511615beca40b21c5064e7cc2f868e20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335",
"triggerID" : "179649bf511615beca40b21c5064e7cc2f868e20",
"triggerType" : "PUSH"
}, {
"hash" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2466",
"triggerID" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"triggerType" : "PUSH"
}, {
"hash" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2468",
"triggerID" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"triggerType" : "PUSH"
}, {
"hash" : "f34959ac0a8ffe85115e01b940671567c5868b4e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f34959ac0a8ffe85115e01b940671567c5868b4e",
"triggerType" : "PUSH"
}, {
"hash" : "75c009240312e77dba5dac7c02ecbea3815363f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2546",
"triggerID" : "75c009240312e77dba5dac7c02ecbea3815363f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
* f34959ac0a8ffe85115e01b940671567c5868b4e UNKNOWN
* 75c009240312e77dba5dac7c02ecbea3815363f8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2546)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1888498250
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
Review Comment:
Do we need to create a TreeSet of recordKeys? I don't think order of record keys matters here.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
+ }
+ } else {
+ // When and Only when the latest log record is non-tombstone, logRecordMap contains its recordKey.
+ logRecordsMap.remove(recordKey);
Review Comment:
So this is the main fix. If i understand correctly, after this step the logRecordsMap will not contain any tomstone record, therefore we do not need to do the check in `HoodieMetadataPayload.combineSecondaryIndexRecord` later right?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
+ }
+ } else {
+ // When and Only when the latest log record is non-tombstone, logRecordMap contains its recordKey.
+ logRecordsMap.remove(recordKey);
+ deletedRecordsFromLogs.add(recordKey);
+ }
+ });
+
+ // Return non-tombstone records from the log files.
+ logRecordsMap.forEach((key, value) -> recordKeyMap.put(
+ key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey())));
+ // Return non-tombstone records from the base file.
+ if (baseFileRecords != null) {
+ baseFileRecords.forEach((key, value) -> {
+ if (!deletedRecordsFromLogs.contains(key)) {
+ recordKeyMap.put(key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey()));
Review Comment:
Both are fine IMO, but maybe I am biased due to prior work. Ideally, I would like to retain the semantics of `combineSecondaryIndexRecord` but the current patch is also simple to understand. Plus, if we filter out tombstone records before calling `combineSecondaryIndexRecord` then there is no point in keeping it. Please remove this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1889108682
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
Review Comment:
Yeah, the order of the record key here does not matter. I can check which one is more performant regarding the search.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2549491241
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
}, {
"hash" : "179649bf511615beca40b21c5064e7cc2f868e20",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335",
"triggerID" : "179649bf511615beca40b21c5064e7cc2f868e20",
"triggerType" : "PUSH"
}, {
"hash" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2466",
"triggerID" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
* 179649bf511615beca40b21c5064e7cc2f868e20 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335)
* f2fb700c13719e3eb5ea09b45dc44d5dae402d68 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2466)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1889493647
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
Review Comment:
do we have test for this scenario ?
i.e.
add a record to logfile1, delete in logfile2 and then add it back in logfile3.
if the test works, I am good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8643] Fix a read bug for SI [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2527273668
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44118113b05a9543a8f48e4abfd1c47773531ea2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326)
* 8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1881210987
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
+ }
+ } else {
+ // When and Only when the latest log record is non-tombstone, logRecordMap contains its recordKey.
+ logRecordsMap.remove(recordKey);
+ deletedRecordsFromLogs.add(recordKey);
+ }
+ });
+
+ // Return non-tombstone records from the log files.
+ logRecordsMap.forEach((key, value) -> recordKeyMap.put(
+ key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey())));
+ // Return non-tombstone records from the base file.
+ if (baseFileRecords != null) {
+ baseFileRecords.forEach((key, value) -> {
+ if (!deletedRecordsFromLogs.contains(key)) {
+ recordKeyMap.put(key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey()));
Review Comment:
@nsivabalan , the reason is that we discuss here is that the logic of this function is too complex to understand. so we should try to make it as simple as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8643] Fix a read bug for SI [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2527276838
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44118113b05a9543a8f48e4abfd1c47773531ea2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326)
* 8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331)
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2550308671
Yeah, it may trigger the same bug before or a different bug. Will check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1889111198
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
Review Comment:
Generally HashSet is faster than TreeSet; will change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2549620554
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
}, {
"hash" : "179649bf511615beca40b21c5064e7cc2f868e20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335",
"triggerID" : "179649bf511615beca40b21c5064e7cc2f868e20",
"triggerType" : "PUSH"
}, {
"hash" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2466",
"triggerID" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"triggerType" : "PUSH"
}, {
"hash" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2468",
"triggerID" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
* f2fb700c13719e3eb5ea09b45dc44d5dae402d68 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2466)
* 8f39643f964d9c03a08fa7c903b0031dc4a28897 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2468)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2563938828
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
}, {
"hash" : "179649bf511615beca40b21c5064e7cc2f868e20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335",
"triggerID" : "179649bf511615beca40b21c5064e7cc2f868e20",
"triggerType" : "PUSH"
}, {
"hash" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2466",
"triggerID" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"triggerType" : "PUSH"
}, {
"hash" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2468",
"triggerID" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"triggerType" : "PUSH"
}, {
"hash" : "f34959ac0a8ffe85115e01b940671567c5868b4e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f34959ac0a8ffe85115e01b940671567c5868b4e",
"triggerType" : "PUSH"
}, {
"hash" : "75c009240312e77dba5dac7c02ecbea3815363f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2546",
"triggerID" : "75c009240312e77dba5dac7c02ecbea3815363f8",
"triggerType" : "PUSH"
}, {
"hash" : "3ccf599b0a231a028281cfa8db20b9ed028899da",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3ccf599b0a231a028281cfa8db20b9ed028899da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
* f34959ac0a8ffe85115e01b940671567c5868b4e UNKNOWN
* 75c009240312e77dba5dac7c02ecbea3815363f8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2546)
* 3ccf599b0a231a028281cfa8db20b9ed028899da UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8643] Fix a read bug for SI [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2527280019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331)
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2527643576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
}, {
"hash" : "179649bf511615beca40b21c5064e7cc2f868e20",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335",
"triggerID" : "179649bf511615beca40b21c5064e7cc2f868e20",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
* 179649bf511615beca40b21c5064e7cc2f868e20 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1881145721
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
Review Comment:
should we also do
```
deletedRecordsFromLogs.remove(recordKey);
```
within the if block.
just incase a record is deleted first and added back later. we should not treat it as deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8643] Fix a read bug for SI [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2527298747
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
}, {
"hash" : "179649bf511615beca40b21c5064e7cc2f868e20",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "179649bf511615beca40b21c5064e7cc2f868e20",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331)
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
* 179649bf511615beca40b21c5064e7cc2f868e20 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2527416074
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
}, {
"hash" : "179649bf511615beca40b21c5064e7cc2f868e20",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335",
"triggerID" : "179649bf511615beca40b21c5064e7cc2f868e20",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331)
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
* 179649bf511615beca40b21c5064e7cc2f868e20 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1888498250
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
Review Comment:
Do we need to create a TreeSet of recordKeys? I don't think order of record keys matters here. I believe it was there already, probably to optimize `contains` check. But, even a simple HashSet would do right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1889133507
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
+ }
+ } else {
+ // When and Only when the latest log record is non-tombstone, logRecordMap contains its recordKey.
+ logRecordsMap.remove(recordKey);
+ deletedRecordsFromLogs.add(recordKey);
+ }
+ });
+
+ // Return non-tombstone records from the log files.
+ logRecordsMap.forEach((key, value) -> recordKeyMap.put(
+ key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey())));
+ // Return non-tombstone records from the base file.
+ if (baseFileRecords != null) {
+ baseFileRecords.forEach((key, value) -> {
+ if (!deletedRecordsFromLogs.contains(key)) {
+ recordKeyMap.put(key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey()));
Review Comment:
Yeah, I will Siva's approach to avoid back-and-forth arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2549500208
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
}, {
"hash" : "179649bf511615beca40b21c5064e7cc2f868e20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335",
"triggerID" : "179649bf511615beca40b21c5064e7cc2f868e20",
"triggerType" : "PUSH"
}, {
"hash" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2466",
"triggerID" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"triggerType" : "PUSH"
}, {
"hash" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
* f2fb700c13719e3eb5ea09b45dc44d5dae402d68 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2466)
* 8f39643f964d9c03a08fa7c903b0031dc4a28897 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1881187158
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
+ }
+ } else {
+ // When and Only when the latest log record is non-tombstone, logRecordMap contains its recordKey.
+ logRecordsMap.remove(recordKey);
+ deletedRecordsFromLogs.add(recordKey);
+ }
+ });
+
+ // Return non-tombstone records from the log files.
+ logRecordsMap.forEach((key, value) -> recordKeyMap.put(
+ key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey())));
+ // Return non-tombstone records from the base file.
+ if (baseFileRecords != null) {
+ baseFileRecords.forEach((key, value) -> {
+ if (!deletedRecordsFromLogs.contains(key)) {
+ recordKeyMap.put(key, SecondaryIndexKeyUtils.getSecondaryKeyFromSecondaryIndexKey(value.getRecordKey()));
Review Comment:
@nsivabalan , the it is the merge that cause the bug. Suppose we have `br` and a non-delete `lr`. In this case, after the merge, `lr` is returned. Now it will go through the ` if (!deletedRecordsFromLogs.contains(key))`, since `deletedRecordsFromLogs` contains its rk, `lr` is gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8643] Fix a read bug for SI [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2526203704
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44118113b05a9543a8f48e4abfd1c47773531ea2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8643] Fix a read bug for SI [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2526203222
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44118113b05a9543a8f48e4abfd1c47773531ea2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1888492001
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>> fetchBaseFileAllRecords
return SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String> reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+ Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+ HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey = SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
Review Comment:
I think it's ok because the log records are handled in temporal order so it won't be treated as deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2549475050
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
}, {
"hash" : "179649bf511615beca40b21c5064e7cc2f868e20",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335",
"triggerID" : "179649bf511615beca40b21c5064e7cc2f868e20",
"triggerType" : "PUSH"
}, {
"hash" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
* 179649bf511615beca40b21c5064e7cc2f868e20 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335)
* f2fb700c13719e3eb5ea09b45dc44d5dae402d68 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2563950318
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
}, {
"hash" : "179649bf511615beca40b21c5064e7cc2f868e20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335",
"triggerID" : "179649bf511615beca40b21c5064e7cc2f868e20",
"triggerType" : "PUSH"
}, {
"hash" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2466",
"triggerID" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"triggerType" : "PUSH"
}, {
"hash" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2468",
"triggerID" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"triggerType" : "PUSH"
}, {
"hash" : "f34959ac0a8ffe85115e01b940671567c5868b4e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f34959ac0a8ffe85115e01b940671567c5868b4e",
"triggerType" : "PUSH"
}, {
"hash" : "75c009240312e77dba5dac7c02ecbea3815363f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2546",
"triggerID" : "75c009240312e77dba5dac7c02ecbea3815363f8",
"triggerType" : "PUSH"
}, {
"hash" : "3ccf599b0a231a028281cfa8db20b9ed028899da",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2583",
"triggerID" : "3ccf599b0a231a028281cfa8db20b9ed028899da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
* f34959ac0a8ffe85115e01b940671567c5868b4e UNKNOWN
* 75c009240312e77dba5dac7c02ecbea3815363f8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2546)
* 3ccf599b0a231a028281cfa8db20b9ed028899da Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2583)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2563993405
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
}, {
"hash" : "179649bf511615beca40b21c5064e7cc2f868e20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335",
"triggerID" : "179649bf511615beca40b21c5064e7cc2f868e20",
"triggerType" : "PUSH"
}, {
"hash" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2466",
"triggerID" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"triggerType" : "PUSH"
}, {
"hash" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2468",
"triggerID" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"triggerType" : "PUSH"
}, {
"hash" : "f34959ac0a8ffe85115e01b940671567c5868b4e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f34959ac0a8ffe85115e01b940671567c5868b4e",
"triggerType" : "PUSH"
}, {
"hash" : "75c009240312e77dba5dac7c02ecbea3815363f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2546",
"triggerID" : "75c009240312e77dba5dac7c02ecbea3815363f8",
"triggerType" : "PUSH"
}, {
"hash" : "3ccf599b0a231a028281cfa8db20b9ed028899da",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2583",
"triggerID" : "3ccf599b0a231a028281cfa8db20b9ed028899da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
* f34959ac0a8ffe85115e01b940671567c5868b4e UNKNOWN
* 3ccf599b0a231a028281cfa8db20b9ed028899da Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2583)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8643] Fix a read bug for SI [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2526277820
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44118113b05a9543a8f48e4abfd1c47773531ea2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2557635987
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
}, {
"hash" : "179649bf511615beca40b21c5064e7cc2f868e20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335",
"triggerID" : "179649bf511615beca40b21c5064e7cc2f868e20",
"triggerType" : "PUSH"
}, {
"hash" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2466",
"triggerID" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"triggerType" : "PUSH"
}, {
"hash" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2468",
"triggerID" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"triggerType" : "PUSH"
}, {
"hash" : "f34959ac0a8ffe85115e01b940671567c5868b4e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f34959ac0a8ffe85115e01b940671567c5868b4e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
* 8f39643f964d9c03a08fa7c903b0031dc4a28897 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2468)
* f34959ac0a8ffe85115e01b940671567c5868b4e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2550334266
@codope After the refactoring, the test failed quickly. Should be a regression. Let me change back to my previous version to confirm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2549497558
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
}, {
"hash" : "179649bf511615beca40b21c5064e7cc2f868e20",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335",
"triggerID" : "179649bf511615beca40b21c5064e7cc2f868e20",
"triggerType" : "PUSH"
}, {
"hash" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2466",
"triggerID" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"triggerType" : "PUSH"
}, {
"hash" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
* 179649bf511615beca40b21c5064e7cc2f868e20 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335)
* f2fb700c13719e3eb5ea09b45dc44d5dae402d68 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2466)
* 8f39643f964d9c03a08fa7c903b0031dc4a28897 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #12447:
URL: https://github.com/apache/hudi/pull/12447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8648] Fix a bug for secondary index deletion [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12447:
URL: https://github.com/apache/hudi/pull/12447#issuecomment-2557671044
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2326",
"triggerID" : "44118113b05a9543a8f48e4abfd1c47773531ea2",
"triggerType" : "PUSH"
}, {
"hash" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2331",
"triggerID" : "8200082ffb2bee53ae6dc04fc5d42fe63a07e0e4",
"triggerType" : "PUSH"
}, {
"hash" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71c47fb3e3b347f106ea2589522eb57b178aa39",
"triggerType" : "PUSH"
}, {
"hash" : "179649bf511615beca40b21c5064e7cc2f868e20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2335",
"triggerID" : "179649bf511615beca40b21c5064e7cc2f868e20",
"triggerType" : "PUSH"
}, {
"hash" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2466",
"triggerID" : "f2fb700c13719e3eb5ea09b45dc44d5dae402d68",
"triggerType" : "PUSH"
}, {
"hash" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2468",
"triggerID" : "8f39643f964d9c03a08fa7c903b0031dc4a28897",
"triggerType" : "PUSH"
}, {
"hash" : "f34959ac0a8ffe85115e01b940671567c5868b4e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f34959ac0a8ffe85115e01b940671567c5868b4e",
"triggerType" : "PUSH"
}, {
"hash" : "75c009240312e77dba5dac7c02ecbea3815363f8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "75c009240312e77dba5dac7c02ecbea3815363f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b71c47fb3e3b347f106ea2589522eb57b178aa39 UNKNOWN
* 8f39643f964d9c03a08fa7c903b0031dc4a28897 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2468)
* f34959ac0a8ffe85115e01b940671567c5868b4e UNKNOWN
* 75c009240312e77dba5dac7c02ecbea3815363f8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org