You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "suryaprasanna (via GitHub)" <gi...@apache.org> on 2025/12/05 06:22:07 UTC
[PR] feat(storage): add config to block duplicates while writing to HFiles [hudi]
suryaprasanna opened a new pull request, #17495:
URL: https://github.com/apache/hudi/pull/17495
### Describe the issue this Pull Request addresses
During record_index bootstrapping, duplicate records can be written to HFiles, which causes the validation check during re-bootstrapping to fail. This PR introduces a configuration option to control whether duplicate keys should be allowed when writing to HFiles.
### Summary and Changelog
Added a new configuration property `hoodie.hfile.writer.allow.duplicates` to control duplicate key handling in HFile writes. When set to `false` (default), the writer will throw an exception if duplicate keys are encountered, preventing bootstrap failures during record index re-bootstrapping.
**Changes:**
- Added `HFILE_WRITER_TO_ALLOW_DUPLICATES` config property in `HoodieStorageConfig` (default: `false`)
- Added `hfileWriterToAllowDuplicates()` method in `HoodieWriteConfig` to expose the config
- Modified `HoodieHFileConfig` to accept and store the `allowDuplicatesToBeInserted` parameter
- Updated `HoodieAvroHFileWriter` to conditionally check for duplicates based on config
- Changed duplicate key detection from always throwing exception to config-controlled behavior
- Updated `HoodieMetadataWriteUtils` to pass through the config to metadata table writes
- Enhanced `testDuplicatesDuringRecordIndexBootstrap` test to validate both config values
### Impact
**Configuration Change:**
- New config property: `hoodie.hfile.writer.allow.duplicates` (default: `false`)
- When `false`: Maintains existing behavior - throws `HoodieDuplicateKeyException` on duplicate keys
- When `true`: Allows duplicate keys to be written to HFiles (with logging)
**Performance Impact:** Minimal - duplicate check only runs when config is disabled (default behavior)
### Risk Level
**low** - This is a config-gated change with safe defaults. The default behavior (`false`) maintains existing duplicate prevention logic. Users experiencing bootstrap failures due to duplicates can opt-in to allowing duplicates by setting the config to `true`.
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk.
If medium or high, explain what verification was done to mitigate the risks. -->
### Documentation Update
Config documentation:
- Added `hoodie.hfile.writer.allow.duplicates` config property with description: "Allows duplicates to be written into HFile." (default: `false`)
- The config description must be updated if new configs are added or the default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website. Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3629468493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941",
"triggerID" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"triggerType" : "PUSH"
}, {
"hash" : "710266f5da56a496efba11206c1b012a28c05cd0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10033",
"triggerID" : "710266f5da56a496efba11206c1b012a28c05cd0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 710266f5da56a496efba11206c1b012a28c05cd0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10033)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #17495:
URL: https://github.com/apache/hudi/pull/17495#discussion_r2733942636
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -2350,6 +2350,10 @@ public long getHFileMaxFileSize() {
return getLong(HoodieStorageConfig.HFILE_MAX_FILE_SIZE);
}
+ public boolean isAllowDuplicatesWithHfileWritesEnabled() {
Review Comment:
minor.
we can just name the method `allowDuplicatesWithHfileWrites`
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/storage/hadoop/HoodieAvroHFileWriter.java:
##########
@@ -127,9 +130,12 @@ public boolean canWrite() {
@Override
public void writeAvro(String recordKey, IndexedRecord record) throws IOException {
- if (prevRecordKey.equals(recordKey)) {
- throw new HoodieDuplicateKeyException("Duplicate recordKey " + recordKey + " found while writing to HFile."
- + "Record payload: " + record);
+ if (!this.hfileConfig.isAllowDuplicatesOnHfileWrites()) {
+ // When this config is enabled, do not allow duplicates to be written to hFile.
Review Comment:
minor. the comment needs to be fixed.
`When this config is not enabled, do not allow duplicates to be written to hFile.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #17495:
URL: https://github.com/apache/hudi/pull/17495#discussion_r2734367435
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieStorageConfig.java:
##########
@@ -80,6 +80,14 @@ public class HoodieStorageConfig extends HoodieConfig {
.markAdvanced()
.withDocumentation("Target file size in bytes for HFile base files.");
+ public static final ConfigProperty<Boolean> HFILE_WRITER_TO_ALLOW_DUPLICATES = ConfigProperty
+ .key("hoodie.hfile.writes.allow.duplicates")
+ .defaultValue(false)
+ .withDocumentation("When bootstrapping RI, if the main dataset contains duplicates then "
+ + "it will fail the bootstrap job. TO avoid the failure and bootstrap the RI with dups"
Review Comment:
typically in OSS, we use "RLI" as abbreviation for RecordLevelIndex
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3747121357
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941",
"triggerID" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"triggerType" : "PUSH"
}, {
"hash" : "710266f5da56a496efba11206c1b012a28c05cd0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10033",
"triggerID" : "710266f5da56a496efba11206c1b012a28c05cd0",
"triggerType" : "PUSH"
}, {
"hash" : "d621cb87f18d0ab50046040a308898bccb9f1e71",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11004",
"triggerID" : "d621cb87f18d0ab50046040a308898bccb9f1e71",
"triggerType" : "PUSH"
}, {
"hash" : "b7dc5b3a4f2537df29866378b6f8410a466ea5ed",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b7dc5b3a4f2537df29866378b6f8410a466ea5ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 710266f5da56a496efba11206c1b012a28c05cd0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10033)
* d621cb87f18d0ab50046040a308898bccb9f1e71 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11004)
* b7dc5b3a4f2537df29866378b6f8410a466ea5ed UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3807807734
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941",
"triggerID" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"triggerType" : "PUSH"
}, {
"hash" : "710266f5da56a496efba11206c1b012a28c05cd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10033",
"triggerID" : "710266f5da56a496efba11206c1b012a28c05cd0",
"triggerType" : "PUSH"
}, {
"hash" : "d621cb87f18d0ab50046040a308898bccb9f1e71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11004",
"triggerID" : "d621cb87f18d0ab50046040a308898bccb9f1e71",
"triggerType" : "PUSH"
}, {
"hash" : "b7dc5b3a4f2537df29866378b6f8410a466ea5ed",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11008",
"triggerID" : "b7dc5b3a4f2537df29866378b6f8410a466ea5ed",
"triggerType" : "PUSH"
}, {
"hash" : "3f9f642cbceb52938caea3933e477d51ca60a609",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11416",
"triggerID" : "3f9f642cbceb52938caea3933e477d51ca60a609",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7dc5b3a4f2537df29866378b6f8410a466ea5ed Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11008)
* 3f9f642cbceb52938caea3933e477d51ca60a609 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11416)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3802119129
> hey @suryaprasanna : again, trying to gauge the necessity of this feature. we can't have duplicates within RLI partition in MDT right. So, how would this be helpful. would you mind clarifying the need please.
Consider a scenario where the dataset already contain duplicates. Now, when we do record_index bootstrap the job will fail and thereby stopping the ingestion as well, as ingestion would require record_index to be created. Apart from Record index there are also other indexes that are supported by Hudi but they are not performant enough. So, idea here is to resume ingestion by rebootstrapping the record_index with duplicates, and in the offline run duplicates removal tool. Since, the feature is under a config the current production still works. Let me know, let me know if we need to detailed discussion on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #17495:
URL: https://github.com/apache/hudi/pull/17495
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3747543692
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941",
"triggerID" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"triggerType" : "PUSH"
}, {
"hash" : "710266f5da56a496efba11206c1b012a28c05cd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10033",
"triggerID" : "710266f5da56a496efba11206c1b012a28c05cd0",
"triggerType" : "PUSH"
}, {
"hash" : "d621cb87f18d0ab50046040a308898bccb9f1e71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11004",
"triggerID" : "d621cb87f18d0ab50046040a308898bccb9f1e71",
"triggerType" : "PUSH"
}, {
"hash" : "b7dc5b3a4f2537df29866378b6f8410a466ea5ed",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11008",
"triggerID" : "b7dc5b3a4f2537df29866378b6f8410a466ea5ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7dc5b3a4f2537df29866378b6f8410a466ea5ed Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11008)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3808360917
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941",
"triggerID" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"triggerType" : "PUSH"
}, {
"hash" : "710266f5da56a496efba11206c1b012a28c05cd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10033",
"triggerID" : "710266f5da56a496efba11206c1b012a28c05cd0",
"triggerType" : "PUSH"
}, {
"hash" : "d621cb87f18d0ab50046040a308898bccb9f1e71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11004",
"triggerID" : "d621cb87f18d0ab50046040a308898bccb9f1e71",
"triggerType" : "PUSH"
}, {
"hash" : "b7dc5b3a4f2537df29866378b6f8410a466ea5ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11008",
"triggerID" : "b7dc5b3a4f2537df29866378b6f8410a466ea5ed",
"triggerType" : "PUSH"
}, {
"hash" : "3f9f642cbceb52938caea3933e477d51ca60a609",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11416",
"triggerID" : "3f9f642cbceb52938caea3933e477d51ca60a609",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3f9f642cbceb52938caea3933e477d51ca60a609 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11416)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3775135289
hey @suryaprasanna : again, trying to gauge the necessity of this feature.
we can't have duplicates within RLI partition in MDT right.
So, how would this be helpful. would you mind clarifying the need please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to block duplicates while writing to HFiles [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3615487085
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941",
"triggerID" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd5a4f78082ce982f0f86db8f6fa3675e044295a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to block duplicates while writing to HFiles [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3629462363
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941",
"triggerID" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"triggerType" : "PUSH"
}, {
"hash" : "710266f5da56a496efba11206c1b012a28c05cd0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10033",
"triggerID" : "710266f5da56a496efba11206c1b012a28c05cd0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd5a4f78082ce982f0f86db8f6fa3675e044295a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941)
* 710266f5da56a496efba11206c1b012a28c05cd0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10033)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #17495:
URL: https://github.com/apache/hudi/pull/17495#discussion_r2651957231
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -2347,6 +2347,10 @@ public long getHFileMaxFileSize() {
return getLong(HoodieStorageConfig.HFILE_MAX_FILE_SIZE);
}
+ public boolean hfileWriterToAllowDuplicates() {
Review Comment:
`allowDuplicatesWithHfileWrites()`
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieStorageConfig.java:
##########
@@ -80,6 +80,11 @@ public class HoodieStorageConfig extends HoodieConfig {
.markAdvanced()
.withDocumentation("Target file size in bytes for HFile base files.");
+ public static final ConfigProperty<Boolean> HFILE_WRITER_TO_ALLOW_DUPLICATES = ConfigProperty
Review Comment:
`hoodie.hfile.writes.allow.duplicates`
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieHFileConfig.java:
##########
@@ -72,4 +75,8 @@ public BloomFilter getBloomFilter() {
public String getKeyFieldName() {
return keyFieldName;
}
+
+ public boolean allowDuplicatesToBeInserted() {
Review Comment:
same here
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieHFileConfig.java:
##########
@@ -30,19 +30,22 @@ public class HoodieHFileConfig {
private final StorageConfiguration storageConf;
private final BloomFilter bloomFilter;
private final String keyFieldName;
+ private final boolean allowDuplicatesToBeInserted;
Review Comment:
lets try to align the variable from its root. so that we may not get confused if this is a new one.
for eg,
lets stick to
`allowDuplicatesOnHfileWrites` consistently across all layers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to block duplicates while writing to HFiles [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3615498147
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941",
"triggerID" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd5a4f78082ce982f0f86db8f6fa3675e044295a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #17495:
URL: https://github.com/apache/hudi/pull/17495#discussion_r2734038772
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/storage/hadoop/HoodieAvroHFileWriter.java:
##########
@@ -127,9 +130,12 @@ public boolean canWrite() {
@Override
public void writeAvro(String recordKey, IndexedRecord record) throws IOException {
- if (prevRecordKey.equals(recordKey)) {
- throw new HoodieDuplicateKeyException("Duplicate recordKey " + recordKey + " found while writing to HFile."
- + "Record payload: " + record);
+ if (!this.hfileConfig.isAllowDuplicatesOnHfileWrites()) {
+ // When this config is enabled, do not allow duplicates to be written to hFile.
Review Comment:
Fixed the comments and also expanded the description for hoodie.hfile.writer.allow.duplicates config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3747269276
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941",
"triggerID" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"triggerType" : "PUSH"
}, {
"hash" : "710266f5da56a496efba11206c1b012a28c05cd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10033",
"triggerID" : "710266f5da56a496efba11206c1b012a28c05cd0",
"triggerType" : "PUSH"
}, {
"hash" : "d621cb87f18d0ab50046040a308898bccb9f1e71",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11004",
"triggerID" : "d621cb87f18d0ab50046040a308898bccb9f1e71",
"triggerType" : "PUSH"
}, {
"hash" : "b7dc5b3a4f2537df29866378b6f8410a466ea5ed",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11008",
"triggerID" : "b7dc5b3a4f2537df29866378b6f8410a466ea5ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d621cb87f18d0ab50046040a308898bccb9f1e71 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11004)
* b7dc5b3a4f2537df29866378b6f8410a466ea5ed Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11008)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3747196056
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941",
"triggerID" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"triggerType" : "PUSH"
}, {
"hash" : "710266f5da56a496efba11206c1b012a28c05cd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10033",
"triggerID" : "710266f5da56a496efba11206c1b012a28c05cd0",
"triggerType" : "PUSH"
}, {
"hash" : "d621cb87f18d0ab50046040a308898bccb9f1e71",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11004",
"triggerID" : "d621cb87f18d0ab50046040a308898bccb9f1e71",
"triggerType" : "PUSH"
}, {
"hash" : "b7dc5b3a4f2537df29866378b6f8410a466ea5ed",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b7dc5b3a4f2537df29866378b6f8410a466ea5ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d621cb87f18d0ab50046040a308898bccb9f1e71 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11004)
* b7dc5b3a4f2537df29866378b6f8410a466ea5ed UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3747115691
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941",
"triggerID" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"triggerType" : "PUSH"
}, {
"hash" : "710266f5da56a496efba11206c1b012a28c05cd0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10033",
"triggerID" : "710266f5da56a496efba11206c1b012a28c05cd0",
"triggerType" : "PUSH"
}, {
"hash" : "d621cb87f18d0ab50046040a308898bccb9f1e71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d621cb87f18d0ab50046040a308898bccb9f1e71",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 710266f5da56a496efba11206c1b012a28c05cd0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10033)
* d621cb87f18d0ab50046040a308898bccb9f1e71 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #17495:
URL: https://github.com/apache/hudi/pull/17495#discussion_r2701416233
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -2347,6 +2347,10 @@ public long getHFileMaxFileSize() {
return getLong(HoodieStorageConfig.HFILE_MAX_FILE_SIZE);
}
+ public boolean hfileWriterToAllowDuplicates() {
Review Comment:
Made the necessary changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #17495:
URL: https://github.com/apache/hudi/pull/17495#discussion_r2734037302
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -2350,6 +2350,10 @@ public long getHFileMaxFileSize() {
return getLong(HoodieStorageConfig.HFILE_MAX_FILE_SIZE);
}
+ public boolean isAllowDuplicatesWithHfileWritesEnabled() {
Review Comment:
Rename the method as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3807795842
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941",
"triggerID" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"triggerType" : "PUSH"
}, {
"hash" : "710266f5da56a496efba11206c1b012a28c05cd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10033",
"triggerID" : "710266f5da56a496efba11206c1b012a28c05cd0",
"triggerType" : "PUSH"
}, {
"hash" : "d621cb87f18d0ab50046040a308898bccb9f1e71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11004",
"triggerID" : "d621cb87f18d0ab50046040a308898bccb9f1e71",
"triggerType" : "PUSH"
}, {
"hash" : "b7dc5b3a4f2537df29866378b6f8410a466ea5ed",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11008",
"triggerID" : "b7dc5b3a4f2537df29866378b6f8410a466ea5ed",
"triggerType" : "PUSH"
}, {
"hash" : "3f9f642cbceb52938caea3933e477d51ca60a609",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3f9f642cbceb52938caea3933e477d51ca60a609",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7dc5b3a4f2537df29866378b6f8410a466ea5ed Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11008)
* 3f9f642cbceb52938caea3933e477d51ca60a609 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to block duplicates while writing to HFiles [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3629455580
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941",
"triggerID" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"triggerType" : "PUSH"
}, {
"hash" : "710266f5da56a496efba11206c1b012a28c05cd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "710266f5da56a496efba11206c1b012a28c05cd0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd5a4f78082ce982f0f86db8f6fa3675e044295a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9941)
* 710266f5da56a496efba11206c1b012a28c05cd0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to block duplicates while writing to HFiles [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3615482769
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd5a4f78082ce982f0f86db8f6fa3675e044295a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd5a4f78082ce982f0f86db8f6fa3675e044295a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #17495:
URL: https://github.com/apache/hudi/pull/17495#issuecomment-3697827252
hey @suryaprasanna : may I know the purpose of the PR. when we might allow duplicates added to hfile or for any of the partitions in MDT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(storage): add config to allow duplicates while writing to HFiles [hudi]
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #17495:
URL: https://github.com/apache/hudi/pull/17495#discussion_r2701417061
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieHFileConfig.java:
##########
@@ -30,19 +30,22 @@ public class HoodieHFileConfig {
private final StorageConfiguration storageConf;
private final BloomFilter bloomFilter;
private final String keyFieldName;
+ private final boolean allowDuplicatesToBeInserted;
Review Comment:
Sure, made the necessary changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org