You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "prashantwason (via GitHub)" <gi...@apache.org> on 2026/01/29 19:57:07 UTC
[PR] feat(spark): Add guardrail to prevent writes when Spark speculative execution is enabled [hudi]
prashantwason opened a new pull request, #18045:
URL: https://github.com/apache/hudi/pull/18045
### Describe the issue this Pull Request addresses
Closes #18044
When Spark speculative execution is enabled (`spark.speculation=true`), it can lead to data corruption and duplicate records in Hudi tables. This happens because speculative execution launches duplicate task attempts, which can result in multiple write attempts to the same file slices, broken/partial parquet files, and duplicate records.
This is also related to issue #9615, which reports broken parquet files after compaction with speculative execution enabled.
### Summary and Changelog
Adds a configuration-controlled guardrail that prevents `SparkRDDWriteClient` creation when Spark speculative execution is detected:
- Added new config `hoodie.block.writes.on.speculative.execution` (default: `true`) in `HoodieWriteConfig`
- Added `checkSpeculativeExecution()` method in `SparkRDDWriteClient` that is called during client initialization
- Throws `HoodieException` with a clear error message if speculative execution is enabled and the guardrail is active
- Added comprehensive tests for the guardrail behavior
### Impact
- **New Config**: `hoodie.block.writes.on.speculative.execution` - When enabled (default), throws an exception if Spark speculative execution is enabled during client creation. Users who understand the risks can set this to `false` to bypass the guardrail.
- **User-facing Change**: Users with `spark.speculation=true` will now get a clear error message instead of potential data corruption.
### Risk Level
low - This is an additive change that provides a safety guardrail. The default behavior (blocking writes with speculative execution) is intentionally conservative to prevent data corruption. Users who need speculative execution can explicitly opt-out.
### Documentation Update
The config description is self-documenting. The new configuration property includes detailed documentation explaining:
- What the guardrail does
- Why speculative execution is problematic for Hudi
- How to bypass it if needed
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): Add guardrail to prevent writes when Spark speculative execution is enabled [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18045:
URL: https://github.com/apache/hudi/pull/18045#issuecomment-3820437653
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84ec066ddb9b3c016c522d67412f340d1663dd73",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11510",
"triggerID" : "84ec066ddb9b3c016c522d67412f340d1663dd73",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84ec066ddb9b3c016c522d67412f340d1663dd73 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11510)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): Add guardrail to prevent writes when Spark speculative execution is enabled [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #18045:
URL: https://github.com/apache/hudi/pull/18045
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): Add guardrail to prevent writes when Spark speculative execution is enabled [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #18045:
URL: https://github.com/apache/hudi/pull/18045#discussion_r2743627149
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -779,6 +779,18 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("When table is upgraded from pre 0.12 to 0.12, we check for \"default\" partition and fail if found one. "
+ "Users are expected to rewrite the data in those partitions. Enabling this config will bypass this validation");
+ /**
+ * Config that determines whether to block writes when Spark speculative execution is enabled.
+ */
+ public static final ConfigProperty<Boolean> BLOCK_WRITES_ON_SPECULATIVE_EXECUTION = ConfigProperty
Review Comment:
I am wondering why do we even need a config.
why can't we just enable this out of the box.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): Add guardrail to prevent writes when Spark speculative execution is enabled [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18045:
URL: https://github.com/apache/hudi/pull/18045#issuecomment-3857137871
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84ec066ddb9b3c016c522d67412f340d1663dd73",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11510",
"triggerID" : "84ec066ddb9b3c016c522d67412f340d1663dd73",
"triggerType" : "PUSH"
}, {
"hash" : "fbeb571b061353491c0502698458b214a2601772",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fbeb571b061353491c0502698458b214a2601772",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84ec066ddb9b3c016c522d67412f340d1663dd73 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11510)
* fbeb571b061353491c0502698458b214a2601772 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): Add guardrail to prevent writes when Spark speculative execution is enabled [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18045:
URL: https://github.com/apache/hudi/pull/18045#issuecomment-3858565973
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84ec066ddb9b3c016c522d67412f340d1663dd73",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11510",
"triggerID" : "84ec066ddb9b3c016c522d67412f340d1663dd73",
"triggerType" : "PUSH"
}, {
"hash" : "fbeb571b061353491c0502698458b214a2601772",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11735",
"triggerID" : "fbeb571b061353491c0502698458b214a2601772",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fbeb571b061353491c0502698458b214a2601772 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11735)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): Add guardrail to prevent writes when Spark speculative execution is enabled [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #18045:
URL: https://github.com/apache/hudi/pull/18045#discussion_r2743965057
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -779,6 +779,18 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("When table is upgraded from pre 0.12 to 0.12, we check for \"default\" partition and fail if found one. "
+ "Users are expected to rewrite the data in those partitions. Enabling this config will bypass this validation");
+ /**
+ * Config that determines whether to block writes when Spark speculative execution is enabled.
+ */
+ public static final ConfigProperty<Boolean> BLOCK_WRITES_ON_SPECULATIVE_EXECUTION = ConfigProperty
Review Comment:
If we are absolutely sure that HUDI does not work with speculative execution then yes, we dont need the config. This is the determination we need to make. In that case we would also need to update the docs somewhere to point this out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): Add guardrail to prevent writes when Spark speculative execution is enabled [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18045:
URL: https://github.com/apache/hudi/pull/18045#issuecomment-3820018392
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84ec066ddb9b3c016c522d67412f340d1663dd73",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11510",
"triggerID" : "84ec066ddb9b3c016c522d67412f340d1663dd73",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84ec066ddb9b3c016c522d67412f340d1663dd73 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11510)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): Add guardrail to prevent writes when Spark speculative execution is enabled [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18045:
URL: https://github.com/apache/hudi/pull/18045#issuecomment-3819998541
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84ec066ddb9b3c016c522d67412f340d1663dd73",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "84ec066ddb9b3c016c522d67412f340d1663dd73",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84ec066ddb9b3c016c522d67412f340d1663dd73 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): Add guardrail to prevent writes when Spark speculative execution is enabled [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18045:
URL: https://github.com/apache/hudi/pull/18045#issuecomment-3857146407
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84ec066ddb9b3c016c522d67412f340d1663dd73",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11510",
"triggerID" : "84ec066ddb9b3c016c522d67412f340d1663dd73",
"triggerType" : "PUSH"
}, {
"hash" : "fbeb571b061353491c0502698458b214a2601772",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11735",
"triggerID" : "fbeb571b061353491c0502698458b214a2601772",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84ec066ddb9b3c016c522d67412f340d1663dd73 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11510)
* fbeb571b061353491c0502698458b214a2601772 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11735)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): Add guardrail to prevent writes when Spark speculative execution is enabled [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #18045:
URL: https://github.com/apache/hudi/pull/18045#discussion_r2756396161
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -779,6 +779,18 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("When table is upgraded from pre 0.12 to 0.12, we check for \"default\" partition and fail if found one. "
+ "Users are expected to rewrite the data in those partitions. Enabling this config will bypass this validation");
+ /**
+ * Config that determines whether to block writes when Spark speculative execution is enabled.
+ */
+ public static final ConfigProperty<Boolean> BLOCK_WRITES_ON_SPECULATIVE_EXECUTION = ConfigProperty
Review Comment:
The config provides an escape hatch for users who understand the risks and have specific needs. Here's the rationale:
1. **Default is safe** - The guardrail is enabled by default (`true`), so users are protected out of the box.
2. **Cluster-level settings** - Some users have `spark.speculation=true` set at the cluster or session level by their platform team and may not be able to change it easily. The config allows them to explicitly acknowledge the risk for Hudi writes.
3. **Consistent with other guardrails** - This follows the pattern of other bypass configs in Hudi like `SKIP_DEFAULT_PARTITION_VALIDATION` which also default to the safe behavior but allow users to opt-out when they understand the implications.
4. **Gradual adoption** - For users upgrading who suddenly hit this exception, the config provides a way to temporarily bypass while they work on disabling speculative execution properly.
That said, if the consensus is to make this unconditional, I'm happy to remove the config and simplify the implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org