You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "bibhu107 (via GitHub)" <gi...@apache.org> on 2025/01/01 11:48:01 UTC
[PR] [Draft][HUDI-7944]Adding Config to stop job incase of failure in any table during Multi… [hudi]
bibhu107 opened a new pull request, #12563:
URL: https://github.com/apache/hudi/pull/12563
### **Change Logs and Description Update**
#### **Overview**
This proposal introduces a feature allowing clients to designate specific tables as critical. If any critical table fails during a write operation, the sync job will terminate immediately. The implementation is highly configurable, offering both strict and lenient failure handling policies to accommodate diverse use cases.
---
### **Proposed Configurations**
1. **Critical Table Flag** (Table-level Configuration):
- **Description**: Marks specific tables as critical.
- **Behavior**: The job fails immediately if any critical table encounters a failure.
- **Default**: `false` (tables are non-critical by default).
2. **Failed Table Limit** (Global Configuration for Multi-Streamer Job):
- **Description**: Sets a threshold for the number of table failures allowed before the job fails.
- **Default**: `0` (fail on any table failure unless overridden).
3. **Failed Table Handling Enabled** (Global Configuration for Multi-Streamer Job):
- **Description**: Toggles enforcement of the failed table limit.
- **Default**: `false` (disabled by default).
---
### **Configuration Scenarios**
#### **Global Failure Threshold**
- The job fails if the total number of failed tables exceeds the configured limit.
- **Example Configuration**:
```properties
hoodie.streamer.failed_table_limit_enabled=true
hoodie.streamer.failed_table_limit=2
```
#### **Table-Specific Failure Rule**
- Specific tables can be marked as critical:
```properties
hoodie.streamer.table.t2.criticalTable=true
```
### **Impact**
None for current jobs, clients given option to add configs in future releases.
---
### **Risk Level**
**Low**:
---
### **Documentation Updates**
---
### **Contributor's Checklist**
- [X] Read the contributor's guide.
- [ ] Clearly state Change Logs and Impact.
- [ ] Add adequate tests to validate failure configurations.
- [X] CI passed for implemented changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Draft][HUDI-7944]Adding Config to stop job incase of failure in any table during Multi… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12563:
URL: https://github.com/apache/hudi/pull/12563#issuecomment-2566999237
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2611",
"triggerID" : "eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2611)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Draft][HUDI-7944]Adding Config to stop job incase of failure in any table during Multi… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12563:
URL: https://github.com/apache/hudi/pull/12563#issuecomment-2644537117
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2611",
"triggerID" : "eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755",
"triggerType" : "PUSH"
}, {
"hash" : "4e9770ff24ff112b660826b1de0c3aafa038dac7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3496",
"triggerID" : "4e9770ff24ff112b660826b1de0c3aafa038dac7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2611)
* 4e9770ff24ff112b660826b1de0c3aafa038dac7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3496)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Draft][HUDI-7944]Adding Config to stop job incase of failure in any table during Multi… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12563:
URL: https://github.com/apache/hudi/pull/12563#issuecomment-2644536409
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2611",
"triggerID" : "eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755",
"triggerType" : "PUSH"
}, {
"hash" : "4e9770ff24ff112b660826b1de0c3aafa038dac7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4e9770ff24ff112b660826b1de0c3aafa038dac7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2611)
* 4e9770ff24ff112b660826b1de0c3aafa038dac7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Draft][HUDI-7944]Adding Config to stop job incase of failure in any table during Multi… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12563:
URL: https://github.com/apache/hudi/pull/12563#issuecomment-2566975311
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2611",
"triggerID" : "eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2611)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Draft][HUDI-7944]Adding Config to stop job incase of failure in any table during Multi… [hudi]
Posted by "bibhu107 (via GitHub)" <gi...@apache.org>.
bibhu107 closed pull request #12563: [Draft][HUDI-7944]Adding Config to stop job incase of failure in any table during Multi…
URL: https://github.com/apache/hudi/pull/12563
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Draft][HUDI-7944]Adding Config to stop job incase of failure in any table during Multi… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12563:
URL: https://github.com/apache/hudi/pull/12563#issuecomment-2566974801
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Draft][HUDI-7944]Adding Config to stop job incase of failure in any table during Multi… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12563:
URL: https://github.com/apache/hudi/pull/12563#issuecomment-2644704153
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2611",
"triggerID" : "eb0b4d9a31074d547d0aa4f6fe8a6a288cfd1755",
"triggerType" : "PUSH"
}, {
"hash" : "4e9770ff24ff112b660826b1de0c3aafa038dac7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3496",
"triggerID" : "4e9770ff24ff112b660826b1de0c3aafa038dac7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e9770ff24ff112b660826b1de0c3aafa038dac7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3496)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org