You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "big-doudou (via GitHub)" <gi...@apache.org> on 2023/07/13 11:24:15 UTC
[GitHub] [hudi] big-doudou opened a new pull request, #9182: fix duplicate fileId on TM exception partial-failover and recovery
big-doudou opened a new pull request, #9182:
URL: https://github.com/apache/hudi/pull/9182
### Change Logs
https://github.com/apache/hudi/issues/8892
### Risk level (write none, low medium or high below)
### Documentation Update
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1652964890
> send the bootstrap event to clean up these files and metadata
The log files are actually cleaned by the cleaner/rollback procedure, senind the bootstrap event does not work as you expected. Before these log files are removed, they are actually invisible to the reader/view(or there should be a bug).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] big-doudou commented on pull request #9182: fix duplicate fileId on TM exception partial-failover and recovery
Posted by "big-doudou (via GitHub)" <gi...@apache.org>.
big-doudou commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1649258169
I think this piece of code at that time was to solve the problem of instant!=null blocking writing during fault recovery, but it ignored the special case of repeating the file id under the bucket index
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] big-doudou commented on pull request #9182: fix duplicate fileId on TM exception partial-failover and recovery
Posted by "big-doudou (via GitHub)" <gi...@apache.org>.
big-doudou commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1649253338
Flink sink hudi uses bucket index. If the amount of data between checkpoints is relatively large, part of the data will be flushed to hdfs first, and a file ID will be generated at this time. If the TM restarts abnormally before the checkpoint is completed, this code will judge flink job partial-failover and recovery, and Bootstrap() will not be executed. Therefore, the previously generated instant is reused, and the old log file will not be cleaned up, resulting in duplicate file ids in the same bucket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1652814806
> failover causes the bootstrap event not to be sent
Why we must send the bootstrap event in this case, the bootstrap event itself is a empty event without any metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9182: fix duplicate fileId on TM exception partial-failover and recovery
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1635293185
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18554",
"triggerID" : "bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e",
"triggerType" : "PUSH"
}, {
"hash" : "bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18575",
"triggerID" : "1635138022",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18554) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18575)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1649298354
> I think this piece of code at that time was to solve the problem of instant!=null blocking writing during fault recovery, but it ignored the special case of repeating the file id under the bucket index
Maybe you are right, but in `StreamWriteOperatorCoordinator#subtaskFailed`, we would anyway cleans the write metadata of the failed task, maybe it is because the event does not got received by the coordinator ?
The `#subtaskFailed` is invoked right before a new attempt got scheduled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1650862935
Each failed attempt of a subtask would trigger invocation of `StreamWriteOperatorCoordinator#subtaskFailed`, the original write metadata would got cleaned,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9182: fix duplicate fileId on TM exception partial-failover and recovery
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9182:
URL: https://github.com/apache/hudi/pull/9182#discussion_r1273084610
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/common/AbstractStreamWriteFunction.java:
##########
@@ -220,7 +220,6 @@ private void sendBootstrapEvent() {
if (this.currentInstant != null) {
LOG.info("Recover task[{}] for instant [{}] with attemptId [{}]", taskID, this.currentInstant, attemptId);
this.currentInstant = null;
- return;
}
Review Comment:
Can you elaborate a little more why the fix solves your problem ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] big-doudou commented on pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
Posted by "big-doudou (via GitHub)" <gi...@apache.org>.
big-doudou commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1649384345
> > I think this piece of code at that time was to solve the problem of instant!=null blocking writing during fault recovery, but it ignored the special case of repeating the file id under the bucket index
>
> Maybe you are right, but in `StreamWriteOperatorCoordinator#subtaskFailed`, we would anyway clean the write metadata of the failed task, maybe it is because the event does not got received by the coordinator ?
>
> The `#subtaskFailed` is invoked right before a new attempt got scheduled.
How to clean the write metadata in StreamWriteOperatorCoordinator#subtaskFailed
Is it the behavior of OperatorCoordinator after receiving BootstrapEvent?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] big-doudou commented on pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
Posted by "big-doudou (via GitHub)" <gi...@apache.org>.
big-doudou commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1654900164
> > how should these log files be cleaned up. Duplicate bucket id files cause tasks to fail to start all the time
>
> The log expected to be cleaned when the instant is committed (we have a marker machanism to ensure the retried files got cleaned), then issue here is why these partitial files are visible to the `BucketStreamWriter`, that's the direction we should dig into.
>
> See `BaseHoodieWriteClient#commit (line 279)`, the `finalizeWrite` would clean up the retried files based on markers.
>
> In `BucketStreamWriteFunction:line 160`, you can debug whether these intermediate logs are visible to the function view `getLatestFileSlices`, if you have a existing table to test this, would be helpful.
Thank you for such a detailed answer. let me test it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] big-doudou commented on pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
Posted by "big-doudou (via GitHub)" <gi...@apache.org>.
big-doudou commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1651615360
> > How does this affect metadata cleaning?
>
> It removes the preceeding partial metadata if there is any.
Before the checkpoint is completed, BucketStreamWrite flush buffer record, At this moment, TM restart.
Step1: TM RUNING -> CANCELING
Step2: JM -> subtaskFailed() set eventBuffer = null
Step3: TM restart, send bootstrap event to coordinator
Step4: JM handle bootstrap event to clean metadata
JM -> subtaskFailed(): only the event before checkpoint is lost, it will not affect the process after restart.
https://github.com/apache/hudi/blob/2a0223933884cb044e7aa56f205cae926358a030/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/common/AbstractStreamWriteFunction.java#L223
this case is: failover causes the bootstrap event not to be sent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] big-doudou closed pull request #9182: fix duplicate fileId on TM exception partial-failover and recovery
Posted by "big-doudou (via GitHub)" <gi...@apache.org>.
big-doudou closed pull request #9182: fix duplicate fileId on TM exception partial-failover and recovery
URL: https://github.com/apache/hudi/pull/9182
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] big-doudou commented on pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
Posted by "big-doudou (via GitHub)" <gi...@apache.org>.
big-doudou commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1653037352
StreamWriteOperatorCoordinator#handleBootstrapEvent() -> initInstant() -> startInstant() -> this.writeClient.startCommit() -> tableServiceClient.rollbackFailedWrites() -> rollbackFailedWrites() -> rollback()
so TM need send bootstrap event to rollback old instant.Otherwise, two files with the same bucket id will be generated under the current partition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1654867431
> how should these log files be cleaned up. Duplicate bucket id files cause tasks to fail to start all the time
The log expected to be cleaned when the instant is committed (we have a marker machanism to ensure the retried files got cleaned), then issue here is why these partitial files are visible to the `BucketStreamWriter`, that's the direction we should dig into.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] big-doudou commented on pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
Posted by "big-doudou (via GitHub)" <gi...@apache.org>.
big-doudou commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1654860685
>
Maybe you are right, if the partial failover will not be rolled back, how should these log files be cleaned up. Duplicate bucket id files cause tasks to fail to start all the time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] big-doudou commented on pull request #9182: fix duplicate fileId on TM exception partial-failover and recovery
Posted by "big-doudou (via GitHub)" <gi...@apache.org>.
big-doudou commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1635138022
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] big-doudou commented on pull request #9182: fix duplicate fileId on TM exception partial-failover and recovery
Posted by "big-doudou (via GitHub)" <gi...@apache.org>.
big-doudou commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1649258564
> I think this piece of code at that time was to solve the problem of instant!=null blocking writing during fault recovery, but it ignored the special case of repeating the file id under the bucket index
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1651449174
> How does this affect metadata cleaning?
It removes the preceeding partial metadata if there is any.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9182: fix duplicate fileId on TM exception partial-failover and recovery
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1635166364
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18554",
"triggerID" : "bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e",
"triggerType" : "PUSH"
}, {
"hash" : "bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18575",
"triggerID" : "1635138022",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18554) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18575)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 closed pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
URL: https://github.com/apache/hudi/pull/9182
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] big-doudou commented on pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
Posted by "big-doudou (via GitHub)" <gi...@apache.org>.
big-doudou commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1652846394
> > failover causes the bootstrap event not to be sent
>
> Why we must send the bootstrap event in this case, the bootstrap event itself is a empty event without any metadata.
instantA is not commit, The instantA.log file has not been cleaned up
send the bootstrap event to clean up these files and metadata
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] big-doudou commented on pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
Posted by "big-doudou (via GitHub)" <gi...@apache.org>.
big-doudou commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1649384915
> > I think this piece of code at that time was to solve the problem of instant!=null blocking writing during fault recovery, but it ignored the special case of repeating the file id under the bucket index
>
> Maybe you are right, but in `StreamWriteOperatorCoordinator#subtaskFailed`, we would anyway clean the write metadata of the failed task, maybe it is because the event does not got received by the coordinator ?
>
> The `#subtaskFailed` is invoked right before a new attempt got scheduled.
How to clean the write metadata in StreamWriteOperatorCoordinator#subtaskFailed
Is it the behavior of OperatorCoordinator after receiving BootstrapEvent?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9182: fix duplicate fileId on TM exception partial-failover and recovery
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1634344403
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18554",
"triggerID" : "bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18554)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9182: fix duplicate fileId on TM exception partial-failover and recovery
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1634128627
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9182: fix duplicate fileId on TM exception partial-failover and recovery
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1634140754
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18554",
"triggerID" : "bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bcd43bfe1bffd7f67bc0c9ca8d43f7c27270086e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18554)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] big-doudou commented on pull request #9182: fix duplicate fileId on TM exception partial-failover and recovery
Posted by "big-doudou (via GitHub)" <gi...@apache.org>.
big-doudou commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1637323244
> Thanks for the contribution, can you explain what the purpose of the PR @big-doudou ?
https://github.com/apache/hudi/issues/8892#issuecomment-1632159235
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9182: fix duplicate fileId on TM exception partial-failover and recovery
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1636970409
Thanks for the contribution, can you explain what the purpose of the PR @big-doudou ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] big-doudou commented on pull request #9182: fix duplicate fileId on TM exception partial-failover and recovery
Posted by "big-doudou (via GitHub)" <gi...@apache.org>.
big-doudou commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1649187842
@danny0405 Can you help review this pr, this error can be reproduced 100%, please check for details https://github.com/apache/hudi/issues/8892#issuecomment-1632159235
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] big-doudou commented on pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
Posted by "big-doudou (via GitHub)" <gi...@apache.org>.
big-doudou commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1650926469
> Each failed attempt of a subtask would trigger invocation of `StreamWriteOperatorCoordinator#subtaskFailed`, the original write metadata would got cleaned,
The StreamWriteOperatorCoordinator#subtaskFailed just set eventBuffer=null. How does this affect metadata cleaning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9182: [HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9182:
URL: https://github.com/apache/hudi/pull/9182#issuecomment-1654847760
> StreamWriteOperatorCoordinator#handleBootstrapEvent() -> initInstant() -> startInstant() -> this.writeClient.startCommit() -> tableServiceClient.rollbackFailedWrites() -> rollbackFailedWrites() -> rollback() so TM need send bootstrap event to rollback old instant.Otherwise, two files with the same bucket id will be generated under the current partition
Only global failover would trigger the rollback, partial failover would not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org