You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "yl09099 (via GitHub)" <gi...@apache.org> on 2024/12/19 02:33:16 UTC
[I] [Bug] Incomplete clearing of Mapstatus in the Write Stage retry phase results in data loss. [incubator-uniffle]
yl09099 opened a new issue, #2300:
URL: https://github.com/apache/incubator-uniffle/issues/2300
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
During the Write Stage retry phase, the MapOutputTrackerMaster clears the MapStatus corresponding to the shuffleId on the Driver side. However, when a large number of partitions are encountered, the MapStatus may not be completely cleared. Retry at the Stage, the task becomes less, resulting in data loss. At present, I encountered a 40000 Partition, resulting in data loss.
Below is a screenshot of my problem：
### Affects Version(s)
0.10.0
### Uniffle Server Log Output
_No response_
### Uniffle Engine Log Output
_No response_
### Uniffle Server Configurations
_No response_
### Uniffle Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Bug] Long tail tasks in the Write Stage retry phase results in data loss. [incubator-uniffle]
Posted by "yl09099 (via GitHub)" <gi...@apache.org>.
yl09099 commented on issue #2300:
URL: https://github.com/apache/incubator-uniffle/issues/2300#issuecomment-2553090645
This is mainly because the long-tail task before the Stage retries, resulting in data loss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Bug] Incomplete clearing of Mapstatus in the Write Stage retry phase results in data loss. [incubator-uniffle]
Posted by "yl09099 (via GitHub)" <gi...@apache.org>.
yl09099 commented on issue #2300:
URL: https://github.com/apache/incubator-uniffle/issues/2300#issuecomment-2552650630
@rickyma @jerqi @maobaolong @zuston @advancedxy Please help to look at this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org