You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by "zhengchenyu (via GitHub)" <gi...@apache.org> on 2023/07/26 10:15:39 UTC

[GitHub] [incubator-uniffle] zhengchenyu opened a new issue, #1042: [FEATURE] [MR] Fast fail application when too many fetch failed happen.

zhengchenyu opened a new issue, #1042:
URL: https://github.com/apache/incubator-uniffle/issues/1042

   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   
   
   ### Search before asking
   
   - [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Describe the feature
   
   In #477 and #1011, job will resubmit upstream stage/vertex when many fetch failed. It is not useful for MR, because there are only 2 stage. For MR, we should fast fail the job.
   But now MR can not fast fail. When the selected shuffle server is down or stuck, MR job fail until the number of task attempts is larger than max. It is very slow.
   For now, shuffle does not send fetch failed event. We should fail the job according to the number of fetch failed event.
   
   ### Motivation
   
   _No response_
   
   ### Describe the solution
   
   _No response_
   
   ### Additional context
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-uniffle] zhengchenyu commented on issue #1042: [FEATURE] [MR] Fast fail job when too many fetch failed happen.

Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on issue #1042:
URL: https://github.com/apache/incubator-uniffle/issues/1042#issuecomment-1651491729

   MR example job have 8 map and 30 reduce. Here I stuck the shuffle server. application fail very slow unit the number of task attempts exceed max.
   
   <img width="1897" alt="截屏2023-07-26 下午5 57 02" src="https://github.com/apache/incubator-uniffle/assets/10381583/e31a658a-97ef-4bdb-88aa-a23cf66d87f9">
   <img width="1726" alt="截屏2023-07-26 下午5 56 27" src="https://github.com/apache/incubator-uniffle/assets/10381583/4e19020c-d6c5-4d90-83cc-7b6e44c467ca">
   
   the number of failed task is 74, most of attempts are unnecessary.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org