You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zhengchenyu (via GitHub)" <gi...@apache.org> on 2024/01/16 04:43:10 UTC
[I] [Improvement] [SPARK] Modify task attempt id to avoid information redundancy. [incubator-uniffle]
zhengchenyu opened a new issue, #1458:
URL: https://github.com/apache/incubator-uniffle/issues/1458
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
In spark, task attempt id is used to distinguish different task attempts.
We know task attempt id is from [driver](https://github.com/apache/spark/blob/a58362ecbbf7c4e5d5f848411834cf2a9ef298b3/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L232). It is used to distinguish all task attempts in the entire spark applicaiton. For example, the id of task attempt in stage 0 is in [0,100), the id of task attempt in stage 1 will be [100,200).
But RSS's requirement for task id is that it only needs to be able to distinguish different ids of the same shuffle. For id of task attempt in stage, we can use [0,100], because the IDs between each shuffle are isolated. Starting counting from 100 is very wasteful.
### How should we improve?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Improvement] [SPARK] Modify task attempt id to avoid information redundancy. [incubator-uniffle]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu closed issue #1458: [Improvement] [SPARK] Modify task attempt id to avoid information redundancy.
URL: https://github.com/apache/incubator-uniffle/issues/1458
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Improvement] [SPARK] Modify task attempt id to avoid information redundancy. [incubator-uniffle]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on issue #1458:
URL: https://github.com/apache/incubator-uniffle/issues/1458#issuecomment-1965880904
This issue is solved by https://github.com/apache/incubator-uniffle/pull/1529, close it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org