You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/11/16 06:39:49 UTC
[GitHub] [incubator-uniffle] xianjingfeng opened a new issue, #328: [Improvement] Cleanup unused shuffle servers after stage completed
xianjingfeng opened a new issue, #328:
URL: https://github.com/apache/incubator-uniffle/issues/328
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
If there are many stages in one applicaiton, spark driver will send heartbeat to every shuffle servers, and it may cause app expired in shuffle server side. And if we support decommission in the future, it will cause shuffle difficult to exit.
### How should we improve?
Cleanup unused shuffle servers after stage completed
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #328: [Improvement] Cleanup unused shuffle servers after stage completed
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #328:
URL: https://github.com/apache/incubator-uniffle/issues/328#issuecomment-1316469709
Have you joined our `Wechat` group?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #328: [Improvement] Cleanup unused shuffle servers after stage completed
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #328:
URL: https://github.com/apache/incubator-uniffle/issues/328#issuecomment-1316466326
Spark has a mechanism called `ExchangeReuse`. It means that we can use the older stage' shuffle data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on issue #328: [Improvement] Cleanup unused shuffle servers after stage completed
Posted by GitBox <gi...@apache.org>.
xianjingfeng commented on issue #328:
URL: https://github.com/apache/incubator-uniffle/issues/328#issuecomment-1316477383
> Spark has a mechanism called `ExchangeReuse`. It means that we can use the older stage' shuffle data. We have already a pr #249, I don't know whether this pr can solve your problem.
#249 will clean the older stage' shuffle data, and it will not clean shuffle servers in shuffleServerInfoSet, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on issue #328: [Improvement] Cleanup unused shuffle servers after stage completed
Posted by GitBox <gi...@apache.org>.
xianjingfeng commented on issue #328:
URL: https://github.com/apache/incubator-uniffle/issues/328#issuecomment-1316477629
> Have you joined our `Wechat` group?
No, how to join it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #328: [Improvement] Cleanup unused shuffle servers after stage completed
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #328:
URL: https://github.com/apache/incubator-uniffle/issues/328#issuecomment-1316478175
> > Spark has a mechanism called `ExchangeReuse`. It means that we can use the older stage' shuffle data. We have already a pr #249, I don't know whether this pr can solve your problem.
>
> #249 will clean the older stage' shuffle data, and it will not clean shuffle servers in shuffleServerInfoSet, right?
Yes. you're right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston closed issue #328: [Improvement] Cleanup unused shuffle servers after stage completed
Posted by GitBox <gi...@apache.org>.
zuston closed issue #328: [Improvement] Cleanup unused shuffle servers after stage completed
URL: https://github.com/apache/incubator-uniffle/issues/328
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on issue #328: [Improvement] Cleanup unused shuffle servers after stage completed
Posted by GitBox <gi...@apache.org>.
xianjingfeng commented on issue #328:
URL: https://github.com/apache/incubator-uniffle/issues/328#issuecomment-1316504439
> > > Spark has a mechanism called `ExchangeReuse`. It means that we can use the older stage' shuffle data. We have already a pr #249, I don't know whether this pr can solve your problem.
> >
> >
> > #249 will clean the older stage' shuffle data, and it will not clean shuffle servers in shuffleServerInfoSet, right?
>
> Yes. you're right.
So it is ok if we cleanup unused shuffle servers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #328: [Improvement] Cleanup unused shuffle servers after stage completed
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #328:
URL: https://github.com/apache/incubator-uniffle/issues/328#issuecomment-1316479088
> > Have you joined our `Wechat` group?
>
> No, how to join it?
You can scan the QR code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #328: [Improvement] Cleanup unused shuffle servers after stage completed
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #328:
URL: https://github.com/apache/incubator-uniffle/issues/328#issuecomment-1316507805
> > > > Spark has a mechanism called `ExchangeReuse`. It means that we can use the older stage' shuffle data. We have already a pr #249, I don't know whether this pr can solve your problem.
> > >
> > >
> > > #249 will clean the older stage' shuffle data, and it will not clean shuffle servers in shuffleServerInfoSet, right?
> >
> >
> > Yes. you're right.
>
> So it is ok if we cleanup unused shuffle servers?
Yes, if we already removed the shuffle data in those shuffle servers, we can remove the shuffle servers in the shuffleServerInfoSet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org