You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/10/14 03:11:29 UTC

[GitHub] [incubator-uniffle] zuston opened a new issue, #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

zuston opened a new issue, #263:
URL: https://github.com/apache/incubator-uniffle/issues/263

   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   
   
   ### Search before asking
   
   - [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### What would you like to be improved?
   
   Currently, the cluster of shuffle-servers is hard to update quickly. In the on-premise deployment, we have to write the custom ansible playbook to rolling update, and the upgrading process is too long.
   
   The rolling update process have two steps:
   1. Add the exclude shuffle-servers id into exclude node file, which should be recognized by coordinator and no long assigned
   2. Waiting the all apps to finished in these excluded shuffle-servers.
   
   Because the shutdown of shuffle-server will lose all app state and cleanup the datafile, we have to handle with care. 
   
   If we could store the all state into levelDB or rockdb or localfile(Like Yarn Nodemanager), maybe it will benefit for us to restart quickly and wont make app fail.
   
   ### How should we improve?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] jerqi commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1280371310

   It will bring much complexity.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] zuston commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
zuston commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1280583926

   > > > It will bring much complexity.
   > > 
   > > 
   > > Currently, it will cost several days to rolling update the shuffle-server. The cluster number is now 200+. This limits our rapid development and upgrade.
   > > Could u share some experience about tencent upgrade solutions?
   > 
   > We have fewer nodes because our shuffle server occupy the entire machine.
   
   Got it. We will expand the cluster to 500+ in the future. So the upgrade is the big problem for us.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] zuston commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
zuston commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1280565553

   > It will bring much complexity.
   
   Currently, it will cost several days to rolling update the shuffle-server. The cluster number is now 200+. This limits our rapid development and upgrade.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] jerqi commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1286663174

   OK, I think it's more convenient that we shutdown the shuffle servers for a period of time. All the applications use ess , we upgrade all the shuffle servers, we let  the applications use the rss again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] jerqi commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1286686537

   > > OK, I think it's more convenient that we shutdown the shuffle servers for a period of time. All the applications use ess , we upgrade all the shuffle servers, we let the applications use the rss again.
   > 
   > Emm... I practiced by your way and it also have to wait all apps finished which will cost 1 Days(due to some ML jobs).
   > 
   > Besides In our co-location environment(Uniffle ShuffleServer on NodeManager), this is acceptable and I introduce the safemode to reject all newer assignment from newly app in our internal uniffle version.
   > 
   > But if the spark job is running on the k8s pod of less local disk or no ESS, how to upgrade?
   
   You're right. StateStore is useful. But if you does some poc, it have great effective, or you use this feature in your environment, it's acceptable. This feature is a big feature, we should be careful. It might be more suitable to merge this feature when Uniffle are more stable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] jerqi commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1302873080

   I hope the behavior can be managed by our admin api, you can go ahead with your plan, but please make compatible design with admin api.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] zuston commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
zuston commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1278416635

   @jerqi @frankliee @colinmjj @kaijchen Feel free to discuss more. I met such problems in our internal uniffle upgrading, so proposal this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] jerqi commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1282230424

   It's useful feature but it's a little complex. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] zuston commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
zuston commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1286680008

   > OK, I think it's more convenient that we shutdown the shuffle servers for a period of time. All the applications use ess , we upgrade all the shuffle servers, we let the applications use the rss again.
   
   Emm... I also do this using your way. And In our co-location environment(Uniffle ShuffleServer on NodeManager), this is acceptable and I introduce the safemode to reject all newer assignment from newly app in our internal uniffle version.
   
   But if the spark job is running on the k8s pod of less local disk or no ESS, how to upgrade? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] zuston commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
zuston commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1306994949

   I think the shuffle-server will follow the next steps when upgrading
   
   ### Dump and stop
   1. Stop the grpc server to avoid extra data sent or got (client will have to hold on waiting and retry due to the connection refused failure)
   2. Make its state unhealthy, which should be recognized by coordinator to avoid extra assignments
   3. Dumping all state to file
   
   ### Restore
   1. Restore state from the file
   2. Start the grpc server to be in service
   
   I want to use kryo to serialize the partial state into file, and then recover from it. Do you have some good ideas on it? @jerqi 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] jerqi commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1280577303

   > > It will bring much complexity.
   > 
   > Currently, it will cost several days to rolling update the shuffle-server. The cluster number is now 200+. This limits our rapid development and upgrade.
   > 
   > Could u share some experience about tencent upgrade solutions?
   
   We have fewer nodes because our shuffle server occupy the entire machine. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] zuston commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
zuston commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1281776570

   Does the Uniffle community need this feature? @jerqi @frankliee 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] zuston commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
zuston commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1286699375

   Agree with u about its big change. So I revisit this problem do we really need a statestore? Make a big change just for upgrading? Actually the shuffle-servers' state only be valid in short period which depends on jobs lifetime.
   
   So I think we only need a mechanism  triggered by external call to let shuffle-server dump all its state to localfile/leveldb. After that, the shuffle-server could be restarted and recover from external localfile state. This change may be not big and acceptable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] zuston commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
zuston commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1279648695

   > Besides from state store, we also need to wait the shuffle servers to flush their memory data to local or hdfs files
   
   Yes, this should be involved in design. Besides, the client should wait instead of connection failure when the corresponding server is restarting.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] zuston commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
zuston commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1302855824

   I think I will take this issue that dumping all state when shuffle-server is triggered by external call. If you don't have any idea, I will go ahead and propose a design. @jerqi 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] jerqi commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1307015758

   > I think the shuffle-server will follow the next steps when upgrading
   > 
   > ### Dump and stop
   > 1. Stop the grpc server to avoid extra data sent or got (client will have to hold on waiting and retry due to the connection refused failure)
   > 2. Make its state unhealthy, which should be recognized by coordinator to avoid extra assignments
   > 3. Dumping all state to file
   > 
   > ### Restore
   > 1. Restore state from the file
   > 2. Start the grpc server to be in service
   > 
   > I want to use `kryo` to serialize the partial state into file, and then recover from it. Do you have some good ideas on it? @jerqi
   
   You should also consider timeout of heartbeat will delete the shuffle. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] frankliee commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
frankliee commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1278545704

   Besides from state store, we also need to wait the memory data flushing to local or hdfs files


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] jerqi commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1286980531

   > Agree with u about its big change. So I revisit this problem do we really need a statestore? Make a big change just for upgrading? Actually the shuffle-servers' state only be valid in short period which depends on jobs lifetime.
   > 
   > So I think we only need a mechanism triggered by external call to let shuffle-server dump all its state to localfile/leveldb. After that, the shuffle-server could be restarted and recover from external localfile state. This change may be not big and acceptable.
   
   More reasonable. But if we need a extra api for shuffle server,  we should consider it with https://github.com/apache/incubator-uniffle/issues/80


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] zuston commented on issue #263: [Improvement] Introduce statestore of shuffle-server for upgrading quickly

Posted by GitBox <gi...@apache.org>.
zuston commented on issue #263:
URL: https://github.com/apache/incubator-uniffle/issues/263#issuecomment-1283802963

   I think the complexity is acceptable compared to the ease of upgrading it brings


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org