[GitHub] [incubator-uniffle] zjf2012 commented on pull request #637: [#615] improvement: Reduce task binary by removing 'partitionToServers' fro…

["zjf2012 (via GitHub)" <gi...@apache.org>]

zjf2012 commented on PR #637:
URL: https://github.com/apache/incubator-uniffle/pull/637#issuecomment-1439313903

   > The code lgtm overall. some minor questions:
   > 
   > > **How was this patch tested?**
   > > tested with 10000 partitions shuffle
   > 
   > Do you have some data about the serialized task before and after, is there any slowdown when using broadcast and how many shuffle servers in your environments?
   > 
   > And just to be safe, how much the size of broadcast is occupied when your 10000 partition test? Just to make sure it don't bring too much memory pressure to driver.
   
   For now, I don't have more servers. So, I only use two shuffle servers. Before my optimization, both map task and reduce task have more than 670KB binary size. After optimization, they reduce to less than 6KB. It's dramatic.
   
   Broadcast uses bittorrent-like way to distribute variable to each executor once. Executors can get some chunk of broadcast variable from other executors instead of all from driver. And task serialize/deserialize time drops a lot. So in theory, it has no way to slow down job.
   
   The size of the broadcast should be less than 670KB deduced from above statement. I'll try to capture it today.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org