[GitHub] [incubator-uniffle] zjf2012 commented on a diff in pull request #637: [#615] improvement: Reduce task binary by removing 'partitionToServers' from RssShuffleHandle

["zjf2012 (via GitHub)" <gi...@apache.org>]

zjf2012 commented on code in PR #637:
URL: https://github.com/apache/incubator-uniffle/pull/637#discussion_r1122513824


##########
client-spark/common/src/main/java/org/apache/spark/shuffle/RssShuffleHandle.java:
##########
@@ -69,19 +69,56 @@ public ShuffleDependency<K, V, C> getDependency() {
     return dependency;
   }
 
-  public Map<Integer, List<ShuffleServerInfo>> getPartitionToServers() {
-    return partitionToServers;
-  }
-
   public int getShuffleId() {
     return shuffleId();
   }
 
-  public Set<ShuffleServerInfo> getShuffleServersForData() {
-    return shuffleServersForData;
+  /**
+   * Tried to get cached {@link ShuffleHandleInfo} from local thread first and then memory if not existing in local.
+   * If not cached, one of competing threads gets chance to deserialize it and caches it for other threads.
+   *
+   * @return
+   */
+  private ShuffleHandleInfo getCurrentHandleInfo() {
+    // local first
+    ShuffleHandleInfo info = _localHandleInfo.get().get();

Review Comment:
   Yes, it's disabled by default. I forgot to mention another factor that the broadcast variable (ShuffleHandleInfo, not RssShuffleHandle) is about 4MB without JavaSerializer.
   
    
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org