[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #603: [#291] feat: Support async read

["jerqi (via GitHub)" <gi...@apache.org>]

jerqi commented on code in PR #603:
URL: https://github.com/apache/incubator-uniffle/pull/603#discussion_r1108359368


##########
common/src/main/java/org/apache/uniffle/common/config/RssClientConf.java:
##########
@@ -43,4 +43,10 @@ public class RssClientConf {
       .defaultValue(ShuffleDataDistributionType.NORMAL)
       .withDescription("The type of partition shuffle data distribution, including normal and local_order. "
           + "The default value is normal. This config is only valid in Spark3.x");
+
+  public static final ConfigOption<Integer> ASYNC_DATA_PREFETCH_BATCH_NUM = ConfigOptions
+      .key("rss.client.data.prefetch.batch.num")
+      .intType()
+      .defaultValue(2)

Review Comment:
   Should we add `checkValue`?
   Should we add some docs?



##########
client/src/main/java/org/apache/uniffle/client/request/CreateShuffleReadClientRequest.java:
##########
@@ -45,6 +45,44 @@ public class CreateShuffleReadClientRequest {
   private IdHelper idHelper;
   private ShuffleDataDistributionType shuffleDataDistributionType = ShuffleDataDistributionType.NORMAL;
   private boolean expectedTaskIdsBitmapFilterEnable = false;
+  private int prefetchBatchNum;
+
+  public CreateShuffleReadClientRequest(

Review Comment:
   Async read will cost more memory?
   Do we need use Spark MemoryManager to manage our read buffer?



##########
client-spark/spark2/src/main/java/org/apache/spark/shuffle/reader/RssShuffleReader.java:
##########
@@ -48,6 +48,8 @@
 import org.apache.uniffle.common.ShuffleServerInfo;
 import org.apache.uniffle.common.config.RssConf;
 
+import static org.apache.uniffle.common.config.RssClientConf.ASYNC_DATA_PREFETCH_BATCH_NUM;

Review Comment:
   Could we avoid static importing here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org