[GitHub] [incubator-celeborn] waitinfuture commented on a diff in pull request #1778: [CELEBORN-860][DOC] Document on ShuffleClient

["waitinfuture (via GitHub)" <gi...@apache.org>]

waitinfuture commented on code in PR #1778:
URL: https://github.com/apache/incubator-celeborn/pull/1778#discussion_r1279103745


##########
docs/developers/shuffleclient.md:
##########
@@ -111,4 +115,62 @@ The process of `SOFT_SPLIT` is as follows:
 ![softsplit](../../assets/img/softsplit.svg)
 
 `LifecycleManager` keeps the split information and tells reducer to read from all splits of the `PartitionLocation`
-to guarantee no data is lost.
\ No newline at end of file
+to guarantee no data is lost.
+
+## Read Data
+### API specification
+`ShuffleClient` provides an API that creates an InputStream to read data from a partition id. Users can also set
+`startMapIndex` and `endMapIndex` to read data within the map range.
+```java
+  public abstract CelebornInputStream readPartition(
+      int shuffleId,
+      int partitionId,
+      int attemptNumber,
+      int startMapIndex,
+      int endMapIndex)
+```
+
+- `shuffleId` is the unique shuffle id of the application
+- `partitionId` is the partition id to read from
+- `attemptNumber` is the attempt id of reduce task, can be safely set to any value
+- `startMapIndex` is the index of start map index of interested map range, set to 0 if you want to read all
+  partition data
+- `endMapIndex` is the index of end map index of interested map range, set to a very large value if you want
+  to read all partition data
+
+The returned input stream is guaranteed to be `Exactly Once`, meaning no data lost and no duplicated reading, or else
+an exception will be thrown, see [Here](../../developers/faulttolerant#exactly-once).
+
+### Get PartitionLocations
+To read data from a partition id, `ShuffleClient` first checks whether the mapping from partition id to all
+`PartitionLocation`s are locally cached, if not, `ShuffleClient` sends GetReducerFileGroup to `LifecycleManager`
+for the mapping, see [Here](../../developers/lifecyclemanager#).

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org