[GitHub] [incubator-uniffle] smallzhongfeng commented on a diff in pull request #680: [#678] improvement: Write hdfs files asynchronously when `detectStorage`

["smallzhongfeng (via GitHub)" <gi...@apache.org>]

smallzhongfeng commented on code in PR #680:
URL: https://github.com/apache/incubator-uniffle/pull/680#discussion_r1125375094


##########
coordinator/src/main/java/org/apache/uniffle/coordinator/strategy/storage/AppBalanceSelectStorageStrategy.java:
##########
@@ -84,23 +85,35 @@ public void sortPathByRankValue(String path, String test) {
   public void detectStorage() {
     uris = Lists.newCopyOnWriteArrayList(remoteStoragePathRankValue.entrySet());
     if (remoteStoragePathRankValue.size() > 1) {
+      CountDownLatch countDownLatch = new CountDownLatch(uris.size());
       for (Map.Entry<String, RankValue> uri : uris) {
-        if (uri.getKey().startsWith(ApplicationManager.getPathSchema().get(0))) {
-          RankValue rankValue = remoteStoragePathRankValue.get(uri.getKey());
-          rankValue.setHealthy(new AtomicBoolean(true));
-          Path remotePath = new Path(uri.getKey());
-          String rssTest = uri.getKey() + "/rssTest-" + getCoordinatorId();
-          Path testPath = new Path(rssTest);
-          try {
-            FileSystem fs = HadoopFilesystemProvider.getFilesystem(remotePath, hdfsConf);
-            readAndWriteHdfsStorage(fs, testPath, uri.getKey(), rankValue);
-          } catch (Exception e) {
-            rankValue.setHealthy(new AtomicBoolean(false));
-            LOG.error("Storage read and write error, we will not use this remote path {}.", uri, e);
-          } finally {
-            sortPathByRankValue(uri.getKey(), rssTest);
+        Thread detectThread = new Thread(() -> {

Review Comment:
   I use `parallelStream`, the two should be similar. right?



##########
coordinator/src/main/java/org/apache/uniffle/coordinator/strategy/storage/LowestIOSampleCostSelectStorageStrategy.java:
##########
@@ -99,26 +100,38 @@ public void sortPathByRankValue(
   public void detectStorage() {

Review Comment:
   Updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org