You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/12/15 06:28:43 UTC

[GitHub] [incubator-uniffle] zuston opened a new pull request, #424: [Improvement] Avoid selecting storage when it reaches the high watermark

zuston opened a new pull request, #424:
URL: https://github.com/apache/incubator-uniffle/pull/424

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://github.com/apache/incubator-uniffle/blob/master/CONTRIBUTING.md
     2. Ensure you have added or run the appropriate tests for your PR
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]XXXX Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054288092


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,33 +143,53 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    LocalStorage storage = partitionsOfStorage.get(UnionKey.toKey(appId, shuffleId, partitionId));
+    if (storage != null) {
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+        }
+      } else {
+        return storage;
+      }
     }
-    event.setUnderStorage(storage);
-    return storage;
+
+    List<LocalStorage> candidates = localStorages
+        .stream()
+        .filter(x -> x.canWrite() && !x.isCorrupted())
+        .collect(Collectors.toList());
+    final LocalStorage selectedStorage = candidates.get(
+        ShuffleStorageUtils.getStorageIndex(
+            candidates.size(),
+            appId,
+            shuffleId,
+            partitionId
+        )
+    );
+    return partitionsOfStorage.compute(
+        UnionKey.toKey(appId, shuffleId, partitionId),
+        (key, localStorage) -> {
+          // If this is the first time to select storage or existing storage is corrupted,
+          // we should refresh the cache.
+          if (localStorage == null || localStorage.isCorrupted()) {

Review Comment:
   Emm... I have removed the throw exception logic, so it has kept consistent with the original logic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#issuecomment-1361235334

   Merged, thanks @zuston 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
jerqi commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054035843


##########
common/src/main/java/org/apache/uniffle/common/UnionKey.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.common;
+
+import org.apache.commons.lang3.StringUtils;
+
+/**
+ * This class is to wrap multi elements to be as union key.
+ */
+public class UnionKey {
+  private static final String SPLIT_KEY = "_";
+
+  public static String toKey(Object... factors) {

Review Comment:
   Could we give this method a better name? The name is a little general.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
xianjingfeng commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049340143


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");

Review Comment:
   In currnet codebase. if one storage is corrupted, data will be written to another storage. In this case, we will lost some data of this replica, but client can still read some data. But if exception thrown here, all data will be drop. If we use multi replicas, it will be useful.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049358342


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");

Review Comment:
   Make sense for multiple replica.
   
   But for single replica, it's not necessary. Moreover, if in single replica and enable fallback strategy, switching other storage will cause data lost. cc @jerqi 
   
   I prefer fast fail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050715430


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+      }
+      return storage;
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    LocalStorage storage =
+        localStorages
+            .stream()
+            .filter(x -> x.canWrite() && !x.isCorrupted())
+            .collect(Collectors.toList())
+            .get(
+                ShuffleStorageUtils.getStorageIndex(
+                    localStorages.size(),
+                    appId,
+                    shuffleId,
+                    partitionId
+                )
+            );
     event.setUnderStorage(storage);
+
+    // store it to cache.
+    partitionsOfStorage.putIfAbsent(appId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).putIfAbsent(shuffleId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).get(shuffleId).put(partitionId, storage);
     return storage;
   }
 
   @Override
   public Storage selectStorage(ShuffleDataReadEvent event) {
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
 
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    LocalStorage storage = null;
+    try {
+      storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+    } catch (NullPointerException npe) {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1053942780


##########
integration-test/common/src/test/java/org/apache/uniffle/test/DiskErrorToleranceTest.java:
##########
@@ -89,6 +90,7 @@ public void closeClient() {
   }
 
   @Test

Review Comment:
   Yes, let's remove it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1053964422


##########
server/src/main/java/org/apache/uniffle/server/ShuffleDataReadEvent.java:
##########
@@ -21,12 +21,14 @@ public class ShuffleDataReadEvent {
 
   private String appId;
   private int shuffleId;
+  private int partitionId;
   private int startPartition;
 
-  public ShuffleDataReadEvent(String appId, int shuffleId, int startPartition) {
+  public ShuffleDataReadEvent(String appId, int shuffleId, int partitionId, int[] range) {

Review Comment:
   > Just thought it was a small change and shouldn't take you too much time.
   
   Emm... The changing time is OK for me.  Let me do it.
   
   I just prefer preserving this range to make more clear for the invoking side, the standalone `startPartition` looks strange.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1053961462


##########
server/src/main/java/org/apache/uniffle/server/ShuffleDataReadEvent.java:
##########
@@ -21,12 +21,14 @@ public class ShuffleDataReadEvent {
 
   private String appId;
   private int shuffleId;
+  private int partitionId;
   private int startPartition;
 
-  public ShuffleDataReadEvent(String appId, int shuffleId, int startPartition) {
+  public ShuffleDataReadEvent(String appId, int shuffleId, int partitionId, int[] range) {

Review Comment:
   If you want to leave it as it is now. I'm ok. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049327280


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+      }
+      return storage;
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    LocalStorage storage =
+        localStorages
+            .stream()
+            .filter(x -> x.canWrite() && !x.isCorrupted())
+            .collect(Collectors.toList())
+            .get(
+                ShuffleStorageUtils.getStorageIndex(
+                    localStorages.size(),
+                    appId,
+                    shuffleId,
+                    partitionId
+                )
+            );
     event.setUnderStorage(storage);
+
+    // store it to cache.
+    partitionsOfStorage.putIfAbsent(appId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).putIfAbsent(shuffleId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).get(shuffleId).put(partitionId, storage);
     return storage;
   }
 
   @Override
   public Storage selectStorage(ShuffleDataReadEvent event) {
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
 
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    LocalStorage storage = null;
+    try {
+      storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+    } catch (NullPointerException npe) {

Review Comment:
   Emm.. Yes, but I hate to write too much template code.. Do you have any ideas on this? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
xianjingfeng commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049305908


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+      }
+      return storage;
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    LocalStorage storage =
+        localStorages
+            .stream()
+            .filter(x -> x.canWrite() && !x.isCorrupted())
+            .collect(Collectors.toList())
+            .get(
+                ShuffleStorageUtils.getStorageIndex(
+                    localStorages.size(),

Review Comment:
   It seems a bug



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] codecov-commenter commented on pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#issuecomment-1354159163

   # [Codecov](https://codecov.io/gh/apache/incubator-uniffle/pull/424?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#424](https://codecov.io/gh/apache/incubator-uniffle/pull/424?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (129088b) into [master](https://codecov.io/gh/apache/incubator-uniffle/commit/877b4ed0b18dc31185a05172e45d4cccc88195b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (877b4ed) will **decrease** coverage by `0.88%`.
   > The diff coverage is `57.69%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master     #424      +/-   ##
   ============================================
   - Coverage     58.43%   57.54%   -0.89%     
   + Complexity     1613     1394     -219     
   ============================================
     Files           195      173      -22     
     Lines         11063     9272    -1791     
     Branches        976      823     -153     
   ============================================
   - Hits           6465     5336    -1129     
   + Misses         4220     3593     -627     
   + Partials        378      343      -35     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-uniffle/pull/424?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...pache/uniffle/server/ShuffleServerGrpcService.java](https://codecov.io/gh/apache/incubator-uniffle/pull/424/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9TaHVmZmxlU2VydmVyR3JwY1NlcnZpY2UuamF2YQ==) | `0.80% <0.00%> (-0.01%)` | :arrow_down: |
   | [...he/uniffle/server/buffer/ShuffleBufferManager.java](https://codecov.io/gh/apache/incubator-uniffle/pull/424/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9idWZmZXIvU2h1ZmZsZUJ1ZmZlck1hbmFnZXIuamF2YQ==) | `82.74% <ø> (ø)` | |
   | [...rg/apache/uniffle/storage/common/LocalStorage.java](https://codecov.io/gh/apache/incubator-uniffle/pull/424/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmFnZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvdW5pZmZsZS9zdG9yYWdlL2NvbW1vbi9Mb2NhbFN0b3JhZ2UuamF2YQ==) | `43.22% <ø> (ø)` | |
   | [...pache/uniffle/storage/common/LocalStorageMeta.java](https://codecov.io/gh/apache/incubator-uniffle/pull/424/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmFnZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvdW5pZmZsZS9zdG9yYWdlL2NvbW1vbi9Mb2NhbFN0b3JhZ2VNZXRhLmphdmE=) | `74.28% <0.00%> (-1.45%)` | :arrow_down: |
   | [.../org/apache/uniffle/server/ShuffleTaskManager.java](https://codecov.io/gh/apache/incubator-uniffle/pull/424/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9TaHVmZmxlVGFza01hbmFnZXIuamF2YQ==) | `74.66% <31.57%> (-2.57%)` | :arrow_down: |
   | [...he/uniffle/server/storage/LocalStorageManager.java](https://codecov.io/gh/apache/incubator-uniffle/pull/424/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9zdG9yYWdlL0xvY2FsU3RvcmFnZU1hbmFnZXIuamF2YQ==) | `83.92% <74.62%> (-7.32%)` | :arrow_down: |
   | [...rg/apache/uniffle/server/ShuffleDataReadEvent.java](https://codecov.io/gh/apache/incubator-uniffle/pull/424/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9TaHVmZmxlRGF0YVJlYWRFdmVudC5qYXZh) | `90.00% <80.00%> (-10.00%)` | :arrow_down: |
   | [.../java/org/apache/uniffle/server/ShuffleServer.java](https://codecov.io/gh/apache/incubator-uniffle/pull/424/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9TaHVmZmxlU2VydmVyLmphdmE=) | `64.49% <0.00%> (-2.90%)` | :arrow_down: |
   | [...org/apache/spark/shuffle/writer/AddBlockEvent.java](https://codecov.io/gh/apache/incubator-uniffle/pull/424/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50LXNwYXJrL2NvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc3Bhcmsvc2h1ZmZsZS93cml0ZXIvQWRkQmxvY2tFdmVudC5qYXZh) | | |
   | [...ava/org/apache/spark/shuffle/RssShuffleHandle.java](https://codecov.io/gh/apache/incubator-uniffle/pull/424/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50LXNwYXJrL2NvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc3Bhcmsvc2h1ZmZsZS9Sc3NTaHVmZmxlSGFuZGxlLmphdmE=) | | |
   | ... and [23 more](https://codecov.io/gh/apache/incubator-uniffle/pull/424/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050339070


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+      }
+      return storage;
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    LocalStorage storage =
+        localStorages
+            .stream()
+            .filter(x -> x.canWrite() && !x.isCorrupted())
+            .collect(Collectors.toList())
+            .get(
+                ShuffleStorageUtils.getStorageIndex(
+                    localStorages.size(),
+                    appId,
+                    shuffleId,
+                    partitionId
+                )
+            );
     event.setUnderStorage(storage);
+
+    // store it to cache.
+    partitionsOfStorage.putIfAbsent(appId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).putIfAbsent(shuffleId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).get(shuffleId).put(partitionId, storage);
     return storage;
   }
 
   @Override
   public Storage selectStorage(ShuffleDataReadEvent event) {
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
 
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    LocalStorage storage = null;
+    try {
+      storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+    } catch (NullPointerException npe) {

Review Comment:
   Another problem: If we use nested map, it's hard to guarantee thread safe. Right? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
jerqi commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050395328


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -69,8 +71,8 @@ public class LocalStorageManager extends SingleStorageManager {
   private final List<LocalStorage> localStorages;
   private final List<String> storageBasePaths;
   private final LocalStorageChecker checker;
-  private List<LocalStorage> unCorruptedStorages = Lists.newArrayList();
-  private final Set<String> corruptedStorages = Sets.newConcurrentHashSet();
+
+  private final Map<PartitionUnionKey, LocalStorage> partitionsOfStorage;

Review Comment:
   Maybe it will occupy many memory.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1051399753


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -171,11 +171,18 @@ public Storage selectStorage(ShuffleDataFlushEvent event) {
             partitionId
         )
     );
-    event.setUnderStorage(storage);
 
     // store it to cache.
-    partitionsOfStorage.put(UnionKey.toKey(appId, shuffleId, partitionId), storage);
-    return storage;
+    final LocalStorage selectedStorage = storage;
+    return partitionsOfStorage.compute(

Review Comment:
   Use the concurrent hashmap compute method to ensure the thread safe
   
   cc @advancedxy , please help check this part.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
xianjingfeng commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049309200


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+      }
+      return storage;
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    LocalStorage storage =
+        localStorages
+            .stream()
+            .filter(x -> x.canWrite() && !x.isCorrupted())
+            .collect(Collectors.toList())
+            .get(
+                ShuffleStorageUtils.getStorageIndex(
+                    localStorages.size(),
+                    appId,
+                    shuffleId,
+                    partitionId
+                )
+            );
     event.setUnderStorage(storage);
+
+    // store it to cache.
+    partitionsOfStorage.putIfAbsent(appId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).putIfAbsent(shuffleId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).get(shuffleId).put(partitionId, storage);
     return storage;
   }
 
   @Override
   public Storage selectStorage(ShuffleDataReadEvent event) {
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
 
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    LocalStorage storage = null;
+    try {
+      storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+    } catch (NullPointerException npe) {

Review Comment:
   Is there any performance problem with this way?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
xianjingfeng commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049352676


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+      }
+      return storage;
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    LocalStorage storage =
+        localStorages
+            .stream()
+            .filter(x -> x.canWrite() && !x.isCorrupted())
+            .collect(Collectors.toList())
+            .get(
+                ShuffleStorageUtils.getStorageIndex(
+                    localStorages.size(),
+                    appId,
+                    shuffleId,
+                    partitionId
+                )
+            );
     event.setUnderStorage(storage);
+
+    // store it to cache.
+    partitionsOfStorage.putIfAbsent(appId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).putIfAbsent(shuffleId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).get(shuffleId).put(partitionId, storage);
     return storage;
   }
 
   @Override
   public Storage selectStorage(ShuffleDataReadEvent event) {
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
 
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    LocalStorage storage = null;
+    try {
+      storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+    } catch (NullPointerException npe) {

Review Comment:
   I have no other good ideas. WDYT. @jerqi  I think when it relate to reading and writing, we should give priority to performance.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054283884


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,33 +143,53 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    LocalStorage storage = partitionsOfStorage.get(UnionKey.toKey(appId, shuffleId, partitionId));
+    if (storage != null) {
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+        }
+      } else {
+        return storage;
+      }
     }
-    event.setUnderStorage(storage);
-    return storage;
+
+    List<LocalStorage> candidates = localStorages
+        .stream()
+        .filter(x -> x.canWrite() && !x.isCorrupted())
+        .collect(Collectors.toList());
+    final LocalStorage selectedStorage = candidates.get(
+        ShuffleStorageUtils.getStorageIndex(
+            candidates.size(),
+            appId,
+            shuffleId,
+            partitionId
+        )
+    );
+    return partitionsOfStorage.compute(
+        UnionKey.toKey(appId, shuffleId, partitionId),
+        (key, localStorage) -> {
+          // If this is the first time to select storage or existing storage is corrupted,
+          // we should refresh the cache.
+          if (localStorage == null || localStorage.isCorrupted()) {

Review Comment:
   > For an event, if the storage is selected but event don't write any data to this storage (maybe the event is in pending queue), that means we could replace it to new storage, which won't cause data lost.
   
   This only make sense when [L154-L156](https://github.com/apache/incubator-uniffle/pull/424/files#diff-74b29a246646d7b44aa81c4436760fda3edc5f645a3c576dd1b50405848db2a6R154-R156) doesn't throw an exception, right?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy merged pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy merged PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054048881


##########
server/src/main/java/org/apache/uniffle/server/ShuffleTaskManager.java:
##########
@@ -403,8 +427,11 @@ public ShuffleIndexResult getShuffleIndex(
     request.setPartitionNum(partitionNum);
     request.setStorageType(storageType);
     request.setRssBaseConf(conf);
-
-    Storage storage = storageManager.selectStorage(new ShuffleDataReadEvent(appId, shuffleId, partitionId));
+    int[] range = ShuffleStorageUtils.getPartitionRange(partitionId, partitionNumPerRange, partitionNum);
+    Storage storage = storageManager.selectStorage(new ShuffleDataReadEvent(appId, shuffleId, partitionId, range[0]));
+    if (storage == null) {
+      throw new FileNotFoundException("No such data in current storage manager.");

Review Comment:
   The exception of FileNotFoundException could be handled by `getLocalShuffleIndex` grpc layer, because maybe there is no local storage when directly flushing to HDFS.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049347092


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+      }
+      return storage;
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    LocalStorage storage =
+        localStorages
+            .stream()
+            .filter(x -> x.canWrite() && !x.isCorrupted())
+            .collect(Collectors.toList())
+            .get(
+                ShuffleStorageUtils.getStorageIndex(
+                    localStorages.size(),
+                    appId,
+                    shuffleId,
+                    partitionId
+                )
+            );
     event.setUnderStorage(storage);
+
+    // store it to cache.
+    partitionsOfStorage.putIfAbsent(appId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).putIfAbsent(shuffleId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).get(shuffleId).put(partitionId, storage);
     return storage;
   }
 
   @Override
   public Storage selectStorage(ShuffleDataReadEvent event) {
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
 
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    LocalStorage storage = null;
+    try {
+      storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+    } catch (NullPointerException npe) {

Review Comment:
   Nop. It should be removed by appId or (appId, shuffleId) in `removeResources` method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049800641


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -69,8 +69,9 @@ public class LocalStorageManager extends SingleStorageManager {
   private final List<LocalStorage> localStorages;
   private final List<String> storageBasePaths;
   private final LocalStorageChecker checker;
-  private List<LocalStorage> unCorruptedStorages = Lists.newArrayList();
-  private final Set<String> corruptedStorages = Sets.newConcurrentHashSet();
+
+  // AppId -> ShuffleId -> PartitionId -> LocalStorage
+  private final Map<String, Map<Integer, Map<Integer, LocalStorage>>> partitionsOfStorage = Maps.newConcurrentMap();

Review Comment:
   > The mechanism of uploading local files has been removed in current codebase. And this PR is just to avoid selecting storage which has reached high watermark.
   
   could you post a issue or PR link here. I didn't know that context.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050443177


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");

Review Comment:
   @advancedxy Due to this change, it will make `diskErrorTest` fail. So I will remove it temporarily



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050715146


##########
server/src/main/java/org/apache/uniffle/server/ShuffleDataReadEvent.java:
##########
@@ -17,16 +17,20 @@
 
 package org.apache.uniffle.server;
 
+import org.apache.uniffle.common.PartitionRange;
+
 public class ShuffleDataReadEvent {
 
   private String appId;
   private int shuffleId;
-  private int startPartition;
+  private int partitionId;
+  private PartitionRange partitionRange;
 
-  public ShuffleDataReadEvent(String appId, int shuffleId, int startPartition) {
+  public ShuffleDataReadEvent(String appId, int shuffleId, int partitionId, int[] range) {

Review Comment:
   Done



##########
integration-test/common/src/test/java/org/apache/uniffle/test/DiskErrorToleranceTest.java:
##########
@@ -87,8 +86,7 @@ public void createClient() {
   public void closeClient() {
     shuffleServerClient.close();
   }
-
-  @Test
+  
   public void diskErrorTest() throws Exception {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050488135


##########
integration-test/common/src/test/java/org/apache/uniffle/test/DiskErrorToleranceTest.java:
##########
@@ -87,8 +86,7 @@ public void createClient() {
   public void closeClient() {
     shuffleServerClient.close();
   }
-
-  @Test
+  
   public void diskErrorTest() throws Exception {

Review Comment:
   > @advancedxy Due to this change, it will make diskErrorTest fail. So I remove it temporarily
   
   For this case, I believe a @Disabled annotation with a reason would be better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#issuecomment-1354592276

   Could you help review again? @advancedxy @jerqi @advancedxy 
   
    And I'm doing some multiple disk selection for one partition based on this. I think I could push this forward quickly. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
jerqi commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050350005


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +142,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(PartitionUnionKey.of(appId, shuffleId, partitionId));
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+        }
+      } else {
+        return storage;
+      }
+    } catch (NullPointerException npe) {

Review Comment:
   Why do we catch `NullPointerException`? It's strange ....



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050512050


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -69,8 +71,8 @@ public class LocalStorageManager extends SingleStorageManager {
   private final List<LocalStorage> localStorages;
   private final List<String> storageBasePaths;
   private final LocalStorageChecker checker;
-  private List<LocalStorage> unCorruptedStorages = Lists.newArrayList();
-  private final Set<String> corruptedStorages = Sets.newConcurrentHashSet();
+
+  private final Map<PartitionUnionKey, LocalStorage> partitionsOfStorage;

Review Comment:
   So far, yes



##########
integration-test/common/src/test/java/org/apache/uniffle/test/DiskErrorToleranceTest.java:
##########
@@ -87,8 +86,7 @@ public void createClient() {
   public void closeClient() {
     shuffleServerClient.close();
   }
-
-  @Test
+  
   public void diskErrorTest() throws Exception {

Review Comment:
   Got it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050714442


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -246,38 +300,70 @@ public void checkAndClearLeakedShuffleData(Collection<String> appIds) {
     }
   }
 
-  void repair() {
-    boolean hasNewCorruptedStorage = false;
-    for (LocalStorage storage : localStorages) {
-      if (storage.isCorrupted() && !corruptedStorages.contains(storage.getBasePath())) {
-        hasNewCorruptedStorage = true;
-        corruptedStorages.add(storage.getBasePath());
-      }
+  public List<LocalStorage> getStorages() {
+    return localStorages;
+  }
+
+  static class PartitionUnionKey {

Review Comment:
   WDYT? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
xianjingfeng commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049335160


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+      }
+      return storage;
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    LocalStorage storage =
+        localStorages
+            .stream()
+            .filter(x -> x.canWrite() && !x.isCorrupted())
+            .collect(Collectors.toList())
+            .get(
+                ShuffleStorageUtils.getStorageIndex(
+                    localStorages.size(),
+                    appId,
+                    shuffleId,
+                    partitionId
+                )
+            );
     event.setUnderStorage(storage);
+
+    // store it to cache.
+    partitionsOfStorage.putIfAbsent(appId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).putIfAbsent(shuffleId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).get(shuffleId).put(partitionId, storage);
     return storage;
   }
 
   @Override
   public Storage selectStorage(ShuffleDataReadEvent event) {
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
 
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    LocalStorage storage = null;
+    try {
+      storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+    } catch (NullPointerException npe) {

Review Comment:
   How about union key? use object or splice characters?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049590943


##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerGrpcService.java:
##########
@@ -571,10 +574,11 @@ public void getLocalShuffleIndex(GetLocalShuffleIndexRequest request,
     String requestInfo = "appId[" + appId + "], shuffleId[" + shuffleId + "], partitionId["
         + partitionId + "]";
 
-    shuffleServer.getStorageManager()
-        .selectStorage(new ShuffleDataReadEvent(appId, shuffleId, partitionId))
-        .updateReadMetrics(new StorageReadMetrics(appId, shuffleId));
-
+    Storage storage = shuffleServer.getStorageManager()
+        .selectStorage(new ShuffleDataReadEvent(appId, shuffleId, partitionId));
+    if (storage != null) {
+      storage.updateReadMetrics(new StorageReadMetrics(appId, shuffleId));

Review Comment:
   I will



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#issuecomment-1360909151

   Left a minor comment, otherwise I'm ok with this PR.
   
   @jerqi please take another look if you have time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1053955120


##########
server/src/main/java/org/apache/uniffle/server/ShuffleDataReadEvent.java:
##########
@@ -21,12 +21,14 @@ public class ShuffleDataReadEvent {
 
   private String appId;
   private int shuffleId;
+  private int partitionId;
   private int startPartition;
 
-  public ShuffleDataReadEvent(String appId, int shuffleId, int startPartition) {
+  public ShuffleDataReadEvent(String appId, int shuffleId, int partitionId, int[] range) {

Review Comment:
   Emm... If we could calculate start partition directly in another PR, this part could be optimized. Now, we could preserve this. WDYT? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
jerqi commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054040815


##########
server/src/main/java/org/apache/uniffle/server/ShuffleTaskManager.java:
##########
@@ -403,8 +427,11 @@ public ShuffleIndexResult getShuffleIndex(
     request.setPartitionNum(partitionNum);
     request.setStorageType(storageType);
     request.setRssBaseConf(conf);
-
-    Storage storage = storageManager.selectStorage(new ShuffleDataReadEvent(appId, shuffleId, partitionId));
+    int[] range = ShuffleStorageUtils.getPartitionRange(partitionId, partitionNumPerRange, partitionNum);
+    Storage storage = storageManager.selectStorage(new ShuffleDataReadEvent(appId, shuffleId, partitionId, range[0]));
+    if (storage == null) {
+      throw new FileNotFoundException("No such data in current storage manager.");

Review Comment:
   Why do we throw a `FileNotFoundException`?



##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,33 +143,53 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    LocalStorage storage = partitionsOfStorage.get(UnionKey.toKey(appId, shuffleId, partitionId));
+    if (storage != null) {
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");

Review Comment:
   Do we ensure that we should add this exception?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054103717


##########
common/src/main/java/org/apache/uniffle/common/UnionKey.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.common;
+
+import org.apache.commons.lang3.StringUtils;
+
+/**
+ * This class is to wrap multi elements to be as union key.
+ */
+public class UnionKey {
+  private static final String SPLIT_KEY = "_";
+
+  public static String toKey(Object... factors) {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#issuecomment-1361002611

   Updated @jerqi 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054022904


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,33 +143,53 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    LocalStorage storage = partitionsOfStorage.get(UnionKey.toKey(appId, shuffleId, partitionId));
+    if (storage != null) {
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+        }
+      } else {
+        return storage;
+      }
     }
-    event.setUnderStorage(storage);
-    return storage;
+
+    List<LocalStorage> candidates = localStorages
+        .stream()
+        .filter(x -> x.canWrite() && !x.isCorrupted())
+        .collect(Collectors.toList());
+    final LocalStorage selectedStorage = candidates.get(
+        ShuffleStorageUtils.getStorageIndex(
+            candidates.size(),
+            appId,
+            shuffleId,
+            partitionId
+        )
+    );
+    return partitionsOfStorage.compute(
+        UnionKey.toKey(appId, shuffleId, partitionId),
+        (key, localStorage) -> {
+          // If this is the first time to select storage or existing storage is corrupted,
+          // we should refresh the cache.
+          if (localStorage == null || localStorage.isCorrupted()) {

Review Comment:
   On a second thought, in which case localStorage would be `isCorrupted`? Since in L152 - L155, we would already throw an exception there?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054288092


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,33 +143,53 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    LocalStorage storage = partitionsOfStorage.get(UnionKey.toKey(appId, shuffleId, partitionId));
+    if (storage != null) {
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+        }
+      } else {
+        return storage;
+      }
     }
-    event.setUnderStorage(storage);
-    return storage;
+
+    List<LocalStorage> candidates = localStorages
+        .stream()
+        .filter(x -> x.canWrite() && !x.isCorrupted())
+        .collect(Collectors.toList());
+    final LocalStorage selectedStorage = candidates.get(
+        ShuffleStorageUtils.getStorageIndex(
+            candidates.size(),
+            appId,
+            shuffleId,
+            partitionId
+        )
+    );
+    return partitionsOfStorage.compute(
+        UnionKey.toKey(appId, shuffleId, partitionId),
+        (key, localStorage) -> {
+          // If this is the first time to select storage or existing storage is corrupted,
+          // we should refresh the cache.
+          if (localStorage == null || localStorage.isCorrupted()) {

Review Comment:
   Emm... I have removed the throw exception logic, so it has kept consistent with the original logic in this part.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050443177


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");

Review Comment:
   @advancedxy Due to this change, it will make `diskErrorTest` fail. So I remove it temporarily



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049589643


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+      }
+      return storage;
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    LocalStorage storage =
+        localStorages
+            .stream()
+            .filter(x -> x.canWrite() && !x.isCorrupted())
+            .collect(Collectors.toList())
+            .get(
+                ShuffleStorageUtils.getStorageIndex(
+                    localStorages.size(),
+                    appId,
+                    shuffleId,
+                    partitionId
+                )
+            );
     event.setUnderStorage(storage);
+
+    // store it to cache.
+    partitionsOfStorage.putIfAbsent(appId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).putIfAbsent(shuffleId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).get(shuffleId).put(partitionId, storage);
     return storage;
   }
 
   @Override
   public Storage selectStorage(ShuffleDataReadEvent event) {
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
 
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    LocalStorage storage = null;
+    try {
+      storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+    } catch (NullPointerException npe) {

Review Comment:
   I think I will optimize it by union key, which will keep thread safe.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049586857


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -69,8 +69,9 @@ public class LocalStorageManager extends SingleStorageManager {
   private final List<LocalStorage> localStorages;
   private final List<String> storageBasePaths;
   private final LocalStorageChecker checker;
-  private List<LocalStorage> unCorruptedStorages = Lists.newArrayList();
-  private final Set<String> corruptedStorages = Sets.newConcurrentHashSet();
+
+  // AppId -> ShuffleId -> PartitionId -> LocalStorage
+  private final Map<String, Map<Integer, Map<Integer, LocalStorage>>> partitionsOfStorage = Maps.newConcurrentMap();

Review Comment:
   The mechanism of uploading local files has been removed in current codebase. And this PR is just to avoid selecting storage which has reached high watermark.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1051399753


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -171,11 +171,18 @@ public Storage selectStorage(ShuffleDataFlushEvent event) {
             partitionId
         )
     );
-    event.setUnderStorage(storage);
 
     // store it to cache.
-    partitionsOfStorage.put(UnionKey.toKey(appId, shuffleId, partitionId), storage);
-    return storage;
+    final LocalStorage selectedStorage = storage;
+    return partitionsOfStorage.compute(

Review Comment:
   Use the concurrent hashmap compute method to ensure the thread safe
   
   cc @advancedxy , please help check this part.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050677967


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -246,38 +300,70 @@ public void checkAndClearLeakedShuffleData(Collection<String> appIds) {
     }
   }
 
-  void repair() {
-    boolean hasNewCorruptedStorage = false;
-    for (LocalStorage storage : localStorages) {
-      if (storage.isCorrupted() && !corruptedStorages.contains(storage.getBasePath())) {
-        hasNewCorruptedStorage = true;
-        corruptedStorages.add(storage.getBasePath());
-      }
+  public List<LocalStorage> getStorages() {
+    return localStorages;
+  }
+
+  static class PartitionUnionKey {

Review Comment:
   Using the string as key will be more memory efficient, the only problem isn't very interpretable when removing element by its prefix.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050679528


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -217,6 +244,33 @@ public void removeResources(PurgeEvent event) {
     deleteHandler.delete(deletePaths.toArray(new String[deletePaths.size()]), appId, user);
   }
 
+  private void cleanupStorageSelectionCache(PurgeEvent event) {
+    Function<PartitionUnionKey, Boolean> deleteConditionFunc = null;
+    if (event instanceof AppPurgeEvent) {
+      deleteConditionFunc = partitionUnionKey -> partitionUnionKey.carryWithAppId(event.getAppId());
+    } else if (event instanceof ShufflePurgeEvent) {
+      deleteConditionFunc =
+          partitionUnionKey -> partitionUnionKey.carryWithAppIdAndShuffleIds(
+              event.getAppId(),
+              new HashSet<>(event.getShuffleIds())

Review Comment:
   Make sense. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
xianjingfeng commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049297134


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");

Review Comment:
   We have remove this exception in #281, why put it back?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049326324


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+      }
+      return storage;
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    LocalStorage storage =
+        localStorages
+            .stream()
+            .filter(x -> x.canWrite() && !x.isCorrupted())
+            .collect(Collectors.toList())
+            .get(
+                ShuffleStorageUtils.getStorageIndex(
+                    localStorages.size(),

Review Comment:
   Yes. I have fixed in the latest commit



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049588872


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,60 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+        } else {
+          // clear the last selection due to non written.
+          partitionsOfStorage.get(appId).get(shuffleId).remove(partitionId);
+        }
+      } else {
+        return storage;
+      }
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    List<LocalStorage> candidates = localStorages
+        .stream()
+        .filter(x -> x.canWrite() && !x.isCorrupted())
+        .collect(Collectors.toList());
+    LocalStorage storage = candidates.get(
+        ShuffleStorageUtils.getStorageIndex(
+            candidates.size(),
+            appId,
+            shuffleId,
+            partitionId
+        )
+    );
     event.setUnderStorage(storage);
+
+    // store it to cache.
+    partitionsOfStorage.computeIfAbsent(appId, key -> Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).computeIfAbsent(shuffleId, key -> Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).get(shuffleId).put(partitionId, storage);
     return storage;
   }
 
   @Override
   public Storage selectStorage(ShuffleDataReadEvent event) {
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
 
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    LocalStorage storage = null;
+    try {
+      storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);

Review Comment:
   If the writeEvent has been written, it will reuse the storage instead of selecting again.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050340953


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -69,8 +69,9 @@ public class LocalStorageManager extends SingleStorageManager {
   private final List<LocalStorage> localStorages;
   private final List<String> storageBasePaths;
   private final LocalStorageChecker checker;
-  private List<LocalStorage> unCorruptedStorages = Lists.newArrayList();
-  private final Set<String> corruptedStorages = Sets.newConcurrentHashSet();
+
+  // AppId -> ShuffleId -> PartitionId -> LocalStorage
+  private final Map<String, Map<Integer, Map<Integer, LocalStorage>>> partitionsOfStorage = Maps.newConcurrentMap();

Review Comment:
   Here https://github.com/apache/incubator-uniffle/pull/51 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050362964


##########
integration-test/common/src/test/java/org/apache/uniffle/test/DiskErrorToleranceTest.java:
##########
@@ -87,8 +86,7 @@ public void createClient() {
   public void closeClient() {
     shuffleServerClient.close();
   }
-
-  @Test
+  
   public void diskErrorTest() throws Exception {

Review Comment:
   why the `@Test` is removed?



##########
server/src/main/java/org/apache/uniffle/server/ShuffleDataReadEvent.java:
##########
@@ -17,16 +17,20 @@
 
 package org.apache.uniffle.server;
 
+import org.apache.uniffle.common.PartitionRange;
+
 public class ShuffleDataReadEvent {
 
   private String appId;
   private int shuffleId;
-  private int startPartition;
+  private int partitionId;
+  private PartitionRange partitionRange;
 
-  public ShuffleDataReadEvent(String appId, int shuffleId, int startPartition) {
+  public ShuffleDataReadEvent(String appId, int shuffleId, int partitionId, int[] range) {

Review Comment:
   why is this change?



##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -69,8 +71,8 @@ public class LocalStorageManager extends SingleStorageManager {
   private final List<LocalStorage> localStorages;
   private final List<String> storageBasePaths;
   private final LocalStorageChecker checker;
-  private List<LocalStorage> unCorruptedStorages = Lists.newArrayList();
-  private final Set<String> corruptedStorages = Sets.newConcurrentHashSet();
+
+  private final Map<PartitionUnionKey, LocalStorage> partitionsOfStorage;

Review Comment:
   > From our dashboard, there are only a few thousand partitions running at the same time
   
   For large spark apps, it's common to have ~10K shuffle partitions and let's just one app. 
   
   However maybe we have not reached this kind of scale.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050680802


##########
server/src/main/java/org/apache/uniffle/server/ShuffleDataReadEvent.java:
##########
@@ -17,16 +17,20 @@
 
 package org.apache.uniffle.server;
 
+import org.apache.uniffle.common.PartitionRange;
+
 public class ShuffleDataReadEvent {
 
   private String appId;
   private int shuffleId;
-  private int startPartition;
+  private int partitionId;
+  private PartitionRange partitionRange;
 
-  public ShuffleDataReadEvent(String appId, int shuffleId, int startPartition) {
+  public ShuffleDataReadEvent(String appId, int shuffleId, int partitionId, int[] range) {

Review Comment:
   > Then an int of startPartition should be sufficient, just like how ShuffleDataFlushEvent did.
   
   Yes. I will



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1053957649


##########
integration-test/common/src/test/java/org/apache/uniffle/test/DiskErrorToleranceTest.java:
##########
@@ -89,6 +90,7 @@ public void closeClient() {
   }
 
   @Test

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054055064


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,33 +143,53 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    LocalStorage storage = partitionsOfStorage.get(UnionKey.toKey(appId, shuffleId, partitionId));
+    if (storage != null) {
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050396493


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -69,8 +71,8 @@ public class LocalStorageManager extends SingleStorageManager {
   private final List<LocalStorage> localStorages;
   private final List<String> storageBasePaths;
   private final LocalStorageChecker checker;
-  private List<LocalStorage> unCorruptedStorages = Lists.newArrayList();
-  private final Set<String> corruptedStorages = Sets.newConcurrentHashSet();
+
+  private final Map<PartitionUnionKey, LocalStorage> partitionsOfStorage;

Review Comment:
   From our dashboard, there are only a few thousand partitions running at the same time



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054303754


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,33 +143,53 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    LocalStorage storage = partitionsOfStorage.get(UnionKey.toKey(appId, shuffleId, partitionId));
+    if (storage != null) {
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+        }
+      } else {
+        return storage;
+      }
     }
-    event.setUnderStorage(storage);
-    return storage;
+
+    List<LocalStorage> candidates = localStorages
+        .stream()
+        .filter(x -> x.canWrite() && !x.isCorrupted())
+        .collect(Collectors.toList());
+    final LocalStorage selectedStorage = candidates.get(
+        ShuffleStorageUtils.getStorageIndex(
+            candidates.size(),
+            appId,
+            shuffleId,
+            partitionId
+        )
+    );
+    return partitionsOfStorage.compute(
+        UnionKey.toKey(appId, shuffleId, partitionId),
+        (key, localStorage) -> {
+          // If this is the first time to select storage or existing storage is corrupted,
+          // we should refresh the cache.
+          if (localStorage == null || localStorage.isCorrupted()) {

Review Comment:
   yeah, I know. What bothered me is that:
   - Previously L154-L156 throws an exception, I cannot image in which case, the localStorage.isCorrupted holds true
   - In latest code, this makes sense.
   
   Did i miss something?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054312239


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,33 +143,53 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    LocalStorage storage = partitionsOfStorage.get(UnionKey.toKey(appId, shuffleId, partitionId));
+    if (storage != null) {
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+        }
+      } else {
+        return storage;
+      }
     }
-    event.setUnderStorage(storage);
-    return storage;
+
+    List<LocalStorage> candidates = localStorages
+        .stream()
+        .filter(x -> x.canWrite() && !x.isCorrupted())
+        .collect(Collectors.toList());
+    final LocalStorage selectedStorage = candidates.get(
+        ShuffleStorageUtils.getStorageIndex(
+            candidates.size(),
+            appId,
+            shuffleId,
+            partitionId
+        )
+    );
+    return partitionsOfStorage.compute(
+        UnionKey.toKey(appId, shuffleId, partitionId),
+        (key, localStorage) -> {
+          // If this is the first time to select storage or existing storage is corrupted,
+          // we should refresh the cache.
+          if (localStorage == null || localStorage.isCorrupted()) {

Review Comment:
   > Previously L154-L156 throws an exception, I cannot image in which case, the localStorage.isCorrupted holds true
   
   In the previous version commit, when `localStorage.isCorrupted() == true` and `storage.containsWriteHandler(appId, shuffleId, partitionId) == false`, the code will enter the part you mentioned.
   
   Do I catch you thought? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054316181


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,33 +143,53 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    LocalStorage storage = partitionsOfStorage.get(UnionKey.toKey(appId, shuffleId, partitionId));
+    if (storage != null) {
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+        }
+      } else {
+        return storage;
+      }
     }
-    event.setUnderStorage(storage);
-    return storage;
+
+    List<LocalStorage> candidates = localStorages
+        .stream()
+        .filter(x -> x.canWrite() && !x.isCorrupted())
+        .collect(Collectors.toList());
+    final LocalStorage selectedStorage = candidates.get(
+        ShuffleStorageUtils.getStorageIndex(
+            candidates.size(),
+            appId,
+            shuffleId,
+            partitionId
+        )
+    );
+    return partitionsOfStorage.compute(
+        UnionKey.toKey(appId, shuffleId, partitionId),
+        (key, localStorage) -> {
+          // If this is the first time to select storage or existing storage is corrupted,
+          // we should refresh the cache.
+          if (localStorage == null || localStorage.isCorrupted()) {

Review Comment:
   Ah, I see. Thanks for the clarifying.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050362720


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,60 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+        } else {
+          // clear the last selection due to non written.
+          partitionsOfStorage.get(appId).get(shuffleId).remove(partitionId);
+        }
+      } else {
+        return storage;
+      }
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    List<LocalStorage> candidates = localStorages
+        .stream()
+        .filter(x -> x.canWrite() && !x.isCorrupted())
+        .collect(Collectors.toList());
+    LocalStorage storage = candidates.get(
+        ShuffleStorageUtils.getStorageIndex(
+            candidates.size(),
+            appId,
+            shuffleId,
+            partitionId
+        )
+    );
     event.setUnderStorage(storage);
+
+    // store it to cache.
+    partitionsOfStorage.computeIfAbsent(appId, key -> Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).computeIfAbsent(shuffleId, key -> Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).get(shuffleId).put(partitionId, storage);
     return storage;
   }
 
   @Override
   public Storage selectStorage(ShuffleDataReadEvent event) {
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
 
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    LocalStorage storage = null;
+    try {
+      storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);

Review Comment:
   My fault, I see related code now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050397084


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -69,8 +71,8 @@ public class LocalStorageManager extends SingleStorageManager {
   private final List<LocalStorage> localStorages;
   private final List<String> storageBasePaths;
   private final LocalStorageChecker checker;
-  private List<LocalStorage> unCorruptedStorages = Lists.newArrayList();
-  private final Set<String> corruptedStorages = Sets.newConcurrentHashSet();
+
+  private final Map<PartitionUnionKey, LocalStorage> partitionsOfStorage;

Review Comment:
   And if we use the nested concurrent hashmap, I'm worry about the problem of thread safe.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050705734


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");

Review Comment:
   Sorry, I missed this thread. 
   
   > The whole partition could not be trusted. ShuffleReadClient should switch to another replica as soon as possible.
   Maybe there's some logic I didn't follow?
   
   Let me explain more detail about this. Firstly it need to be corrected that the disk corruption will influence the writing process in current case. Of course, it will cause the later reading if having multiple replica.
   
   So if having single replica and using the `MEMORY_LOCALFILE` or `LOCALFILE` type, once disk is corrupted, which means the data lost, we should make relevant job fast fail. But in current codebase, it ignore this failure and retry and then fail. Of course, this will not cause big problems.
   
   But when using `MEMORY_LOCALFILE_HDFS` and single replica, this will cause rest events flushing to fallback storage, like from localdisk to HDFS. Actually, this is unnecessary, as partial data lost.
   
   Of course, this could be useful for multiple replica. But I think we could support this mechanism after dynamic selection, which will be better.
   
   > ShuffleReadClient should switch to another replica as soon as possible.
   
   
   Yes. I think this is reasonable. 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050353924


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +142,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(PartitionUnionKey.of(appId, shuffleId, partitionId));
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+        }
+      } else {
+        return storage;
+      }
+    } catch (NullPointerException npe) {

Review Comment:
   My fault. I will check whether storage is null. I forgot to remove this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050714993


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -217,6 +244,33 @@ public void removeResources(PurgeEvent event) {
     deleteHandler.delete(deletePaths.toArray(new String[deletePaths.size()]), appId, user);
   }
 
+  private void cleanupStorageSelectionCache(PurgeEvent event) {
+    Function<PartitionUnionKey, Boolean> deleteConditionFunc = null;
+    if (event instanceof AppPurgeEvent) {
+      deleteConditionFunc = partitionUnionKey -> partitionUnionKey.carryWithAppId(event.getAppId());
+    } else if (event instanceof ShufflePurgeEvent) {
+      deleteConditionFunc =
+          partitionUnionKey -> partitionUnionKey.carryWithAppIdAndShuffleIds(
+              event.getAppId(),
+              new HashSet<>(event.getShuffleIds())

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050711508


##########
integration-test/common/src/test/java/org/apache/uniffle/test/DiskErrorToleranceTest.java:
##########
@@ -87,8 +86,7 @@ public void createClient() {
   public void closeClient() {
     shuffleServerClient.close();
   }
-
-  @Test
+  
   public void diskErrorTest() throws Exception {

Review Comment:
   If we choose to bear partial data lost when disk is corrupted, this test could be enabled in later commits.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#issuecomment-1356643251

   Latest commits changelog
   1. Fix potential bug of thread race condition when selecting storage concurrently and add tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#issuecomment-1356644421

   Fix the potential bug of thread race condition. @advancedxy 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
xianjingfeng commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049298338


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+      }
+      return storage;
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    LocalStorage storage =
+        localStorages
+            .stream()
+            .filter(x -> x.canWrite() && !x.isCorrupted())
+            .collect(Collectors.toList())
+            .get(
+                ShuffleStorageUtils.getStorageIndex(
+                    localStorages.size(),
+                    appId,
+                    shuffleId,
+                    partitionId
+                )
+            );
     event.setUnderStorage(storage);
+
+    // store it to cache.
+    partitionsOfStorage.putIfAbsent(appId, Maps.newConcurrentMap());

Review Comment:
    It's very convenient to use `putIfAbsent`, but `Maps.newConcurrentMap()` will be invoked every times? right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054312239


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,33 +143,53 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    LocalStorage storage = partitionsOfStorage.get(UnionKey.toKey(appId, shuffleId, partitionId));
+    if (storage != null) {
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+        }
+      } else {
+        return storage;
+      }
     }
-    event.setUnderStorage(storage);
-    return storage;
+
+    List<LocalStorage> candidates = localStorages
+        .stream()
+        .filter(x -> x.canWrite() && !x.isCorrupted())
+        .collect(Collectors.toList());
+    final LocalStorage selectedStorage = candidates.get(
+        ShuffleStorageUtils.getStorageIndex(
+            candidates.size(),
+            appId,
+            shuffleId,
+            partitionId
+        )
+    );
+    return partitionsOfStorage.compute(
+        UnionKey.toKey(appId, shuffleId, partitionId),
+        (key, localStorage) -> {
+          // If this is the first time to select storage or existing storage is corrupted,
+          // we should refresh the cache.
+          if (localStorage == null || localStorage.isCorrupted()) {

Review Comment:
   > Previously L154-L156 throws an exception, I cannot image in which case, the localStorage.isCorrupted holds true
   
   In the previous version commit, when `localStorage.isCorrupted() == true` and `storage.containsWriteHandler(appId, shuffleId, partitionId) == false`, the code will enter the part you mentioned.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054030594


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,33 +143,53 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    LocalStorage storage = partitionsOfStorage.get(UnionKey.toKey(appId, shuffleId, partitionId));
+    if (storage != null) {
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+        }
+      } else {
+        return storage;
+      }
     }
-    event.setUnderStorage(storage);
-    return storage;
+
+    List<LocalStorage> candidates = localStorages
+        .stream()
+        .filter(x -> x.canWrite() && !x.isCorrupted())
+        .collect(Collectors.toList());
+    final LocalStorage selectedStorage = candidates.get(
+        ShuffleStorageUtils.getStorageIndex(
+            candidates.size(),
+            appId,
+            shuffleId,
+            partitionId
+        )
+    );
+    return partitionsOfStorage.compute(
+        UnionKey.toKey(appId, shuffleId, partitionId),
+        (key, localStorage) -> {
+          // If this is the first time to select storage or existing storage is corrupted,
+          // we should refresh the cache.
+          if (localStorage == null || localStorage.isCorrupted()) {

Review Comment:
   For an event, if the storage is selected but event don't write any data to this storage (maybe the event is in pending queue), that means we could replace it to new storage, which won't cause data lost.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049328673


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");

Review Comment:
   I dont catch the point of removing this exception. Once storage is corrupted, there is no need to use next storage, because uniffle dont support dynamic storage switch, especially for a corrupted storage



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
xianjingfeng commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049297134


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");

Review Comment:
   We had remove this exception in #281, why put it back?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050683064


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+      }
+      return storage;
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    LocalStorage storage =
+        localStorages
+            .stream()
+            .filter(x -> x.canWrite() && !x.isCorrupted())
+            .collect(Collectors.toList())
+            .get(
+                ShuffleStorageUtils.getStorageIndex(
+                    localStorages.size(),
+                    appId,
+                    shuffleId,
+                    partitionId
+                )
+            );
     event.setUnderStorage(storage);
+
+    // store it to cache.
+    partitionsOfStorage.putIfAbsent(appId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).putIfAbsent(shuffleId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).get(shuffleId).put(partitionId, storage);
     return storage;
   }
 
   @Override
   public Storage selectStorage(ShuffleDataReadEvent event) {
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
 
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    LocalStorage storage = null;
+    try {
+      storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+    } catch (NullPointerException npe) {

Review Comment:
   My fault. I forgot to change in this place.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050679022


##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerGrpcService.java:
##########
@@ -511,9 +513,15 @@ public void getLocalShuffleData(GetLocalShuffleDataRequest request,
     String requestInfo = "appId[" + appId + "], shuffleId[" + shuffleId + "], partitionId["
         + partitionId + "]" + "offset[" + offset + "]" + "length[" + length + "]";
 
-    shuffleServer.getStorageManager()
-        .selectStorage(new ShuffleDataReadEvent(appId, shuffleId, partitionId))
-        .updateReadMetrics(new StorageReadMetrics(appId, shuffleId));
+    int[] range = ShuffleStorageUtils.getPartitionRange(partitionId, partitionNumPerRange, partitionNum);

Review Comment:
   Nice catch. You could create a issue to track this.



##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerGrpcService.java:
##########
@@ -511,9 +513,15 @@ public void getLocalShuffleData(GetLocalShuffleDataRequest request,
     String requestInfo = "appId[" + appId + "], shuffleId[" + shuffleId + "], partitionId["
         + partitionId + "]" + "offset[" + offset + "]" + "length[" + length + "]";
 
-    shuffleServer.getStorageManager()
-        .selectStorage(new ShuffleDataReadEvent(appId, shuffleId, partitionId))
-        .updateReadMetrics(new StorageReadMetrics(appId, shuffleId));
+    int[] range = ShuffleStorageUtils.getPartitionRange(partitionId, partitionNumPerRange, partitionNum);

Review Comment:
   Nice catch. You could create an issue to track this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050731392


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -246,38 +300,70 @@ public void checkAndClearLeakedShuffleData(Collection<String> appIds) {
     }
   }
 
-  void repair() {
-    boolean hasNewCorruptedStorage = false;
-    for (LocalStorage storage : localStorages) {
-      if (storage.isCorrupted() && !corruptedStorages.contains(storage.getBasePath())) {
-        hasNewCorruptedStorage = true;
-        corruptedStorages.add(storage.getBasePath());
-      }
+  public List<LocalStorage> getStorages() {
+    return localStorages;
+  }
+
+  static class PartitionUnionKey {

Review Comment:
   I introduce the `UnionKey` class to wrap the multiple elements as key, the key type is string type.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050492915


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");

Review Comment:
   For this case: 
   In fact, I'm prefer to keep the old logic in this pr. 
   If we want to optimize with dynamic switching. we can introduce this change in that pr.
   
   However, I think we should reconsider the corrupted cases. Once one replica of local storage is corrupted. The
   whole partition could not be trusted. ShuffleReadClient should switch to another replica as soon as possible. 
   Maybe there's some logic I didn't follow?
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1053930774


##########
server/src/main/java/org/apache/uniffle/server/ShuffleDataReadEvent.java:
##########
@@ -21,12 +21,14 @@ public class ShuffleDataReadEvent {
 
   private String appId;
   private int shuffleId;
+  private int partitionId;
   private int startPartition;
 
-  public ShuffleDataReadEvent(String appId, int shuffleId, int startPartition) {
+  public ShuffleDataReadEvent(String appId, int shuffleId, int partitionId, int[] range) {

Review Comment:
   I don't think `int[] range` is necessary.  Just a `int startPartition` should be sufficient. And we can avoid the range `int[]` creation.



##########
integration-test/common/src/test/java/org/apache/uniffle/test/DiskErrorToleranceTest.java:
##########
@@ -89,6 +90,7 @@ public void closeClient() {
   }
 
   @Test

Review Comment:
   The `@Test` annotation should be removed?



##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -217,6 +244,34 @@ public void removeResources(PurgeEvent event) {
     deleteHandler.delete(deletePaths.toArray(new String[deletePaths.size()]), appId, user);
   }
 
+  private void cleanupStorageSelectionCache(PurgeEvent event) {
+    Function<String, Boolean> deleteConditionFunc = null;
+    if (event instanceof AppPurgeEvent) {
+      deleteConditionFunc = partitionUnionKey -> UnionKey.startsWith(partitionUnionKey, event.getAppId());
+    } else if (event instanceof ShufflePurgeEvent) {
+      deleteConditionFunc =
+          partitionUnionKey -> UnionKey.startsWith(
+              partitionUnionKey,
+              event.getAppId(),
+              event.getShuffleIds()
+          );
+    }
+    deleteElement(
+        partitionsOfStorage,
+        deleteConditionFunc
+    );
+  }
+
+  private <K, V> void deleteElement(Map<K, V> map, Function<K, Boolean> deleteConditionFunc) {

Review Comment:
   I'm a little concerned about the performance. Could you add a logging about how much time cost to delete element?



##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerGrpcService.java:
##########
@@ -511,9 +513,15 @@ public void getLocalShuffleData(GetLocalShuffleDataRequest request,
     String requestInfo = "appId[" + appId + "], shuffleId[" + shuffleId + "], partitionId["
         + partitionId + "]" + "offset[" + offset + "]" + "length[" + length + "]";
 
-    shuffleServer.getStorageManager()
-        .selectStorage(new ShuffleDataReadEvent(appId, shuffleId, partitionId))
-        .updateReadMetrics(new StorageReadMetrics(appId, shuffleId));
+    int[] range = ShuffleStorageUtils.getPartitionRange(partitionId, partitionNumPerRange, partitionNum);

Review Comment:
   Another aspect why `int[]` to `ShuffleDataReadEvent` is unnecessary: we can calculate start partition directly, which could avoid the `int[]` creation. But let's do that in another pr.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1053965754


##########
server/src/main/java/org/apache/uniffle/server/ShuffleDataReadEvent.java:
##########
@@ -21,12 +21,14 @@ public class ShuffleDataReadEvent {
 
   private String appId;
   private int shuffleId;
+  private int partitionId;
   private int startPartition;
 
-  public ShuffleDataReadEvent(String appId, int shuffleId, int startPartition) {
+  public ShuffleDataReadEvent(String appId, int shuffleId, int partitionId, int[] range) {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1053961180


##########
server/src/main/java/org/apache/uniffle/server/ShuffleDataReadEvent.java:
##########
@@ -21,12 +21,14 @@ public class ShuffleDataReadEvent {
 
   private String appId;
   private int shuffleId;
+  private int partitionId;
   private int startPartition;
 
-  public ShuffleDataReadEvent(String appId, int shuffleId, int startPartition) {
+  public ShuffleDataReadEvent(String appId, int shuffleId, int partitionId, int[] range) {

Review Comment:
   emmm, Maybe... If you want to preserve this code, then some comments/todos should be added, which may require the same amount of code modifications to covert `int[] range` to `int startPartition`
   
   Just thought it was a small change and shouldn't take you too much time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
jerqi commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054090577


##########
common/src/main/java/org/apache/uniffle/common/UnionKey.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.common;
+
+import org.apache.commons.lang3.StringUtils;
+
+/**
+ * This class is to wrap multi elements to be as union key.
+ */
+public class UnionKey {
+  private static final String SPLIT_KEY = "_";
+
+  public static String toKey(Object... factors) {

Review Comment:
   `buildKey` is ok.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054046971


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,33 +143,53 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    LocalStorage storage = partitionsOfStorage.get(UnionKey.toKey(appId, shuffleId, partitionId));
+    if (storage != null) {
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");

Review Comment:
   Emm... If switching storage when corruption, it's a little meaningful for multiple replica, but has no mean for single replica.
   
   After rethinking, I think I could preserve original logic, which looks no conflict with this feature.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049326025


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+      }
+      return storage;
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    LocalStorage storage =
+        localStorages
+            .stream()
+            .filter(x -> x.canWrite() && !x.isCorrupted())
+            .collect(Collectors.toList())
+            .get(
+                ShuffleStorageUtils.getStorageIndex(
+                    localStorages.size(),
+                    appId,
+                    shuffleId,
+                    partitionId
+                )
+            );
     event.setUnderStorage(storage);
+
+    // store it to cache.
+    partitionsOfStorage.putIfAbsent(appId, Maps.newConcurrentMap());

Review Comment:
   Nice catch. Let's use `computeIfAbsent`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1049510529


##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerGrpcService.java:
##########
@@ -571,10 +574,11 @@ public void getLocalShuffleIndex(GetLocalShuffleIndexRequest request,
     String requestInfo = "appId[" + appId + "], shuffleId[" + shuffleId + "], partitionId["
         + partitionId + "]";
 
-    shuffleServer.getStorageManager()
-        .selectStorage(new ShuffleDataReadEvent(appId, shuffleId, partitionId))
-        .updateReadMetrics(new StorageReadMetrics(appId, shuffleId));
-
+    Storage storage = shuffleServer.getStorageManager()
+        .selectStorage(new ShuffleDataReadEvent(appId, shuffleId, partitionId));
+    if (storage != null) {
+      storage.updateReadMetrics(new StorageReadMetrics(appId, shuffleId));

Review Comment:
   Not related to this PR. But it occurred to me that it's not necessary to create `new StorageReadMetrics` every local shuffle read. It may put a bit more gc pressure here.
   
   Could you create a new issue to address this problem?



##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,60 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+        } else {
+          // clear the last selection due to non written.
+          partitionsOfStorage.get(appId).get(shuffleId).remove(partitionId);
+        }
+      } else {
+        return storage;
+      }
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    List<LocalStorage> candidates = localStorages
+        .stream()
+        .filter(x -> x.canWrite() && !x.isCorrupted())
+        .collect(Collectors.toList());
+    LocalStorage storage = candidates.get(
+        ShuffleStorageUtils.getStorageIndex(
+            candidates.size(),
+            appId,
+            shuffleId,
+            partitionId
+        )
+    );
     event.setUnderStorage(storage);
+
+    // store it to cache.
+    partitionsOfStorage.computeIfAbsent(appId, key -> Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).computeIfAbsent(shuffleId, key -> Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).get(shuffleId).put(partitionId, storage);
     return storage;
   }
 
   @Override
   public Storage selectStorage(ShuffleDataReadEvent event) {
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
 
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    LocalStorage storage = null;
+    try {
+      storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);

Review Comment:
   I believe this logic is not right.
   
   Think about this case:
   ```
   # App:1, shuffle: 2, partition: 0. 
   WriteEvent(1,2,0) -> disk1
   WriteEvent(1,2,0) -> disk2(moved to disk2 as disk1 reaches its high watermark)
   
   --
   ReadEvent(1,2,0) -> disk2. The data wrote to disk1 is lost as it's not selected.
   ```



##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -69,8 +69,9 @@ public class LocalStorageManager extends SingleStorageManager {
   private final List<LocalStorage> localStorages;
   private final List<String> storageBasePaths;
   private final LocalStorageChecker checker;
-  private List<LocalStorage> unCorruptedStorages = Lists.newArrayList();
-  private final Set<String> corruptedStorages = Sets.newConcurrentHashSet();
+
+  // AppId -> ShuffleId -> PartitionId -> LocalStorage
+  private final Map<String, Map<Integer, Map<Integer, LocalStorage>>> partitionsOfStorage = Maps.newConcurrentMap();

Review Comment:
   I'm not sure and always concerned how much memory pressure it would put on the server. 
   I saw another place have similar structure, such as the following code in ShuffleFlushManager
   ```
     // appId -> shuffleId -> partitionId -> handlers
     private Map<String, Map<Integer, RangeMap<Integer, ShuffleWriteHandler>>> handlers = Maps.newConcurrentMap();
   ```
   PartitionIds could be millions for large apps. Are there any other ways to reconstruct this info?



##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -69,8 +69,9 @@ public class LocalStorageManager extends SingleStorageManager {
   private final List<LocalStorage> localStorages;
   private final List<String> storageBasePaths;
   private final LocalStorageChecker checker;
-  private List<LocalStorage> unCorruptedStorages = Lists.newArrayList();
-  private final Set<String> corruptedStorages = Sets.newConcurrentHashSet();
+
+  // AppId -> ShuffleId -> PartitionId -> LocalStorage
+  private final Map<String, Map<Integer, Map<Integer, LocalStorage>>> partitionsOfStorage = Maps.newConcurrentMap();

Review Comment:
   I did a quick look at how Spark handles local map file distribution..
   I believe  the current strategy is similar with spark's. If high watermark is reached, maybe we should start uploading local files before reaching high watermark?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050339548


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");

Review Comment:
   Gentle ping @jerqi WDYT. Besides, after this PR, I think I will introduce the feature of dynamic switching the localdisk for one single, which will solve your multiple replica problem.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050361700


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -69,8 +69,9 @@ public class LocalStorageManager extends SingleStorageManager {
   private final List<LocalStorage> localStorages;
   private final List<String> storageBasePaths;
   private final LocalStorageChecker checker;
-  private List<LocalStorage> unCorruptedStorages = Lists.newArrayList();
-  private final Set<String> corruptedStorages = Sets.newConcurrentHashSet();
+
+  // AppId -> ShuffleId -> PartitionId -> LocalStorage
+  private final Map<String, Map<Integer, Map<Integer, LocalStorage>>> partitionsOfStorage = Maps.newConcurrentMap();

Review Comment:
   My fault, I see related code now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050513456


##########
server/src/main/java/org/apache/uniffle/server/ShuffleDataReadEvent.java:
##########
@@ -17,16 +17,20 @@
 
 package org.apache.uniffle.server;
 
+import org.apache.uniffle.common.PartitionRange;
+
 public class ShuffleDataReadEvent {
 
   private String appId;
   private int shuffleId;
-  private int startPartition;
+  private int partitionId;
+  private PartitionRange partitionRange;
 
-  public ShuffleDataReadEvent(String appId, int shuffleId, int startPartition) {
+  public ShuffleDataReadEvent(String appId, int shuffleId, int partitionId, int[] range) {

Review Comment:
   The `partitionsOfStorage`'s key is constructed by startPartition. So if the DataReadEvent to get the corresponding storage, we must need to get its startPartition.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050429172


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -69,8 +71,8 @@ public class LocalStorageManager extends SingleStorageManager {
   private final List<LocalStorage> localStorages;
   private final List<String> storageBasePaths;
   private final LocalStorageChecker checker;
-  private List<LocalStorage> unCorruptedStorages = Lists.newArrayList();
-  private final Set<String> corruptedStorages = Sets.newConcurrentHashSet();
+
+  private final Map<PartitionUnionKey, LocalStorage> partitionsOfStorage;

Review Comment:
   > From our dashboard, there are only a few thousand partitions running at the same time
   
   For large spark apps, it's common to have ~10K shuffle partitions and it's just one app. 
   
   However maybe we have not reached this kind of scale.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1050668564


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -217,6 +244,33 @@ public void removeResources(PurgeEvent event) {
     deleteHandler.delete(deletePaths.toArray(new String[deletePaths.size()]), appId, user);
   }
 
+  private void cleanupStorageSelectionCache(PurgeEvent event) {
+    Function<PartitionUnionKey, Boolean> deleteConditionFunc = null;
+    if (event instanceof AppPurgeEvent) {
+      deleteConditionFunc = partitionUnionKey -> partitionUnionKey.carryWithAppId(event.getAppId());
+    } else if (event instanceof ShufflePurgeEvent) {
+      deleteConditionFunc =
+          partitionUnionKey -> partitionUnionKey.carryWithAppIdAndShuffleIds(
+              event.getAppId(),
+              new HashSet<>(event.getShuffleIds())

Review Comment:
   For this case, maybe it's not necessary to convert a list into a set.
   
   Since the shuffleIds in one app is limited, most probably is under 10. A for each search should be as performant as hash lookup.



##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerGrpcService.java:
##########
@@ -511,9 +513,15 @@ public void getLocalShuffleData(GetLocalShuffleDataRequest request,
     String requestInfo = "appId[" + appId + "], shuffleId[" + shuffleId + "], partitionId["
         + partitionId + "]" + "offset[" + offset + "]" + "length[" + length + "]";
 
-    shuffleServer.getStorageManager()
-        .selectStorage(new ShuffleDataReadEvent(appId, shuffleId, partitionId))
-        .updateReadMetrics(new StorageReadMetrics(appId, shuffleId));
+    int[] range = ShuffleStorageUtils.getPartitionRange(partitionId, partitionNumPerRange, partitionNum);

Review Comment:
   `ShuffleStorageUtils.getPartitionRange` is not performant... 
   
   It could be calculated directly in O(1).. However current impl is about O(n) where n is the number of partition ranges. when partition range size =1, n is corresponding to partition numbers.



##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -246,38 +300,70 @@ public void checkAndClearLeakedShuffleData(Collection<String> appIds) {
     }
   }
 
-  void repair() {
-    boolean hasNewCorruptedStorage = false;
-    for (LocalStorage storage : localStorages) {
-      if (storage.isCorrupted() && !corruptedStorages.contains(storage.getBasePath())) {
-        hasNewCorruptedStorage = true;
-        corruptedStorages.add(storage.getBasePath());
-      }
+  public List<LocalStorage> getStorages() {
+    return localStorages;
+  }
+
+  static class PartitionUnionKey {

Review Comment:
   let's move this into a new file?
   
   I'm not sure which is memory efficient: PartitionUnionKey class or simply a String: "appId_shuffleId_partitionId"



##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerGrpcService.java:
##########
@@ -511,9 +513,15 @@ public void getLocalShuffleData(GetLocalShuffleDataRequest request,
     String requestInfo = "appId[" + appId + "], shuffleId[" + shuffleId + "], partitionId["
         + partitionId + "]" + "offset[" + offset + "]" + "length[" + length + "]";
 
-    shuffleServer.getStorageManager()
-        .selectStorage(new ShuffleDataReadEvent(appId, shuffleId, partitionId))
-        .updateReadMetrics(new StorageReadMetrics(appId, shuffleId));
+    int[] range = ShuffleStorageUtils.getPartitionRange(partitionId, partitionNumPerRange, partitionNum);

Review Comment:
   This should be addressed in another pr.



##########
server/src/main/java/org/apache/uniffle/server/ShuffleDataReadEvent.java:
##########
@@ -17,16 +17,20 @@
 
 package org.apache.uniffle.server;
 
+import org.apache.uniffle.common.PartitionRange;
+
 public class ShuffleDataReadEvent {
 
   private String appId;
   private int shuffleId;
-  private int startPartition;
+  private int partitionId;
+  private PartitionRange partitionRange;
 
-  public ShuffleDataReadEvent(String appId, int shuffleId, int startPartition) {
+  public ShuffleDataReadEvent(String appId, int shuffleId, int partitionId, int[] range) {

Review Comment:
   Then an int of `startPartition` should be sufficient, just like how `ShuffleDataFlushEvent` did.
   
   A class in JVM is quite expressive compared to native languages, the partition range class would occupied ~40 bytes per instance. And it would put more pressure to GC.



##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,32 +140,55 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    try {
+      LocalStorage storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+      if (storage.isCorrupted()) {
+        throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+      }
+      return storage;
+    } catch (NullPointerException npe) {
+      // Ignore
     }
+
+    // Firstly getting the storage based on its (appId, shuffleId, partitionId) hash value
+    LocalStorage storage =
+        localStorages
+            .stream()
+            .filter(x -> x.canWrite() && !x.isCorrupted())
+            .collect(Collectors.toList())
+            .get(
+                ShuffleStorageUtils.getStorageIndex(
+                    localStorages.size(),
+                    appId,
+                    shuffleId,
+                    partitionId
+                )
+            );
     event.setUnderStorage(storage);
+
+    // store it to cache.
+    partitionsOfStorage.putIfAbsent(appId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).putIfAbsent(shuffleId, Maps.newConcurrentMap());
+    partitionsOfStorage.get(appId).get(shuffleId).put(partitionId, storage);
     return storage;
   }
 
   @Override
   public Storage selectStorage(ShuffleDataReadEvent event) {
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
 
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    LocalStorage storage = null;
+    try {
+      storage = partitionsOfStorage.get(appId).get(shuffleId).get(partitionId);
+    } catch (NullPointerException npe) {

Review Comment:
   Looks like this code is still wrong. By using `partitionsOfStorage`, there would be no `NPE`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1053942401


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -217,6 +244,34 @@ public void removeResources(PurgeEvent event) {
     deleteHandler.delete(deletePaths.toArray(new String[deletePaths.size()]), appId, user);
   }
 
+  private void cleanupStorageSelectionCache(PurgeEvent event) {
+    Function<String, Boolean> deleteConditionFunc = null;
+    if (event instanceof AppPurgeEvent) {
+      deleteConditionFunc = partitionUnionKey -> UnionKey.startsWith(partitionUnionKey, event.getAppId());
+    } else if (event instanceof ShufflePurgeEvent) {
+      deleteConditionFunc =
+          partitionUnionKey -> UnionKey.startsWith(
+              partitionUnionKey,
+              event.getAppId(),
+              event.getShuffleIds()
+          );
+    }
+    deleteElement(
+        partitionsOfStorage,
+        deleteConditionFunc
+    );
+  }
+
+  private <K, V> void deleteElement(Map<K, V> map, Function<K, Boolean> deleteConditionFunc) {

Review Comment:
   This delete logic is not in the critical path, so it's OK for me.
   
   > Could you add a logging about how much time cost to delete element?
   
   Good idea. I wil



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1053957414


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -217,6 +244,34 @@ public void removeResources(PurgeEvent event) {
     deleteHandler.delete(deletePaths.toArray(new String[deletePaths.size()]), appId, user);
   }
 
+  private void cleanupStorageSelectionCache(PurgeEvent event) {
+    Function<String, Boolean> deleteConditionFunc = null;
+    if (event instanceof AppPurgeEvent) {
+      deleteConditionFunc = partitionUnionKey -> UnionKey.startsWith(partitionUnionKey, event.getAppId());
+    } else if (event instanceof ShufflePurgeEvent) {
+      deleteConditionFunc =
+          partitionUnionKey -> UnionKey.startsWith(
+              partitionUnionKey,
+              event.getAppId(),
+              event.getShuffleIds()
+          );
+    }
+    deleteElement(
+        partitionsOfStorage,
+        deleteConditionFunc
+    );
+  }
+
+  private <K, V> void deleteElement(Map<K, V> map, Function<K, Boolean> deleteConditionFunc) {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054030594


##########
server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java:
##########
@@ -139,33 +143,53 @@ public class LocalStorageManager extends SingleStorageManager {
 
   @Override
   public Storage selectStorage(ShuffleDataFlushEvent event) {
-    LocalStorage storage = localStorages.get(ShuffleStorageUtils.getStorageIndex(
-        localStorages.size(),
-        event.getAppId(),
-        event.getShuffleId(),
-        event.getStartPartition()));
-    if (storage.containsWriteHandler(event.getAppId(), event.getShuffleId(), event.getStartPartition())
-        && storage.isCorrupted()) {
-      LOG.error("storage " + storage.getBasePath() + " is corrupted");
-    }
-    if (storage.isCorrupted()) {
-      storage = getRepairedStorage(event.getAppId(), event.getShuffleId(), event.getStartPartition());
+    String appId = event.getAppId();
+    int shuffleId = event.getShuffleId();
+    int partitionId = event.getStartPartition();
+
+    LocalStorage storage = partitionsOfStorage.get(UnionKey.toKey(appId, shuffleId, partitionId));
+    if (storage != null) {
+      if (storage.isCorrupted()) {
+        if (storage.containsWriteHandler(appId, shuffleId, partitionId)) {
+          throw new RuntimeException("LocalStorage: " + storage.getBasePath() + " is corrupted.");
+        }
+      } else {
+        return storage;
+      }
     }
-    event.setUnderStorage(storage);
-    return storage;
+
+    List<LocalStorage> candidates = localStorages
+        .stream()
+        .filter(x -> x.canWrite() && !x.isCorrupted())
+        .collect(Collectors.toList());
+    final LocalStorage selectedStorage = candidates.get(
+        ShuffleStorageUtils.getStorageIndex(
+            candidates.size(),
+            appId,
+            shuffleId,
+            partitionId
+        )
+    );
+    return partitionsOfStorage.compute(
+        UnionKey.toKey(appId, shuffleId, partitionId),
+        (key, localStorage) -> {
+          // If this is the first time to select storage or existing storage is corrupted,
+          // we should refresh the cache.
+          if (localStorage == null || localStorage.isCorrupted()) {

Review Comment:
   For an event, if the storage is selected but event don't write any data to this storage, that means we could replace it to new storage, which won't cause data lost.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #424: [Improvement] Avoid selecting storage which has reached the high watermark

Posted by GitBox <gi...@apache.org>.
zuston commented on code in PR #424:
URL: https://github.com/apache/incubator-uniffle/pull/424#discussion_r1054041521


##########
common/src/main/java/org/apache/uniffle/common/UnionKey.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.common;
+
+import org.apache.commons.lang3.StringUtils;
+
+/**
+ * This class is to wrap multi elements to be as union key.
+ */
+public class UnionKey {
+  private static final String SPLIT_KEY = "_";
+
+  public static String toKey(Object... factors) {

Review Comment:
   toKey  ->  buildKeyFor ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org