You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "xianjingfeng (via GitHub)" <gi...@apache.org> on 2024/08/21 10:04:46 UTC

[PR] [#2073] fix(server): throws EventRetryException If IOException is encountered during create write handler. [incubator-uniffle]

xianjingfeng opened a new pull request, #2074:
URL: https://github.com/apache/incubator-uniffle/pull/2074

   <!--
   1. Title: [#<issue>] <type>(<scope>): <subject>
      Examples:
        - "[#123] feat(operator): support xxx"
        - "[#233] fix: check null before access result in xxx"
        - "[MINOR] refactor: fix typo in variable name"
        - "[MINOR] docs: fix typo in README"
        - "[#255] test: fix flaky test NameOfTheTest"
      Reference: https://www.conventionalcommits.org/en/v1.0.0/
   2. Contributor guidelines:
      https://github.com/apache/incubator-uniffle/blob/master/CONTRIBUTING.md
   3. If the PR is unfinished, please mark this PR as draft.
   -->
   
   ### What changes were proposed in this pull request?
   
   1. Throws EventRetryException If IOException is encountered during create write handler.
   
   ### Why are the changes needed?
   
   Fix: #2073
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   UT


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2073] fix(server): throws EventRetryException If Exception is encountered during create write handler [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston merged PR #2074:
URL: https://github.com/apache/incubator-uniffle/pull/2074


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2073] fix(server): throws EventRetryException If IOException is encountered during create write handler [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #2074:
URL: https://github.com/apache/incubator-uniffle/pull/2074#discussion_r1726890027


##########
server/src/main/java/org/apache/uniffle/server/ShuffleFlushManager.java:
##########
@@ -169,7 +170,14 @@ public void processFlushEvent(ShuffleDataFlushEvent event) throws Exception {
               storageDataReplica,
               user,
               maxConcurrencyPerPartitionToWrite);
-      ShuffleWriteHandler handler = storage.getOrCreateWriteHandler(request);
+      ShuffleWriteHandler handler;
+      try {
+        handler = storage.getOrCreateWriteHandler(request);

Review Comment:
   If there is no write permission, the retry will overcome this case? Or just retry again to write this disk



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2073] fix(server): throws EventRetryException If IOException is encountered during create write handler [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #2074:
URL: https://github.com/apache/incubator-uniffle/pull/2074#discussion_r1726894832


##########
server/src/main/java/org/apache/uniffle/server/ShuffleFlushManager.java:
##########
@@ -169,7 +170,14 @@ public void processFlushEvent(ShuffleDataFlushEvent event) throws Exception {
               storageDataReplica,
               user,
               maxConcurrencyPerPartitionToWrite);
-      ShuffleWriteHandler handler = storage.getOrCreateWriteHandler(request);
+      ShuffleWriteHandler handler;
+      try {
+        handler = storage.getOrCreateWriteHandler(request);

Review Comment:
   BTW, I hope this could be tested in the UTs



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2073] fix(server): throws EventRetryException If Exception is encountered during create write handler [incubator-uniffle]

Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on PR #2074:
URL: https://github.com/apache/incubator-uniffle/pull/2074#issuecomment-2316624316

   @zuston PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2073] fix(server): throws EventRetryException If IOException is encountered during create write handler [incubator-uniffle]

Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on code in PR #2074:
URL: https://github.com/apache/incubator-uniffle/pull/2074#discussion_r1728151822


##########
server/src/main/java/org/apache/uniffle/server/ShuffleFlushManager.java:
##########
@@ -169,7 +170,14 @@ public void processFlushEvent(ShuffleDataFlushEvent event) throws Exception {
               storageDataReplica,
               user,
               maxConcurrencyPerPartitionToWrite);
-      ShuffleWriteHandler handler = storage.getOrCreateWriteHandler(request);
+      ShuffleWriteHandler handler;
+      try {
+        handler = storage.getOrCreateWriteHandler(request);

Review Comment:
   > If there is no write permission, the retry will overcome this case? Or just retry again to write this disk
   
   It dependency the logic of `StorageManagerSelector`. If EventRetryException be catch in `org.apache.uniffle.server.DefaultFlushEventHandler#handleEventAndUpdateMetrics`, the retryTimes of the flush event will be increase



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2073] fix(server): throws EventRetryException If Exception is encountered during create write handler [incubator-uniffle]

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #2074:
URL: https://github.com/apache/incubator-uniffle/pull/2074#issuecomment-2324045334

   > Should we change the default value of `rss.server.hybrid.storage.fallback.strategy.class` to `RotateStorageManagerFallbackStrategy`? @zuston @jerqi @rickyma @smallzhongfeng
   > 
   > https://github.com/apache/incubator-uniffle/blob/57f0f8bbcd530feb6ab1284c2eed0c08b9e2c9ae/server/src/main/java/org/apache/uniffle/server/storage/HybridStorageManager.java#L85-L88
   
   I prefer not.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2073] fix(server): throws EventRetryException If IOException is encountered during create write handler [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #2074:
URL: https://github.com/apache/incubator-uniffle/pull/2074#discussion_r1728195542


##########
server/src/main/java/org/apache/uniffle/server/ShuffleFlushManager.java:
##########
@@ -169,7 +170,14 @@ public void processFlushEvent(ShuffleDataFlushEvent event) throws Exception {
               storageDataReplica,
               user,
               maxConcurrencyPerPartitionToWrite);
-      ShuffleWriteHandler handler = storage.getOrCreateWriteHandler(request);
+      ShuffleWriteHandler handler;
+      try {
+        handler = storage.getOrCreateWriteHandler(request);

Review Comment:
   Got it. +1 for this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2073] fix(server): throws EventRetryException If IOException is encountered during create write handler [incubator-uniffle]

Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on PR #2074:
URL: https://github.com/apache/incubator-uniffle/pull/2074#issuecomment-2306385312

   Should we change the default value of `rss.server.hybrid.storage.fallback.strategy.class` to `RotateStorageManagerFallbackStrategy`? @zuston @jerqi @rickyma @smallzhongfeng 
   https://github.com/apache/incubator-uniffle/blob/57f0f8bbcd530feb6ab1284c2eed0c08b9e2c9ae/server/src/main/java/org/apache/uniffle/server/storage/HybridStorageManager.java#L85-L88


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2073] fix(server): throws EventRetryException If IOException is encountered during create write handler [incubator-uniffle]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2074:
URL: https://github.com/apache/incubator-uniffle/pull/2074#issuecomment-2301709916

   ## Test Results
    2 792 files  ±0   2 792 suites  ±0   5h 51m 28s :stopwatch: +52s
      988 tests ±0     986 :white_check_mark:  - 1   1 :zzz: ±0  0 :x: ±0  1 :fire: +1 
   12 403 runs  ±0  12 387 :white_check_mark:  - 1  15 :zzz: ±0  0 :x: ±0  1 :fire: +1 
   
   For more details on these errors, see [this check](https://github.com/apache/incubator-uniffle/runs/29050035254).
   
   Results for commit 8634448b. ± Comparison against base commit 0495fbca.
   
   [test-results]:data:application/gzip;base64,H4sIAILBxWYC/03MTQ6DIBCG4asY1l3AMAXsZRoFTEj9aRBWpncvGESW3zOT9yCTm+1OXh3IHh4d2aML7TbRD8FtaxZGlUqU7iF/9Pd671Hrk8RNH/dNxCpMg5sT0ArW+82XFx/X3GSAlJd5RRlwJauV6vOCpnruNqq3ZXEhDaIER0Q1WmMBceDjOAmupTAARksGGnkvLFDy+wOILx47EgEAAA==
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org