[spark] branch master updated: [MINOR][SHUFFLE] Include IOException in warning log of finalizeShuffleMerge

[sr...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 074894ce70e [MINOR][SHUFFLE] Include IOException in warning log of finalizeShuffleMerge
074894ce70e is described below

commit 074894ce70e90717cbc81f7e6abc53d10872cda3
Author: Ted Yu <yu...@gmail.com>
AuthorDate: Sat Jan 21 15:56:40 2023 -0600

    [MINOR][SHUFFLE] Include IOException in warning log of finalizeShuffleMerge
    
    ### What changes were proposed in this pull request?
    This PR adds `ioe` to the warning log of `finalizeShuffleMerge`.
    
    ### Why are the changes needed?
    With `ioe` logged, user would have more clue as to the root cause.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    Existing test suite.
    
    Closes #39654 from tedyu/shuffle-ioe.
    
    Authored-by: Ted Yu <yu...@gmail.com>
    Signed-off-by: Sean Owen <sr...@gmail.com>
---
 .../org/apache/spark/network/shuffle/RemoteBlockPushResolver.java    | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
index fb3f8109a1a..a2e8219228a 100644
--- a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
+++ b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
@@ -814,8 +814,9 @@ public class RemoteBlockPushResolver implements MergedShuffleFileManager {
             }
           } catch (IOException ioe) {
             logger.warn("{} attempt {} shuffle {} shuffleMerge {}: exception while " +
-                "finalizing shuffle partition {}", msg.appId, msg.appAttemptId, msg.shuffleId,
-                msg.shuffleMergeId, partition.reduceId);
+                "finalizing shuffle partition {}. Exception message: {}", msg.appId,
+                msg.appAttemptId, msg.shuffleId, msg.shuffleMergeId, partition.reduceId,
+                ioe.getMessage());
           } finally {
             partition.closeAllFilesAndDeleteIfNeeded(false);
           }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org