[spark] branch master updated: [SPARK-32663][CORE] Avoid individual closing of pooled TransportClients (which must be closed through the pool)

[mr...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

mridulm80 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 79b4dea  [SPARK-32663][CORE] Avoid individual closing of pooled TransportClients (which must be closed through the pool)
79b4dea is described below

commit 79b4dea1b08adc9d4b545a1af29ac50fa603936a
Author: “attilapiros” <pi...@gmail.com>
AuthorDate: Fri Aug 21 01:02:33 2020 -0500

    [SPARK-32663][CORE] Avoid individual closing of pooled TransportClients (which must be closed through the pool)
    
    ### What changes were proposed in this pull request?
    
    Removing the individual `close` method calls on the pooled `TransportClient` instances.
    The pooled clients should be only closed via `TransportClientFactory#close()`.
    
    ### Why are the changes needed?
    
    Reusing a closed `TransportClient` leads to the exception `java.nio.channels.ClosedChannelException`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    This is a trivial case which is not tested by specific test.
    
    Closes #29492 from attilapiros/SPARK-32663.
    
    Authored-by: “attilapiros” <pi...@gmail.com>
    Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
---
 .../org/apache/spark/network/shuffle/ExternalBlockStoreClient.java  | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockStoreClient.java b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockStoreClient.java
index 51dc333..9fdb632 100644
--- a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockStoreClient.java
+++ b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockStoreClient.java
@@ -175,8 +175,6 @@ public class ExternalBlockStoreClient extends BlockStoreClient {
           logger.warn("Error trying to remove RDD blocks " + Arrays.toString(blockIds) +
             " via external shuffle service from executor: " + execId, t);
           numRemovedBlocksFuture.complete(0);
-        } finally {
-          client.close();
         }
       }
 
@@ -185,7 +183,6 @@ public class ExternalBlockStoreClient extends BlockStoreClient {
         logger.warn("Error trying to remove RDD blocks " + Arrays.toString(blockIds) +
           " via external shuffle service from executor: " + execId, e);
         numRemovedBlocksFuture.complete(0);
-        client.close();
       }
     });
     return numRemovedBlocksFuture;
@@ -212,8 +209,6 @@ public class ExternalBlockStoreClient extends BlockStoreClient {
               Arrays.toString(getLocalDirsMessage.execIds) + " via external shuffle service",
               t.getCause());
             hostLocalDirsCompletable.completeExceptionally(t);
-          } finally {
-            client.close();
           }
         }
 
@@ -223,7 +218,6 @@ public class ExternalBlockStoreClient extends BlockStoreClient {
             Arrays.toString(getLocalDirsMessage.execIds) + " via external shuffle service",
             t.getCause());
           hostLocalDirsCompletable.completeExceptionally(t);
-          client.close();
         }
       });
     } catch (IOException | InterruptedException e) {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org