[spark] branch master updated: [SPARK-24938][CORE] Prevent Netty from using onheap memory for headers without regard for configuration

[sr...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 055bf8e  [SPARK-24938][CORE] Prevent Netty from using onheap memory for headers without regard for configuration
055bf8e is described below

commit 055bf8ea1fe8dd4109772a282909ac3d69627f1b
Author: Nihar Sheth <ni...@gmail.com>
AuthorDate: Tue Jan 22 08:41:42 2019 -0600

    [SPARK-24938][CORE] Prevent Netty from using onheap memory for headers without regard for configuration
    
    ## What changes were proposed in this pull request?
    
    In MessageEncoder.java, the header would always be allocated on onheap memory regardless of whether netty was configured to use/prefer onheap or offheap. By default this made netty allocate 16mb of onheap memory for a tiny header message. It would be more practical to use preallocated buffers.
    
    Using a memory monitor tool on a simple spark application, the following services currently allocate 16 mb of onheap memory:
    netty-rpc-client
    netty-blockTransfer-client
    netty-external-shuffle-client
    
    With this change, the memory monitor tool reports all three of these services as using 0 b of onheap memory. The offheap memory allocation does not increase, but more of the already-allocated space is used.
    
    ## How was this patch tested?
    
    Manually tested change using spark-memory-tool https://github.com/squito/spark-memory
    
    Closes #22114 from NiharS/nettybuffer.
    
    Lead-authored-by: Nihar Sheth <ni...@gmail.com>
    Co-authored-by: Nihar Sheth <ns...@cloudera.com>
    Signed-off-by: Sean Owen <se...@databricks.com>
---
 .../src/main/java/org/apache/spark/network/protocol/MessageEncoder.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java b/common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java
index 997f74e..06dc447 100644
--- a/common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java
+++ b/common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java
@@ -79,7 +79,7 @@ public final class MessageEncoder extends MessageToMessageEncoder<Message> {
     // sent.
     int headerLength = 8 + msgType.encodedLength() + in.encodedLength();
     long frameLength = headerLength + (isBodyInFrame ? bodyLength : 0);
-    ByteBuf header = ctx.alloc().heapBuffer(headerLength);
+    ByteBuf header = ctx.alloc().buffer(headerLength);
     header.writeLong(frameLength);
     msgType.encode(header);
     in.encode(header);


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org