[GitHub] vanzin commented on a change in pull request #23278: [SPARK-24920][Core] Allow sharing Netty's memory pool allocators

[GitBox <gi...@apache.org>]

vanzin commented on a change in pull request #23278: [SPARK-24920][Core] Allow sharing Netty's memory pool allocators
URL: https://github.com/apache/spark/pull/23278#discussion_r240405360
 
 

 ##########
 File path: common/network-common/src/main/java/org/apache/spark/network/util/NettyUtils.java
 ##########
 @@ -95,6 +99,21 @@ public static String getRemoteAddress(Channel channel) {
     return "<unknown remote>";
   }
 
+  /**
+   * Returns the lazily created shared pooled ByteBuf allocator for the specified allowCache
+   * parameter value.
+   */
+  public static synchronized PooledByteBufAllocator getSharedPooledByteBufAllocator(
+      boolean allowDirectBufs,
+      boolean allowCache) {
+    final int index = allowCache ? 0 : 1;
+    if (_sharedPooledByteBufAllocator[index] == null) {
+      _sharedPooledByteBufAllocator[index] =
+        createPooledByteBufAllocator(allowDirectBufs, allowCache, 0 /* numCores */);
 
 Review comment:
   Hmm.. it may be good to think about having a better way to define the number of cores here. The issue is that by using the default you may be wasting resources.
   
   e.g. if your container is only requesting 1 CPU but the host actually has 32 CPUs, this will create 64 allocation arenas.
   
   (For example, `SparkTransportConf.fromSparkConf` tries to limit thread pool sizes and thus the size of the allocators by using the configured number of CPUs.)
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org