[GitHub] attilapiros opened a new pull request #23278: [SPARK-24920][Core] Allow sharing Netty's memory pool allocators

[GitBox <gi...@apache.org>]

attilapiros opened a new pull request #23278: [SPARK-24920][Core] Allow sharing Netty's memory pool allocators
URL: https://github.com/apache/spark/pull/23278
 
 
   ## What changes were proposed in this pull request?
   
   Introducing shared polled ByteBuf allocators. 
   This feature can be enabled via the "spark.network.sharedByteBufAllocators" configuration. 
   
   When it is on then only two pooled ByteBuf allocators are created: 
   - one for transport servers where caching is allowed and
   - one for transport clients where caching is disabled
   
   This way the cache allowance remains as before. 
   Both shareable pools are created with numCores parameter set to 0 (which defaults to the available processors) as conf.serverThreads() and conf.clientThreads() are module dependant and the lazy creation of this allocators would lead to unpredicted behaviour.
   
   When "spark.network.sharedByteBufAllocators" is false then a new allocator is created for every transport client and server separately as was before this PR.
   
   ## How was this patch tested?
   
   Existing unit tests.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org