You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "zhoukang (JIRA)" <ji...@apache.org> on 2017/07/26 02:25:00 UTC
[jira] [Closed] (SPARK-21527) Use buffer limit in order to take
advantage of JAVA NIO Util's buffercache
[ https://issues.apache.org/jira/browse/SPARK-21527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhoukang closed SPARK-21527.
----------------------------
Resolution: Invalid
> Use buffer limit in order to take advantage of JAVA NIO Util's buffercache
> ---------------------------------------------------------------------------
>
> Key: SPARK-21527
> URL: https://issues.apache.org/jira/browse/SPARK-21527
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.2.0
> Reporter: zhoukang
>
> Right now, ChunkedByteBuffer#writeFully do not slice bytes first.We observe code in java nio Util below:
> {code:java}
> public static ByteBuffer More ...getTemporaryDirectBuffer(int size) {
> BufferCache cache = bufferCache.get();
> ByteBuffer buf = cache.get(size);
> if (buf != null) {
> return buf;
> } else {
> // No suitable buffer in the cache so we need to allocate a new
> // one. To avoid the cache growing then we remove the first
> // buffer from the cache and free it.
> if (!cache.isEmpty()) {
> buf = cache.removeFirst();
> free(buf);
> }
> return ByteBuffer.allocateDirect(size);
> }
> }
> {code}
> If we slice first with a fixed size, we can use buffer cache and only need to allocate at the first write call.
> Since we allocate new buffer, we can not control the free time of this buffer.This once cause memory issue in our production cluster.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org