[jira] [Updated] (SPARK-21527) Use buffer limit in order to take advantage of JAVA NIO Util's buffercache

["zhoukang (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-21527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

zhoukang updated SPARK-21527:
-----------------------------
    Summary: Use buffer limit in order to take advantage of  JAVA NIO Util's buffercache  (was: Use buffer limit in order to use JAVA NIO Util's buffercache)

> Use buffer limit in order to take advantage of  JAVA NIO Util's buffercache
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-21527
>                 URL: https://issues.apache.org/jira/browse/SPARK-21527
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.2.0
>            Reporter: zhoukang
>
> Right now, ChunkedByteBuffer#writeFully do not slice bytes first.We observe code in java nio Util below:
> {code:java}
> public static ByteBuffer More ...getTemporaryDirectBuffer(int size) {
>         BufferCache cache = bufferCache.get();
>         ByteBuffer buf = cache.get(size);
>         if (buf != null) {
>             return buf;
>         } else {
>             // No suitable buffer in the cache so we need to allocate a new
>             // one. To avoid the cache growing then we remove the first
>             // buffer from the cache and free it.
>             if (!cache.isEmpty()) {
>                 buf = cache.removeFirst();
>                 free(buf);
>             }
>             return ByteBuffer.allocateDirect(size);
>         }
>     }
> {code}
> If we slice first with a fixed size, we can use buffer cache and only need to allocate at the first write call.
> Since we allocate new buffer, we can not control the free time of this buffer.This once cause memory issue in our production cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org