[jira] [Reopened] (CASSANDRA-8670) Large columns + NIO memory pooling causes excessive direct memory usage

["Benedict (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CASSANDRA-8670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Benedict reopened CASSANDRA-8670:
---------------------------------

A little niggle was bugging me, and I decided to check it out, and I think it warrants further consideration:

The [windows|http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/00cd9dc3c2b5/src/windows/native/java/net/SocketOutputStream.c] and [solaris|http://hg.openjdk.java.net/jdk6/jdk6/jdk/annotate/1a53516ce032/src/solaris/native/java/net/SocketOutputStream.c] implementations of SocketOutputStream allocate heap (not Java heap) memory of the total size of the data being written to them, if they're above the size of the stack memory they allocate by default (according to the source, this ranges from 8K to 64K on [Solaris|http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/4dbd83eb0250/src/solaris/native/java/net/net_util_md.h#l123], and 2K on [Windows|http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/0c27202d66c1/src/windows/native/java/net/net_util_md.h#l236], for the stack buffer, with the heap buffer being a little larger)

This raises a few questions: 
# how much of our thread stack frame ends up being used by these methods? Seems we have some headroom with even the minimum stack frame, so maybe not worth worrying about; but larger writes will incur some non-java-heap allocations without much benefit
# should we simply copy our byte[] data into our direct buffer, and call write on this instead? this has the added advantage of avoiding the OutputStream entirely in situations where we have access to the Channel, which removes an entire codepath from the common execution path. There seems to be no downside, especially if our buffer is large enough, since the copying is exactly what the native code does anyway.
# what is behaviour on Linux? I can't for the life of me find the linux implementation; best I can assume is it is very similar to Solaris


> Large columns + NIO memory pooling causes excessive direct memory usage
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-8670
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8670
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Ariel Weisberg
>            Assignee: Ariel Weisberg
>             Fix For: 3.0
>
>         Attachments: largecolumn_test.py
>
>
> If you provide a large byte array to NIO and ask it to populate the byte array from a socket it will allocate a thread local byte buffer that is the size of the requested read no matter how large it is. Old IO wraps new IO for sockets (but not files) so old IO is effected as well.
> Even If you are using Buffered{Input | Output}Stream you can end up passing a large byte array to NIO. The byte array read method will pass the array to NIO directly if it is larger than the internal buffer.  
> Passing large cells between nodes as part of intra-cluster messaging can cause the NIO pooled buffers to quickly reach a high watermark and stay there. This ends up costing 2x the largest cell size because there is a buffer for input and output since they are different threads. This is further multiplied by the number of nodes in the cluster - 1 since each has a dedicated thread pair with separate thread locals.
> Anecdotally it appears that the cost is doubled beyond that although it isn't clear why. Possibly the control connections or possibly there is some way in which multiple 
> Need a workload in CI that tests the advertised limits of cells on a cluster. It would be reasonable to ratchet down the max direct memory for the test to trigger failures if a memory pooling issue is introduced. I don't think we need to test concurrently pulling in a lot of them, but it should at least work serially.
> The obvious fix to address this issue would be to read in smaller chunks when dealing with large values. I think small should still be relatively large (4 megabytes) so that code that is reading from a disk can amortize the cost of a seek. It can be hard to tell what the underlying thing being read from is going to be in some of the contexts where we might choose to implement switching to reading chunks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)