You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Hannes Schmidt (JIRA)" <ji...@apache.org> on 2011/05/15 20:41:47 UTC
[jira] [Created] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
-------------------------------------------------------------------------------
Key: CASSANDRA-2654
URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 0.8.0 beta 2, 0.7.5, 0.6.13
Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
Reporter: Hannes Schmidt
NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044147#comment-13044147 ]
Hudson commented on CASSANDRA-2654:
-----------------------------------
Integrated in Cassandra-0.6 #65 (See [https://builds.apache.org/hudson/job/Cassandra-0.6/65/])
work around nio socket memory retention bug
patch by Hannes Schmidt and jbellis; tested by Thibaut, reviewed by slebresne for CASSANDRA-2654
jbellis :
Files :
* /cassandra/branches/cassandra-0.6/CHANGES.txt
* /cassandra/branches/cassandra-0.6/src/java/org/apache/cassandra/net/IncomingTcpConnection.java
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Assignee: Hannes Schmidt
> Fix For: 0.7.7, 0.8.1
>
> Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4-0.7.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042268#comment-13042268 ]
Sylvain Lebresne commented on CASSANDRA-2654:
---------------------------------------------
looks good, +1
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Fix For: 0.6.12
>
> Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4-0.7.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2654:
--------------------------------------
Attachment: 2654-v2.txt
v2 adds an explanatory comment and simplifies the logic a little
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Fix For: 0.6.12
>
> Attachments: 2654-v2.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Chris Burroughs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048560#comment-13048560 ]
Chris Burroughs commented on CASSANDRA-2654:
--------------------------------------------
Ran with it for about two weeks, no noticeable effect (good or ill) on my 0.7.x cluster. (It was never clear if this if I was experiencing the same problem.)
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Assignee: Hannes Schmidt
> Fix For: 0.7.7, 0.8.1
>
> Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4-0.7.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2654) Work around native
heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Hannes Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034160#comment-13034160 ]
Hannes Schmidt edited comment on CASSANDRA-2654 at 5/16/11 6:07 PM:
--------------------------------------------------------------------
v3: Reverted to original logic but kept comment and naming convention for constants.
was (Author: eyealike):
Reverted to original logic but kept comment and naming convention for constants.
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Fix For: 0.6.12
>
> Attachments: 2654-v2.txt, 2654-v3.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Hannes Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033825#comment-13033825 ]
Hannes Schmidt commented on CASSANDRA-2654:
-------------------------------------------
I think I am wrong about readFully preventing the OOB. Does v2 pass the tests?
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Fix For: 0.6.12
>
> Attachments: 2654-v2.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2654:
--------------------------------------
Attachment: 2654-v4.txt
v4 changes v2 for loop condition to "offset < size - remainder".
As you note, doing a readFully with size of zero is harmless.
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Fix For: 0.6.12
>
> Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Hannes Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hannes Schmidt updated CASSANDRA-2654:
--------------------------------------
Attachment: 2654-v3.txt
Reverted to original logic but kept comment and naming convention for constants.
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Fix For: 0.6.12
>
> Attachments: 2654-v2.txt, 2654-v3.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Hannes Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033798#comment-13033798 ]
Hannes Schmidt commented on CASSANDRA-2654:
-------------------------------------------
Forgot footnotes:
[1] http://www.mail-archive.com/user@cassandra.apache.org/msg13357.html
[2] http://bugs.sun.com/view_bug.do?bug_id=6210541
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Fix For: 0.6.12
>
> Attachments: chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Hannes Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034090#comment-13034090 ]
Hannes Schmidt commented on CASSANDRA-2654:
-------------------------------------------
Since reading from a socket isn't idempotent, what I called "redundant" is actually harmful.
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Fix For: 0.6.12
>
> Attachments: 2654-v2.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Hannes Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hannes Schmidt updated CASSANDRA-2654:
--------------------------------------
Attachment: chunking.diff
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Fix For: 0.6.12
>
> Attachments: chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Thibaut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040183#comment-13040183 ]
Thibaut commented on CASSANDRA-2654:
------------------------------------
Tested this with 0.7.6 on our 100 node production cluster.
I'm only seeing occasional GC calls every few hours in the log, instead of a series of GC calls which forced us to restart cassandra. The patch seems to help a lot.
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Fix For: 0.6.12
>
> Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4-0.7.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2654:
--------------------------------------
Attachment: 2654-v4-0.7.txt
0.7 version of v4
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Fix For: 0.6.12
>
> Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4-0.7.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Chris Burroughs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039164#comment-13039164 ]
Chris Burroughs commented on CASSANDRA-2654:
--------------------------------------------
Patch doesn't apply cleanly to 0.7.x, although it *looks* like the differences are not too great. I could probably arrange a test of a 0.7.x patch in production.
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Fix For: 0.6.12
>
> Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2654:
--------------------------------------
Affects Version/s: (was: 0.8.0 beta 2)
(was: 0.7.5)
(was: 0.6.13)
Fix Version/s: (was: 0.6.12)
0.8.1
0.7.7
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Fix For: 0.7.7, 0.8.1
>
> Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4-0.7.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Hannes Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033824#comment-13033824 ]
Hannes Schmidt commented on CASSANDRA-2654:
-------------------------------------------
I think v2 has issues: it has an out-of-bounds condition it invokes readFully redundantly.
Let's say size is 1, then v2 does this:
contentBytes = new byte[1];
input.readFully(contentBytes, 0, CHUNK_SIZE); // oob
input.readFully(contentBytes, 0, 1); // redundant
If size is CHUNK_SIZE it does this
contentBytes = new byte[CHUNK_SIZE];
input.readFully(contentBytes, 0, CHUNK_SIZE);
input.readFully(contentBytes, CHUNK_SIZE, 0); // redundant, 3rd argument is 0
If size is CHUNK_SIZE+1 it does this
contentBytes = new byte[CHUNK_SIZE+1];
input.readFully(contentBytes, 0, CHUNK_SIZE);
input.readFully(contentBytes, CHUNK_SIZE, CHUNK_SIZE); // oob
input.readFully(contentBytes, CHUNK_SIZE, 1); // redundant
DataInputStream.readFully(a,o,l) actually prevents the OOB but it's not good style to rely on that. The redundant invocations do matter when size is near CHUNK_SIZE.
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Fix For: 0.6.12
>
> Attachments: 2654-v2.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in
sun.nio.ch.Util affecting IncomingTcpConnection
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048565#comment-13048565 ]
Jonathan Ellis commented on CASSANDRA-2654:
-------------------------------------------
it would only be noticeable if you query very large resultsets
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-2654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
> Reporter: Hannes Schmidt
> Assignee: Hannes Schmidt
> Fix For: 0.7.7, 0.8.1
>
> Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4-0.7.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira