You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Hannes Schmidt (JIRA)" <ji...@apache.org> on 2011/05/15 20:41:47 UTC

[jira] [Created] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
-------------------------------------------------------------------------------

                 Key: CASSANDRA-2654
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
             Project: Cassandra
          Issue Type: Improvement
          Components: Core
    Affects Versions: 0.8.0 beta 2, 0.7.5, 0.6.13
         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)

Also observed on Sun/Oracle JDK. Probably platform- and os-independent.

            Reporter: Hannes Schmidt


NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044147#comment-13044147 ] 

Hudson commented on CASSANDRA-2654:
-----------------------------------

Integrated in Cassandra-0.6 #65 (See [https://builds.apache.org/hudson/job/Cassandra-0.6/65/])
    work around nio socket memory retention bug
patch by Hannes Schmidt and jbellis; tested by Thibaut, reviewed by slebresne for CASSANDRA-2654

jbellis : 
Files : 
* /cassandra/branches/cassandra-0.6/CHANGES.txt
* /cassandra/branches/cassandra-0.6/src/java/org/apache/cassandra/net/IncomingTcpConnection.java


> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>            Assignee: Hannes Schmidt
>             Fix For: 0.7.7, 0.8.1
>
>         Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4-0.7.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042268#comment-13042268 ] 

Sylvain Lebresne commented on CASSANDRA-2654:
---------------------------------------------

looks good, +1

> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>             Fix For: 0.6.12
>
>         Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4-0.7.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-2654:
--------------------------------------

    Attachment: 2654-v2.txt

v2 adds an explanatory comment and simplifies the logic a little

> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>             Fix For: 0.6.12
>
>         Attachments: 2654-v2.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Chris Burroughs (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048560#comment-13048560 ] 

Chris Burroughs commented on CASSANDRA-2654:
--------------------------------------------

Ran with it for about two weeks, no noticeable effect (good or ill) on my 0.7.x cluster.  (It was never clear if this if I was experiencing the same problem.)

> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>            Assignee: Hannes Schmidt
>             Fix For: 0.7.7, 0.8.1
>
>         Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4-0.7.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Issue Comment Edited] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Hannes Schmidt (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034160#comment-13034160 ] 

Hannes Schmidt edited comment on CASSANDRA-2654 at 5/16/11 6:07 PM:
--------------------------------------------------------------------

v3: Reverted to original logic but kept comment and naming convention for constants.

      was (Author: eyealike):
    Reverted to original logic but kept comment and naming convention for constants.
  
> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>             Fix For: 0.6.12
>
>         Attachments: 2654-v2.txt, 2654-v3.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Hannes Schmidt (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033825#comment-13033825 ] 

Hannes Schmidt commented on CASSANDRA-2654:
-------------------------------------------

I think I am wrong about readFully preventing the OOB. Does v2 pass the tests?

> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>             Fix For: 0.6.12
>
>         Attachments: 2654-v2.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-2654:
--------------------------------------

    Attachment: 2654-v4.txt

v4 changes v2 for loop condition to "offset < size - remainder".

As you note, doing a readFully with size of zero is harmless.

> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>             Fix For: 0.6.12
>
>         Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Hannes Schmidt (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hannes Schmidt updated CASSANDRA-2654:
--------------------------------------

    Attachment: 2654-v3.txt

Reverted to original logic but kept comment and naming convention for constants.

> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>             Fix For: 0.6.12
>
>         Attachments: 2654-v2.txt, 2654-v3.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Hannes Schmidt (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033798#comment-13033798 ] 

Hannes Schmidt commented on CASSANDRA-2654:
-------------------------------------------

Forgot footnotes:

[1] http://www.mail-archive.com/user@cassandra.apache.org/msg13357.html
[2] http://bugs.sun.com/view_bug.do?bug_id=6210541

> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>             Fix For: 0.6.12
>
>         Attachments: chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Hannes Schmidt (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034090#comment-13034090 ] 

Hannes Schmidt commented on CASSANDRA-2654:
-------------------------------------------

Since reading from a socket isn't idempotent, what I called "redundant" is actually harmful. 

> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>             Fix For: 0.6.12
>
>         Attachments: 2654-v2.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Hannes Schmidt (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hannes Schmidt updated CASSANDRA-2654:
--------------------------------------

    Attachment: chunking.diff

> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>             Fix For: 0.6.12
>
>         Attachments: chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Thibaut (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040183#comment-13040183 ] 

Thibaut commented on CASSANDRA-2654:
------------------------------------

Tested this with 0.7.6 on our 100 node production cluster.

I'm only seeing occasional GC calls every few hours in the log, instead of a series of GC calls which forced us to restart cassandra. The patch seems to help a lot.


> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>             Fix For: 0.6.12
>
>         Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4-0.7.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-2654:
--------------------------------------

    Attachment: 2654-v4-0.7.txt

0.7 version of v4

> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>             Fix For: 0.6.12
>
>         Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4-0.7.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Chris Burroughs (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039164#comment-13039164 ] 

Chris Burroughs commented on CASSANDRA-2654:
--------------------------------------------

Patch doesn't apply cleanly to 0.7.x, although it *looks* like the differences are not too great.  I could probably arrange a test of a 0.7.x patch in production.

> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>             Fix For: 0.6.12
>
>         Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-2654:
--------------------------------------

    Affects Version/s:     (was: 0.8.0 beta 2)
                           (was: 0.7.5)
                           (was: 0.6.13)
        Fix Version/s:     (was: 0.6.12)
                       0.8.1
                       0.7.7

> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>             Fix For: 0.7.7, 0.8.1
>
>         Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4-0.7.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Hannes Schmidt (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033824#comment-13033824 ] 

Hannes Schmidt commented on CASSANDRA-2654:
-------------------------------------------


I think v2 has issues: it has an out-of-bounds condition it invokes readFully redundantly. 

Let's say size is 1, then v2 does this:

contentBytes = new byte[1];
input.readFully(contentBytes, 0, CHUNK_SIZE); // oob
input.readFully(contentBytes, 0, 1); // redundant 

If size is CHUNK_SIZE it does this

contentBytes = new byte[CHUNK_SIZE];
input.readFully(contentBytes, 0, CHUNK_SIZE);
input.readFully(contentBytes, CHUNK_SIZE, 0); // redundant, 3rd argument is 0

If size is CHUNK_SIZE+1 it does this

contentBytes = new byte[CHUNK_SIZE+1];
input.readFully(contentBytes, 0, CHUNK_SIZE);
input.readFully(contentBytes, CHUNK_SIZE, CHUNK_SIZE); // oob
input.readFully(contentBytes, CHUNK_SIZE, 1); // redundant

DataInputStream.readFully(a,o,l) actually prevents the OOB but it's not good style to rely on that. The redundant invocations do matter when size is near CHUNK_SIZE.

> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.6.13, 0.7.5, 0.8.0 beta 2
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>             Fix For: 0.6.12
>
>         Attachments: 2654-v2.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2654) Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048565#comment-13048565 ] 

Jonathan Ellis commented on CASSANDRA-2654:
-------------------------------------------

it would only be noticeable if you query very large resultsets

> Work around native heap leak in sun.nio.ch.Util affecting IncomingTcpConnection
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2654
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2654
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>         Environment: OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> Also observed on Sun/Oracle JDK. Probably platform- and os-independent.
>            Reporter: Hannes Schmidt
>            Assignee: Hannes Schmidt
>             Fix For: 0.7.7, 0.8.1
>
>         Attachments: 2654-v2.txt, 2654-v3.txt, 2654-v4-0.7.txt, 2654-v4.txt, chunking.diff
>
>
> NIO's leaky, per-thread caching of direct buffers in combination with IncomingTcpConnection's eager buffering of messages leads to leakage of large amounts of native heap. Details in [1]. More on the root cause in [2]. Even though it doesn't fix the leak, attached patch has been found to alleviate the problem by keeping the size of each direct buffer modest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira