You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Garrett Smith (JIRA)" <ji...@apache.org> on 2009/03/21 21:48:50 UTC

[jira] Created: (THRIFT-391) Memory leak in fastbinary

Memory leak in fastbinary
-------------------------

                 Key: THRIFT-391
                 URL: https://issues.apache.org/jira/browse/THRIFT-391
             Project: Thrift
          Issue Type: Bug
          Components: Compiler (Python)
         Environment: Linux 2.6.21 (Debian)
            Reporter: Garrett Smith


When a struct has a map member, fastbinary.binary_encode leaks memory.

Use testmem.tar.gz app to recreate the problem.

{quote}
# tar xzf testmem.tar.gz
# cd testmem
# ./testmem fast
{quote}

As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.

To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:

{quote}
# ./testmem default
{quote}

Note (again, using top, pidstat, etc.) that the memory used is constant.

It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (THRIFT-391) Memory leak in fastbinary

Posted by "Alexander Shigin (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/THRIFT-391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shigin updated THRIFT-391:
------------------------------------

    Attachment: thrift-fastbinary-leak.patch

It seems the patch solves the problem. 

Can anyone review it? David, I think we can solve the problem by eliminating Py_INCREF (there is your TODO), but i am not guru of python c-api.

> Memory leak in fastbinary
> -------------------------
>
>                 Key: THRIFT-391
>                 URL: https://issues.apache.org/jira/browse/THRIFT-391
>             Project: Thrift
>          Issue Type: Bug
>          Components: Library (Python)
>         Environment: Linux 2.6.21 (Debian)
>            Reporter: Garrett Smith
>             Fix For: 0.1
>
>         Attachments: testmem.tar.gz, thrift-fastbinary-leak.patch
>
>
> When a struct has a map member, fastbinary.binary_encode leaks memory.
> Use testmem.tar.gz app to recreate the problem.
> {code}
> # tar xzf testmem.tar.gz
> # cd testmem
> # ./testmem fast
> {code}
> As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
> To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
> {code}
> # ./testmem default
> {code}
> Note (again, using top, pidstat, etc.) that the memory used is constant.
> It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (THRIFT-391) Memory leak in fastbinary

Posted by "Alexander Shigin (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/THRIFT-391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shigin updated THRIFT-391:
------------------------------------

      Component/s:     (was: Compiler (Python))
                   Library (Python)
       Patch Info: [Patch Available]
    Fix Version/s: 0.1

> Memory leak in fastbinary
> -------------------------
>
>                 Key: THRIFT-391
>                 URL: https://issues.apache.org/jira/browse/THRIFT-391
>             Project: Thrift
>          Issue Type: Bug
>          Components: Library (Python)
>         Environment: Linux 2.6.21 (Debian)
>            Reporter: Garrett Smith
>             Fix For: 0.1
>
>         Attachments: testmem.tar.gz
>
>
> When a struct has a map member, fastbinary.binary_encode leaks memory.
> Use testmem.tar.gz app to recreate the problem.
> {code}
> # tar xzf testmem.tar.gz
> # cd testmem
> # ./testmem fast
> {code}
> As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
> To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
> {code}
> # ./testmem default
> {code}
> Note (again, using top, pidstat, etc.) that the memory used is constant.
> It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Closed: (THRIFT-391) Memory leak in fastbinary

Posted by "Kevin Clark (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/THRIFT-391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kevin Clark closed THRIFT-391.
------------------------------

    Resolution: Fixed
      Assignee: Alexander Shigin

Pushed in 757634.

> Memory leak in fastbinary
> -------------------------
>
>                 Key: THRIFT-391
>                 URL: https://issues.apache.org/jira/browse/THRIFT-391
>             Project: Thrift
>          Issue Type: Bug
>          Components: Library (Python)
>         Environment: Linux 2.6.21 (Debian)
>            Reporter: Garrett Smith
>            Assignee: Alexander Shigin
>             Fix For: 0.1
>
>         Attachments: testmem.tar.gz, thrift-fastbinary-leak.patch
>
>
> When a struct has a map member, fastbinary.binary_encode leaks memory.
> Use testmem.tar.gz app to recreate the problem.
> {code}
> # tar xzf testmem.tar.gz
> # cd testmem
> # ./testmem fast
> {code}
> As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
> To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
> {code}
> # ./testmem default
> {code}
> Note (again, using top, pidstat, etc.) that the memory used is constant.
> It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (THRIFT-391) Memory leak in fastbinary

Posted by "Jérémie BORDIER (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/THRIFT-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688119#action_12688119 ] 

Jérémie BORDIER commented on THRIFT-391:
----------------------------------------

I'm not a python c-api pro, but the patch looks obvious and correct. I think the Py_INCREF / DECREF should be kept here to avoid unsafe garbage collection in multithreaded environment. So, lgtm, +1.

> Memory leak in fastbinary
> -------------------------
>
>                 Key: THRIFT-391
>                 URL: https://issues.apache.org/jira/browse/THRIFT-391
>             Project: Thrift
>          Issue Type: Bug
>          Components: Library (Python)
>         Environment: Linux 2.6.21 (Debian)
>            Reporter: Garrett Smith
>             Fix For: 0.1
>
>         Attachments: testmem.tar.gz, thrift-fastbinary-leak.patch
>
>
> When a struct has a map member, fastbinary.binary_encode leaks memory.
> Use testmem.tar.gz app to recreate the problem.
> {code}
> # tar xzf testmem.tar.gz
> # cd testmem
> # ./testmem fast
> {code}
> As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
> To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
> {code}
> # ./testmem default
> {code}
> Note (again, using top, pidstat, etc.) that the memory used is constant.
> It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (THRIFT-391) Memory leak in fastbinary

Posted by "Garrett Smith (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/THRIFT-391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Garrett Smith updated THRIFT-391:
---------------------------------

    Attachment: testmem.tar.gz

Refer to issue description for instructions on using this file.

> Memory leak in fastbinary
> -------------------------
>
>                 Key: THRIFT-391
>                 URL: https://issues.apache.org/jira/browse/THRIFT-391
>             Project: Thrift
>          Issue Type: Bug
>          Components: Compiler (Python)
>         Environment: Linux 2.6.21 (Debian)
>            Reporter: Garrett Smith
>         Attachments: testmem.tar.gz
>
>
> When a struct has a map member, fastbinary.binary_encode leaks memory.
> Use testmem.tar.gz app to recreate the problem.
> {code}
> # tar xzf testmem.tar.gz
> # cd testmem
> # ./testmem fast
> {code}
> As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
> To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
> {code}
> # ./testmem default
> {code}
> Note (again, using top, pidstat, etc.) that the memory used is constant.
> It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (THRIFT-391) Memory leak in fastbinary

Posted by "Kevin Clark (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/THRIFT-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688378#action_12688378 ] 

Kevin Clark commented on THRIFT-391:
------------------------------------

Ok, if there's no objection today, I'll push the patch.

> Memory leak in fastbinary
> -------------------------
>
>                 Key: THRIFT-391
>                 URL: https://issues.apache.org/jira/browse/THRIFT-391
>             Project: Thrift
>          Issue Type: Bug
>          Components: Library (Python)
>         Environment: Linux 2.6.21 (Debian)
>            Reporter: Garrett Smith
>             Fix For: 0.1
>
>         Attachments: testmem.tar.gz, thrift-fastbinary-leak.patch
>
>
> When a struct has a map member, fastbinary.binary_encode leaks memory.
> Use testmem.tar.gz app to recreate the problem.
> {code}
> # tar xzf testmem.tar.gz
> # cd testmem
> # ./testmem fast
> {code}
> As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
> To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
> {code}
> # ./testmem default
> {code}
> Note (again, using top, pidstat, etc.) that the memory used is constant.
> It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (THRIFT-391) Memory leak in fastbinary

Posted by "Garrett Smith (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/THRIFT-391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Garrett Smith updated THRIFT-391:
---------------------------------

    Description: 
When a struct has a map member, fastbinary.binary_encode leaks memory.

Use testmem.tar.gz app to recreate the problem.

{code}
# tar xzf testmem.tar.gz
# cd testmem
# ./testmem fast
{code}

As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.

To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:

{code}
# ./testmem default
{code}

Note (again, using top, pidstat, etc.) that the memory used is constant.

It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.

  was:
When a struct has a map member, fastbinary.binary_encode leaks memory.

Use testmem.tar.gz app to recreate the problem.

{quote}
# tar xzf testmem.tar.gz
# cd testmem
# ./testmem fast
{quote}

As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.

To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:

{quote}
# ./testmem default
{quote}

Note (again, using top, pidstat, etc.) that the memory used is constant.

It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.


> Memory leak in fastbinary
> -------------------------
>
>                 Key: THRIFT-391
>                 URL: https://issues.apache.org/jira/browse/THRIFT-391
>             Project: Thrift
>          Issue Type: Bug
>          Components: Compiler (Python)
>         Environment: Linux 2.6.21 (Debian)
>            Reporter: Garrett Smith
>
> When a struct has a map member, fastbinary.binary_encode leaks memory.
> Use testmem.tar.gz app to recreate the problem.
> {code}
> # tar xzf testmem.tar.gz
> # cd testmem
> # ./testmem fast
> {code}
> As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
> To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
> {code}
> # ./testmem default
> {code}
> Note (again, using top, pidstat, etc.) that the memory used is constant.
> It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.