You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Garrett Smith (JIRA)" <ji...@apache.org> on 2009/03/21 21:48:50 UTC
[jira] Created: (THRIFT-391) Memory leak in fastbinary
Memory leak in fastbinary
-------------------------
Key: THRIFT-391
URL: https://issues.apache.org/jira/browse/THRIFT-391
Project: Thrift
Issue Type: Bug
Components: Compiler (Python)
Environment: Linux 2.6.21 (Debian)
Reporter: Garrett Smith
When a struct has a map member, fastbinary.binary_encode leaks memory.
Use testmem.tar.gz app to recreate the problem.
{quote}
# tar xzf testmem.tar.gz
# cd testmem
# ./testmem fast
{quote}
As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
{quote}
# ./testmem default
{quote}
Note (again, using top, pidstat, etc.) that the memory used is constant.
It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-391) Memory leak in fastbinary
Posted by "Alexander Shigin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Shigin updated THRIFT-391:
------------------------------------
Attachment: thrift-fastbinary-leak.patch
It seems the patch solves the problem.
Can anyone review it? David, I think we can solve the problem by eliminating Py_INCREF (there is your TODO), but i am not guru of python c-api.
> Memory leak in fastbinary
> -------------------------
>
> Key: THRIFT-391
> URL: https://issues.apache.org/jira/browse/THRIFT-391
> Project: Thrift
> Issue Type: Bug
> Components: Library (Python)
> Environment: Linux 2.6.21 (Debian)
> Reporter: Garrett Smith
> Fix For: 0.1
>
> Attachments: testmem.tar.gz, thrift-fastbinary-leak.patch
>
>
> When a struct has a map member, fastbinary.binary_encode leaks memory.
> Use testmem.tar.gz app to recreate the problem.
> {code}
> # tar xzf testmem.tar.gz
> # cd testmem
> # ./testmem fast
> {code}
> As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
> To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
> {code}
> # ./testmem default
> {code}
> Note (again, using top, pidstat, etc.) that the memory used is constant.
> It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-391) Memory leak in fastbinary
Posted by "Alexander Shigin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Shigin updated THRIFT-391:
------------------------------------
Component/s: (was: Compiler (Python))
Library (Python)
Patch Info: [Patch Available]
Fix Version/s: 0.1
> Memory leak in fastbinary
> -------------------------
>
> Key: THRIFT-391
> URL: https://issues.apache.org/jira/browse/THRIFT-391
> Project: Thrift
> Issue Type: Bug
> Components: Library (Python)
> Environment: Linux 2.6.21 (Debian)
> Reporter: Garrett Smith
> Fix For: 0.1
>
> Attachments: testmem.tar.gz
>
>
> When a struct has a map member, fastbinary.binary_encode leaks memory.
> Use testmem.tar.gz app to recreate the problem.
> {code}
> # tar xzf testmem.tar.gz
> # cd testmem
> # ./testmem fast
> {code}
> As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
> To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
> {code}
> # ./testmem default
> {code}
> Note (again, using top, pidstat, etc.) that the memory used is constant.
> It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (THRIFT-391) Memory leak in fastbinary
Posted by "Kevin Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Clark closed THRIFT-391.
------------------------------
Resolution: Fixed
Assignee: Alexander Shigin
Pushed in 757634.
> Memory leak in fastbinary
> -------------------------
>
> Key: THRIFT-391
> URL: https://issues.apache.org/jira/browse/THRIFT-391
> Project: Thrift
> Issue Type: Bug
> Components: Library (Python)
> Environment: Linux 2.6.21 (Debian)
> Reporter: Garrett Smith
> Assignee: Alexander Shigin
> Fix For: 0.1
>
> Attachments: testmem.tar.gz, thrift-fastbinary-leak.patch
>
>
> When a struct has a map member, fastbinary.binary_encode leaks memory.
> Use testmem.tar.gz app to recreate the problem.
> {code}
> # tar xzf testmem.tar.gz
> # cd testmem
> # ./testmem fast
> {code}
> As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
> To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
> {code}
> # ./testmem default
> {code}
> Note (again, using top, pidstat, etc.) that the memory used is constant.
> It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-391) Memory leak in fastbinary
Posted by "Jérémie BORDIER (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688119#action_12688119 ]
Jérémie BORDIER commented on THRIFT-391:
----------------------------------------
I'm not a python c-api pro, but the patch looks obvious and correct. I think the Py_INCREF / DECREF should be kept here to avoid unsafe garbage collection in multithreaded environment. So, lgtm, +1.
> Memory leak in fastbinary
> -------------------------
>
> Key: THRIFT-391
> URL: https://issues.apache.org/jira/browse/THRIFT-391
> Project: Thrift
> Issue Type: Bug
> Components: Library (Python)
> Environment: Linux 2.6.21 (Debian)
> Reporter: Garrett Smith
> Fix For: 0.1
>
> Attachments: testmem.tar.gz, thrift-fastbinary-leak.patch
>
>
> When a struct has a map member, fastbinary.binary_encode leaks memory.
> Use testmem.tar.gz app to recreate the problem.
> {code}
> # tar xzf testmem.tar.gz
> # cd testmem
> # ./testmem fast
> {code}
> As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
> To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
> {code}
> # ./testmem default
> {code}
> Note (again, using top, pidstat, etc.) that the memory used is constant.
> It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-391) Memory leak in fastbinary
Posted by "Garrett Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Garrett Smith updated THRIFT-391:
---------------------------------
Attachment: testmem.tar.gz
Refer to issue description for instructions on using this file.
> Memory leak in fastbinary
> -------------------------
>
> Key: THRIFT-391
> URL: https://issues.apache.org/jira/browse/THRIFT-391
> Project: Thrift
> Issue Type: Bug
> Components: Compiler (Python)
> Environment: Linux 2.6.21 (Debian)
> Reporter: Garrett Smith
> Attachments: testmem.tar.gz
>
>
> When a struct has a map member, fastbinary.binary_encode leaks memory.
> Use testmem.tar.gz app to recreate the problem.
> {code}
> # tar xzf testmem.tar.gz
> # cd testmem
> # ./testmem fast
> {code}
> As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
> To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
> {code}
> # ./testmem default
> {code}
> Note (again, using top, pidstat, etc.) that the memory used is constant.
> It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-391) Memory leak in fastbinary
Posted by "Kevin Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688378#action_12688378 ]
Kevin Clark commented on THRIFT-391:
------------------------------------
Ok, if there's no objection today, I'll push the patch.
> Memory leak in fastbinary
> -------------------------
>
> Key: THRIFT-391
> URL: https://issues.apache.org/jira/browse/THRIFT-391
> Project: Thrift
> Issue Type: Bug
> Components: Library (Python)
> Environment: Linux 2.6.21 (Debian)
> Reporter: Garrett Smith
> Fix For: 0.1
>
> Attachments: testmem.tar.gz, thrift-fastbinary-leak.patch
>
>
> When a struct has a map member, fastbinary.binary_encode leaks memory.
> Use testmem.tar.gz app to recreate the problem.
> {code}
> # tar xzf testmem.tar.gz
> # cd testmem
> # ./testmem fast
> {code}
> As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
> To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
> {code}
> # ./testmem default
> {code}
> Note (again, using top, pidstat, etc.) that the memory used is constant.
> It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-391) Memory leak in fastbinary
Posted by "Garrett Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Garrett Smith updated THRIFT-391:
---------------------------------
Description:
When a struct has a map member, fastbinary.binary_encode leaks memory.
Use testmem.tar.gz app to recreate the problem.
{code}
# tar xzf testmem.tar.gz
# cd testmem
# ./testmem fast
{code}
As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
{code}
# ./testmem default
{code}
Note (again, using top, pidstat, etc.) that the memory used is constant.
It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.
was:
When a struct has a map member, fastbinary.binary_encode leaks memory.
Use testmem.tar.gz app to recreate the problem.
{quote}
# tar xzf testmem.tar.gz
# cd testmem
# ./testmem fast
{quote}
As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
{quote}
# ./testmem default
{quote}
Note (again, using top, pidstat, etc.) that the memory used is constant.
It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.
> Memory leak in fastbinary
> -------------------------
>
> Key: THRIFT-391
> URL: https://issues.apache.org/jira/browse/THRIFT-391
> Project: Thrift
> Issue Type: Bug
> Components: Compiler (Python)
> Environment: Linux 2.6.21 (Debian)
> Reporter: Garrett Smith
>
> When a struct has a map member, fastbinary.binary_encode leaks memory.
> Use testmem.tar.gz app to recreate the problem.
> {code}
> # tar xzf testmem.tar.gz
> # cd testmem
> # ./testmem fast
> {code}
> As this is running, use top or pidstat to view the memory consumed by the process. Hit CTRL-C to stop. On my system it steadily increases.
> To compare the results using the 'default' encoding technique (using pure Python TBinaryProtocol), run testmem as follows:
> {code}
> # ./testmem default
> {code}
> Note (again, using top, pidstat, etc.) that the memory used is constant.
> It appears that the 'map' member triggers this. E.g. string, int members do not cause the leak.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.