You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Haitao Yao (JIRA)" <ji...@apache.org> on 2012/09/13 05:27:07 UTC
[jira] [Created] (PIG-2917) SpillableMemoryManager memory leak for
WeakReference
Haitao Yao created PIG-2917:
-------------------------------
Summary: SpillableMemoryManager memory leak for WeakReference
Key: PIG-2917
URL: https://issues.apache.org/jira/browse/PIG-2917
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.11
Reporter: Haitao Yao
Sine the SpillableMemoryManager subscribed the memory notifications, and hold a weak reference of all the spillable, if there's too many weak reference and the memory exceeds the threshold, the current code will not clear the spillable weak references.
the details is illustrated in the attached screenshot of a MemoryAnalyzer.
So we need to clear the spillables when the memory notifications is received.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2917) SpillableMemoryManager memory leak for
WeakReference
Posted by "Haitao Yao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Haitao Yao updated PIG-2917:
----------------------------
Attachment: SpillableMemoryManager.patch
here's the patch
> SpillableMemoryManager memory leak for WeakReference
> ----------------------------------------------------
>
> Key: PIG-2917
> URL: https://issues.apache.org/jira/browse/PIG-2917
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.11
> Reporter: Haitao Yao
> Attachments: SpillableMemoryManager.patch
>
>
> Sine the SpillableMemoryManager subscribed the memory notifications, and hold a weak reference of all the spillable, if there's too many weak reference and the memory exceeds the threshold, the current code will not clear the spillable weak references.
> the details is illustrated in the attached screenshot of a MemoryAnalyzer.
> So we need to clear the spillables when the memory notifications is received.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2917) SpillableMemoryManager memory leak for
WeakReference
Posted by "Haitao Yao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Haitao Yao updated PIG-2917:
----------------------------
Attachment: aa.jpg
here's the screenshots of MemoryAnalyzer.
> SpillableMemoryManager memory leak for WeakReference
> ----------------------------------------------------
>
> Key: PIG-2917
> URL: https://issues.apache.org/jira/browse/PIG-2917
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.11
> Reporter: Haitao Yao
> Attachments: aa.jpg, SpillableMemoryManager.patch
>
>
> Sine the SpillableMemoryManager subscribed the memory notifications, and hold a weak reference of all the spillable, if there's too many weak reference and the memory exceeds the threshold, the current code will not clear the spillable weak references.
> the details is illustrated in the attached screenshot of a MemoryAnalyzer.
> So we need to clear the spillables when the memory notifications is received.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2917) SpillableMemoryManager memory leak
for WeakReference
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454671#comment-13454671 ]
Dmitriy V. Ryaboy commented on PIG-2917:
----------------------------------------
Haitao, I am curious what your script is doing that causes it to allocate that many bags?
I just created a ticket ripping out some unnecessary usage of Spillable objects; suspect your usage is pointing us at another place where we don't really need to be dealing with SMM in the first place.
Patch seems reasonable. Stylistically, I prefer the code below for iteration of Iterables:
{code}
for (WeakReference<Spillable> ref : spillables) {
...
}
{code}
> SpillableMemoryManager memory leak for WeakReference
> ----------------------------------------------------
>
> Key: PIG-2917
> URL: https://issues.apache.org/jira/browse/PIG-2917
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.11
> Reporter: Haitao Yao
> Attachments: aa.jpg, SpillableMemoryManager.patch
>
>
> Sine the SpillableMemoryManager subscribed the memory notifications, and hold a weak reference of all the spillable, if there's too many weak reference and the memory exceeds the threshold, the current code will not clear the spillable weak references.
> the details is illustrated in the attached screenshot of a MemoryAnalyzer.
> So we need to clear the spillables when the memory notifications is received.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (PIG-2917) SpillableMemoryManager memory leak for
WeakReference
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy reassigned PIG-2917:
--------------------------------------
Assignee: Haitao Yao
> SpillableMemoryManager memory leak for WeakReference
> ----------------------------------------------------
>
> Key: PIG-2917
> URL: https://issues.apache.org/jira/browse/PIG-2917
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.11
> Reporter: Haitao Yao
> Assignee: Haitao Yao
> Attachments: aa.jpg, SpillableMemoryManager.patch
>
>
> Sine the SpillableMemoryManager subscribed the memory notifications, and hold a weak reference of all the spillable, if there's too many weak reference and the memory exceeds the threshold, the current code will not clear the spillable weak references.
> the details is illustrated in the attached screenshot of a MemoryAnalyzer.
> So we need to clear the spillables when the memory notifications is received.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2917) SpillableMemoryManager memory leak for
WeakReference
Posted by "Haitao Yao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Haitao Yao updated PIG-2917:
----------------------------
Patch Info: Patch Available
> SpillableMemoryManager memory leak for WeakReference
> ----------------------------------------------------
>
> Key: PIG-2917
> URL: https://issues.apache.org/jira/browse/PIG-2917
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.11
> Reporter: Haitao Yao
>
> Sine the SpillableMemoryManager subscribed the memory notifications, and hold a weak reference of all the spillable, if there's too many weak reference and the memory exceeds the threshold, the current code will not clear the spillable weak references.
> the details is illustrated in the attached screenshot of a MemoryAnalyzer.
> So we need to clear the spillables when the memory notifications is received.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-2917) SpillableMemoryManager memory leak for
WeakReference
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy resolved PIG-2917.
------------------------------------
Resolution: Fixed
Fix Version/s: 0.11
patch committed.
thanks Haitao.
> SpillableMemoryManager memory leak for WeakReference
> ----------------------------------------------------
>
> Key: PIG-2917
> URL: https://issues.apache.org/jira/browse/PIG-2917
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.11
> Reporter: Haitao Yao
> Assignee: Haitao Yao
> Fix For: 0.11
>
> Attachments: aa.jpg, SpillableMemoryManager.patch
>
>
> Sine the SpillableMemoryManager subscribed the memory notifications, and hold a weak reference of all the spillable, if there's too many weak reference and the memory exceeds the threshold, the current code will not clear the spillable weak references.
> the details is illustrated in the attached screenshot of a MemoryAnalyzer.
> So we need to clear the spillables when the memory notifications is received.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira