You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Ashutosh Chauhan (JIRA)" <ji...@apache.org> on 2010/06/25 22:38:49 UTC
[jira] Created: (PIG-1466) Improve log messages for memory usage
Improve log messages for memory usage
-------------------------------------
Key: PIG-1466
URL: https://issues.apache.org/jira/browse/PIG-1466
Project: Pig
Issue Type: Improvement
Components: impl
Affects Versions: 0.7.0
Reporter: Ashutosh Chauhan
Priority: Minor
For anything more then a moderately sized dataset Pig usually spits following messages:
{code}
2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage
threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K) max =
954466304(932096K)
2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection
threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K) max =
954466304(932096K)
{code}
This seems to confuse users a lot. Once these messages are printed, users tend to believe that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be cruising along at ease. We should be little more careful what to print in logs. Currently these are printed when a notification is sent by JVM and some other conditions are met which may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness. At the every least, we should lower the log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1466) Improve log messages for memory usage
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898296#action_12898296 ]
Thejas M Nair commented on PIG-1466:
------------------------------------
bq. It would also be nice to know when GC is called but we can make message to reflect that
Olga, Are you suggesting that we should log everytime the memory manager handler is called or when the memory manager invokes GC after spilling enough memory ?
I am not sure if it is useful to log every call to the memory manager handler, maybe we can log the first time for each type of threshold has been exceeded and then every time we actually spill something to disk.
> Improve log messages for memory usage
> -------------------------------------
>
> Key: PIG-1466
> URL: https://issues.apache.org/jira/browse/PIG-1466
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Thejas M Nair
> Priority: Minor
> Fix For: 0.8.0
>
>
> For anything more then a moderately sized dataset Pig usually spits following messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users tend to believe that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be cruising along at ease. We should be little more careful what to print in logs. Currently these are printed when a notification is sent by JVM and some other conditions are met which may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness. At the every least, we should lower the log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1466) Improve log messages for memory usage
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899489#action_12899489 ]
Olga Natkovich commented on PIG-1466:
-------------------------------------
Thejas, your proposal looks good
> Improve log messages for memory usage
> -------------------------------------
>
> Key: PIG-1466
> URL: https://issues.apache.org/jira/browse/PIG-1466
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Thejas M Nair
> Priority: Minor
> Fix For: 0.8.0
>
>
> For anything more then a moderately sized dataset Pig usually spits following messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users tend to believe that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be cruising along at ease. We should be little more careful what to print in logs. Currently these are printed when a notification is sent by JVM and some other conditions are met which may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness. At the every least, we should lower the log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1466) Improve log messages for memory usage
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900379#action_12900379 ]
Olga Natkovich commented on PIG-1466:
-------------------------------------
I will review this patch today
> Improve log messages for memory usage
> -------------------------------------
>
> Key: PIG-1466
> URL: https://issues.apache.org/jira/browse/PIG-1466
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Thejas M Nair
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: PIG-1466.patch
>
>
> For anything more then a moderately sized dataset Pig usually spits following messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users tend to believe that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be cruising along at ease. We should be little more careful what to print in logs. Currently these are printed when a notification is sent by JVM and some other conditions are met which may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness. At the every least, we should lower the log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1466) Improve log messages for memory usage
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900424#action_12900424 ]
Thejas M Nair commented on PIG-1466:
------------------------------------
bq. "memory handler call- Usage threshold exceeded " and "memory handler call - Collection threshold exceeded " need to be made more neutral so that users do not think it is a problem. Also, I think we want to log this at info level so that we get it by default.
I will change 'exceeded' part . I log the message at info level the first time handler called for each threshold type, in subsequent calls to the handler messages are logged only if some object has been spilled to disk.
> Improve log messages for memory usage
> -------------------------------------
>
> Key: PIG-1466
> URL: https://issues.apache.org/jira/browse/PIG-1466
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Thejas M Nair
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: PIG-1466.patch
>
>
> For anything more then a moderately sized dataset Pig usually spits following messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users tend to believe that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be cruising along at ease. We should be little more careful what to print in logs. Currently these are printed when a notification is sent by JVM and some other conditions are met which may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness. At the every least, we should lower the log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1466) Improve log messages for memory usage
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900413#action_12900413 ]
Olga Natkovich commented on PIG-1466:
-------------------------------------
Patch looks good. Just one comment:
"memory handler call- Usage threshold exceeded " and "memory handler call - Collection threshold exceeded " need to be made more neutral so that users do not think it is a problem. Also, I think we want to log this at info level so that we get it by default.
> Improve log messages for memory usage
> -------------------------------------
>
> Key: PIG-1466
> URL: https://issues.apache.org/jira/browse/PIG-1466
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Thejas M Nair
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: PIG-1466.patch
>
>
> For anything more then a moderately sized dataset Pig usually spits following messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users tend to believe that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be cruising along at ease. We should be little more careful what to print in logs. Currently these are printed when a notification is sent by JVM and some other conditions are met which may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness. At the every least, we should lower the log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-1466) Improve log messages for memory usage
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich reassigned PIG-1466:
-----------------------------------
Assignee: Thejas M Nair
Thejas, can you update the messages since you are already looking at the memory stuff, thanks
> Improve log messages for memory usage
> -------------------------------------
>
> Key: PIG-1466
> URL: https://issues.apache.org/jira/browse/PIG-1466
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Thejas M Nair
> Priority: Minor
> Fix For: 0.8.0
>
>
> For anything more then a moderately sized dataset Pig usually spits following messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users tend to believe that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be cruising along at ease. We should be little more careful what to print in logs. Currently these are printed when a notification is sent by JVM and some other conditions are met which may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness. At the every least, we should lower the log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1466) Improve log messages for memory usage
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898020#action_12898020 ]
Thejas M Nair commented on PIG-1466:
------------------------------------
I am planning to change as following -
1. Change the string "low memory handler called" to "cleaner notified" .
2. Print additional log message stating total spill object memory freed and total number of objects freed. This will be printed the first time a candidate to be freed is found, and then every 10 times the GC is called from the code. (GC is called after a threshold of memory suitable to be freed has been collected.)
> Improve log messages for memory usage
> -------------------------------------
>
> Key: PIG-1466
> URL: https://issues.apache.org/jira/browse/PIG-1466
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Thejas M Nair
> Priority: Minor
> Fix For: 0.8.0
>
>
> For anything more then a moderately sized dataset Pig usually spits following messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users tend to believe that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be cruising along at ease. We should be little more careful what to print in logs. Currently these are printed when a notification is sent by JVM and some other conditions are met which may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness. At the every least, we should lower the log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1466) Improve log messages for memory usage
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated PIG-1466:
-------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
Patch committed to trunk.
> Improve log messages for memory usage
> -------------------------------------
>
> Key: PIG-1466
> URL: https://issues.apache.org/jira/browse/PIG-1466
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Thejas M Nair
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: PIG-1466.patch
>
>
> For anything more then a moderately sized dataset Pig usually spits following messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users tend to believe that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be cruising along at ease. We should be little more careful what to print in logs. Currently these are printed when a notification is sent by JVM and some other conditions are met which may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness. At the every least, we should lower the log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1466) Improve log messages for memory usage
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated PIG-1466:
-------------------------------
Attachment: PIG-1466.patch
With the changes in the patch, the logs from SpillableMemoryManager will look like this -
2010-08-18 18:22:44,365 INFO org.apache.pig.impl.util.SpillableMemoryManager: first memory handler call- Usage threshold exceeded init = 5439488(5312K) used = 657702912(642288K) committed = 715849728(699072K) max = 715849728(699072K)
2010-08-18 18:22:44,728 INFO org.apache.pig.impl.util.SpillableMemoryManager: first memory handler call - Collection threshold exceeded init = 5439488(5312K) used = 658437416(643005K) committed = 715849728(699072K) max = 715849728(699072K)
2010-08-18 18:27:51,174 INFO org.apache.pig.impl.util.SpillableMemoryManager: Spilled an estimate of 73675890 bytes from 1 objects. init = 5439488(5312K) used = 511215656(499234K) committed = 659881984(644416K) max = 715849728(699072K)
2010-08-18 18:29:16,739 INFO org.apache.pig.impl.util.SpillableMemoryManager: Spilled an estimate of 418063724 bytes from 1 objects. init = 5439488(5312K) used = 563165736(549966K) committed = 659881984(644416K) max = 715849728(699072K)
2010-08-18 18:29:22,274 INFO org.apache.pig.impl.util.SpillableMemoryManager: Spilled an estimate of 443888012 bytes from 1 objects. init = 5439488(5312K) used = 553174984(540209K) committed = 715849728(699072K) max = 715849728(699072K)
> Improve log messages for memory usage
> -------------------------------------
>
> Key: PIG-1466
> URL: https://issues.apache.org/jira/browse/PIG-1466
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Thejas M Nair
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: PIG-1466.patch
>
>
> For anything more then a moderately sized dataset Pig usually spits following messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users tend to believe that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be cruising along at ease. We should be little more careful what to print in logs. Currently these are printed when a notification is sent by JVM and some other conditions are met which may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness. At the every least, we should lower the log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1466) Improve log messages for memory usage
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882731#action_12882731 ]
Alan Gates commented on PIG-1466:
---------------------------------
Rather than change the log level can we change it to only print when we truly spill a {{DefaultBag}}? It would be nice to know if there are any cases where we are still doing that.
> Improve log messages for memory usage
> -------------------------------------
>
> Key: PIG-1466
> URL: https://issues.apache.org/jira/browse/PIG-1466
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Priority: Minor
>
> For anything more then a moderately sized dataset Pig usually spits following messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users tend to believe that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be cruising along at ease. We should be little more careful what to print in logs. Currently these are printed when a notification is sent by JVM and some other conditions are met which may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness. At the every least, we should lower the log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1466) Improve log messages for memory usage
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882743#action_12882743 ]
Olga Natkovich commented on PIG-1466:
-------------------------------------
It would also be nice to know when GC is called but we can make message to reflect that
> Improve log messages for memory usage
> -------------------------------------
>
> Key: PIG-1466
> URL: https://issues.apache.org/jira/browse/PIG-1466
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Priority: Minor
> Fix For: 0.8.0
>
>
> For anything more then a moderately sized dataset Pig usually spits following messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users tend to believe that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be cruising along at ease. We should be little more careful what to print in logs. Currently these are printed when a notification is sent by JVM and some other conditions are met which may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness. At the every least, we should lower the log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1466) Improve log messages for memory usage
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated PIG-1466:
-------------------------------
Status: Patch Available (was: Open)
test-patch results. No new test cases are included as this changes only the log messages.
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec]
[exec]
[exec]
[exec] ======================================================================
[exec] ======================================================================
[exec] Finished build.
[exec] ======================================================================
[exec] ======================================================================
> Improve log messages for memory usage
> -------------------------------------
>
> Key: PIG-1466
> URL: https://issues.apache.org/jira/browse/PIG-1466
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Thejas M Nair
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: PIG-1466.patch
>
>
> For anything more then a moderately sized dataset Pig usually spits following messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users tend to believe that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be cruising along at ease. We should be little more careful what to print in logs. Currently these are printed when a notification is sent by JVM and some other conditions are met which may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness. At the every least, we should lower the log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1466) Improve log messages for memory usage
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900375#action_12900375 ]
Thejas M Nair commented on PIG-1466:
------------------------------------
core and contrib test passed when i run them manually. Patch is ready for review.
> Improve log messages for memory usage
> -------------------------------------
>
> Key: PIG-1466
> URL: https://issues.apache.org/jira/browse/PIG-1466
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Thejas M Nair
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: PIG-1466.patch
>
>
> For anything more then a moderately sized dataset Pig usually spits following messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users tend to believe that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be cruising along at ease. We should be little more careful what to print in logs. Currently these are printed when a notification is sent by JVM and some other conditions are met which may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness. At the every least, we should lower the log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1466) Improve log messages for memory usage
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-1466:
--------------------------------
Fix Version/s: 0.8.0
> Improve log messages for memory usage
> -------------------------------------
>
> Key: PIG-1466
> URL: https://issues.apache.org/jira/browse/PIG-1466
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Priority: Minor
> Fix For: 0.8.0
>
>
> For anything more then a moderately sized dataset Pig usually spits following messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users tend to believe that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be cruising along at ease. We should be little more careful what to print in logs. Currently these are printed when a notification is sent by JVM and some other conditions are met which may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness. At the every least, we should lower the log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.