You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by ahgittin <gi...@git.apache.org> on 2016/11/07 11:31:17 UTC
[GitHub] brooklyn-docs pull request #122: info on debugging memory usage
GitHub user ahgittin opened a pull request:
https://github.com/apache/brooklyn-docs/pull/122
info on debugging memory usage
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ahgittin/brooklyn-docs memory
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/brooklyn-docs/pull/122.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #122
----
commit 8f04abc046103dd117945329e1e2b6e6b6bf3ab0
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2016-11-07T11:26:59Z
info on debugging memory usage
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #122: info on debugging memory usage
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/122#discussion_r87766901
--- Diff: guide/ops/troubleshooting/memory-usage.md ---
@@ -0,0 +1,138 @@
+---
+layout: website-normal
+title: "Troubleshooting: Monitoring Memory Usage"
+toc: /guide/toc.json
+---
+
+## Memory Usage
+
+Brooklyn tries to keep in memory as much history of its activity as possible,
+for displaying through the UI, so it is normal for it to consume as much memory
+as it can. It uses "soft references" so these objects will be cleared if needed,
+but **it is not a sign of anything unusual if Brooklyn is using all its available memory**.
+
+The number of active tasks, CPU usage, thread counts, and
+retention of soft reference objects are a much better indication of load.
+This information can be found by looking in the log for lines containing
+`brooklyn gc`, such as:
+
+ 2016-09-16 16:19:43,337 DEBUG o.a.b.c.m.i.BrooklynGarbageCollector [brooklyn-gc]: brooklyn gc (before) - using 910 MB / 3.76 GB memory; 98% soft-reference maybe retention (of 362); 35 threads; tasks: 0 active, 2 unfinished; 31 remembered, 1013 total submitted)
+
+The soft-reference figure is indicative, but the lower this is, the more
+the JVM has decided to get rid of items that were desired to be kept but optional.
+It only tracks some soft-references (those wrapped in `Maybe`),
+and of course if there are many many such items the JVM will have to get rid
+of some, so a lower figure does not necessarily mean a problem.
+Typically however if there's no `OutOfMemoryError` (OOME) reported,
+there's no problem.
+
+
+## Problem Indicators and Resolutions
+
+Two things that *do* normally indicate a problem with memory are:
+
+* `OutOfMemoryError` exceptions being thrown
+* Memory usage high *and* CPU high, where the CPU is spent doing full garbage collection
+
+One possible cause is the JVM doing a poorly-selected GC strategy,
+as described in [Oracle Java bug 6912889](http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6912889).
+This can be confirmed by running the "analyzing soft reference usage" technique below;
+memory should shrink dramatically then increase until the problem recurs.
+This can be fixed by passing `-XX:SoftRefLRUPolicyMSPerMB=1` to the JVM,
+as described in [Brooklyn issue 375](https://issues.apache.org/jira/browse/BROOKLYN-375).
+
+Other common JVM options include `-Xms256m -Xmx1g -XX:MaxPermSize=256m`
+(depending on JVM provider and version) to set the right balance of memory allocation.
+In some cases a larger `-Xmx` value may simply be the fix
+(but this should not be the case unless many or large blueprints are being used).
+
+If the problem is not with soft references but with real memory usage,
+the culprit is likely a memory leak, typically in blueprint design.
+An early warning of this situation is the "soft-reference maybe retention" level decreasing.
+In these situations, follow the steps as described below for "Investigating Leaks".
+
+
+## Analyzing Soft Reference Usage
+
+If you are concerned about memory usage, or doing evaluation on test environments,
+the following method (in the Groovy console) can be invoked to force the system to
+reclaim as much memory as possible, including *all* soft references:
+
+ org.apache.brooklyn.util.javalang.MemoryUsageTracker.forceClearSoftReferences()
+
+In good situations, memory usage should return to a small level.
+This call can be disruptive to the system however so use with care.
+
+The above method can also be configured to run automatically when memory usage
+is detected to hit a certain level. That can be useful if external policies are
+being used to warn on high memory usage, and you want to keep some headroom.
+Many JVM authorities discourage interfering with its garbage collector, however,
+so use with care and study the particular JVM you are using.
+See the class `BrooklynGarbageCollector` for more information.
+
+
+## Investigating Leaks
+
+If a memory leak is found, the first place to look should be the WARN/ERROR logs.
+Many common causes of leaks, including as runaway tasks and cyclic dependent configuration,
+will show their own log errors prior to the memory error.
+
+You should also note the task counts in the `brooklyn gc` messages described above,
+and if there are an exceptional number of tasks or tasks are not clearing,
+other log messages will describe what is happening, and the in-product task
+view can indicate issues.
+
+Sometimes slow leaks can occur if blueprints do not clean up entities or locations.
+These can be diagnosed by noting the number of files written to the persistence location,
+if persistence is being used. Deploying then destroying a blueprint should not leave
+anything behind in the persistence directory.
+
+Where problems have been encountered in the past, we have resolved them and/or
+worked to improve logging and early identification.
+Please report any issues so that we can improve this further.
+In many cases we can also give advice on what other log `grep` patterns can be useful.
+
+
+### Standard Java Techniques
+
+Useful standard Java techniques for tracking memory leaks include:
+
+* `jstack <pid>` to see what tasks are running
+* `jmap -histo:live <pid>` to see what objects are using memory (see below)
+* Memory profilers such as VisualVM or Eclipse MAT, either connected to a running system or
+ against a heap dump generated on an OOME
+
+More information is available on [the Oracle Java web site](https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/memleaks.html).
+
+Note that some of the above techniques will often include soft and weak references that are irrelevant
+to the problem (and will be cleared on an OOME). Objects that may be cached in that way include:
+
+* `BasicConfigKey` (used for the web server and many blueprints)
+* `DslComponent` and `*Task` (used for Brooklyn activities and dependent configuration)
+* `jclouds` items including `ImageImpl` (to cache data on cloud service providers)
+
+On the other hand any of the above may also indicate a leak.
+Taking snapshots after a `forceClearSoftReferences()` (above) invocation and comparing those
+is one technique to filter out noise. Another is to wait until there is an OOME
+and look just after, because that will clear all non-essential data from memory.
+(The `forceClearSoftReferences()` actually works by triggering an OOME, in as safe
+a way as possible.)
+
+If leaked items are found, a profiler will normally let you see their content
+and walk backwards along their references to find out why they are being retained.
+
+
+### Summary of Techniques
+
+The following sequence of techniques is a common approach to investigating and fixing memory issues:
+
+* Note the log lines about `brooklyn gc`, including memory and tasks
+* Do not assume high memory usage alone is an error, as soft reference caches are deliberate;
+ use `forceClearSoftReferences()` to clear these
--- End diff --
@ahgittin (cc @neykov) I thought we were not going to recommend using `forceClearSoftReferences()` this in any kind of production environment. Can we put in a caveat here about not using it in production. I'd be extremely caution about encouraging real users to call this until devs have been using it in anger themselves a lot.
With the use of the (much safer) `-XX:SoftRefLRUPolicyMSPerMB=1`, I'd expect the need for calling this would be greatly reduced.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs issue #122: info on debugging memory usage
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the issue:
https://github.com/apache/brooklyn-docs/pull/122
addressed comments & merging
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #122: info on debugging memory usage
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/brooklyn-docs/pull/122
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #122: info on debugging memory usage
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/122#discussion_r86764032
--- Diff: guide/ops/troubleshooting/memory-usage.md ---
@@ -0,0 +1,94 @@
+---
+layout: website-normal
+title: "Troubleshooting: Monitoring Memory Usage"
+toc: /guide/toc.json
+---
+
+## Memory Usage
+
+Brooklyn tries to keep in memory as much history of its activity as possible,
+for displaying through the UI, so it is normal for it to consume as much memory
+as it can. It uses "soft references" so these objects will be cleared if needed,
+but **it is not a sign of anything unusual if Brooklyn is using all its available memory**.
+
+The number of active tasks, CPU usage, thread counts, and
+retention of soft reference objects are a much better indication of load.
+This information can be found by looking in the log for lines containing
+`brooklyn gc`, such as:
+
+ 2016-09-16 16:19:43,337 DEBUG o.a.b.c.m.i.BrooklynGarbageCollector [brooklyn-gc]: brooklyn gc (before) - using 910 MB / 3.76 GB memory; 98% soft-reference maybe retention (of 362); 35 threads; tasks: 0 active, 2 unfinished; 31 remembered, 1013 total submitted)
+
+The soft-reference figure is indicative, but the lower this is, the more
+the JVM has decided to get rid of items that were desired to be kept but optional.
+It only tracks some soft-references (those wrapped in `Maybe`),
+and of course if there are many many such items the JVM will have to get rid
+of some, so a lower figure does not necessarily mean a problem.
+Typically however if there's no `OutOfMemoryError` (OOME) reported,
+there's no problem.
+
+If you are concerned about memory usage, or doing evaluation on test environments,
+the following method (in the Groovy console) can be invoked to force the system to
+reclaim as much memory as possible, including *all* soft references:
+
+ org.apache.brooklyn.util.javalang.MemoryUsageTracker.forceClearSoftReferences()
+
+If things are happy usage should return to a small level. This is quite disruptive
+to the system however so use with care.
+
+The above method can also be configured to run automatically when memory usage
--- End diff --
I'd go further and say that we discourage this for any production use-case (because it has not had expensive testing).
A better solution for production use-cases is to use`-XX:SoftRefLRUPolicyMSPerMB=1`. See the discussion in https://issues.apache.org/jira/browse/BROOKLYN-375.
We should also describe the symptoms of BROOKLYN-375 - that the CPU usage goes very high, with almost all of that time being taken up by garbage collection.
We need to do more on BROOKLYN-375 - e.g. implement the cache, to stop using soft references!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs issue #122: info on debugging memory usage
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on the issue:
https://github.com/apache/brooklyn-docs/pull/122
@ahgittin useful tips for the docs - thanks. A few comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #122: info on debugging memory usage
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/122#discussion_r86764334
--- Diff: guide/ops/troubleshooting/memory-usage.md ---
@@ -0,0 +1,94 @@
+---
+layout: website-normal
+title: "Troubleshooting: Monitoring Memory Usage"
+toc: /guide/toc.json
+---
+
+## Memory Usage
+
+Brooklyn tries to keep in memory as much history of its activity as possible,
+for displaying through the UI, so it is normal for it to consume as much memory
+as it can. It uses "soft references" so these objects will be cleared if needed,
+but **it is not a sign of anything unusual if Brooklyn is using all its available memory**.
+
+The number of active tasks, CPU usage, thread counts, and
+retention of soft reference objects are a much better indication of load.
+This information can be found by looking in the log for lines containing
+`brooklyn gc`, such as:
+
+ 2016-09-16 16:19:43,337 DEBUG o.a.b.c.m.i.BrooklynGarbageCollector [brooklyn-gc]: brooklyn gc (before) - using 910 MB / 3.76 GB memory; 98% soft-reference maybe retention (of 362); 35 threads; tasks: 0 active, 2 unfinished; 31 remembered, 1013 total submitted)
+
+The soft-reference figure is indicative, but the lower this is, the more
+the JVM has decided to get rid of items that were desired to be kept but optional.
+It only tracks some soft-references (those wrapped in `Maybe`),
+and of course if there are many many such items the JVM will have to get rid
+of some, so a lower figure does not necessarily mean a problem.
+Typically however if there's no `OutOfMemoryError` (OOME) reported,
+there's no problem.
+
+If you are concerned about memory usage, or doing evaluation on test environments,
+the following method (in the Groovy console) can be invoked to force the system to
+reclaim as much memory as possible, including *all* soft references:
+
+ org.apache.brooklyn.util.javalang.MemoryUsageTracker.forceClearSoftReferences()
+
+If things are happy usage should return to a small level. This is quite disruptive
+to the system however so use with care.
+
+The above method can also be configured to run automatically when memory usage
+is detected to hit a certain level. That can be useful if external policies are
+being used to warn on high memory usage, and you want to keep some headroom.
+Many JVM references discourage interfering with its garbage collector, however,
+so use with care and study the particular JVM you are using.
+See the class `BrooklynGarbageCollector` for more information.
+
+
+## Investigation of Memory Leaks
+
+Design problems of course can cause memory leaks, and due to the nature of the
+soft references these can be difficult to notice until they are advanced.
+If the "soft-reference maybe retention" starts to decrease, that can be
+an early warning.
+
+Common problems such as runaway tasks and cyclic dependent configuration will often
+show their own log errors, so also look for these if there is a performance or memory problem.
+
+You should also note the task counts in the `brooklyn gc` messages described above,
+and if there are an exceptional number of tasks or tasks are not clearing,
+other log messages will describe what is happening, and the in-product task
+view can indicate issues. `jstack` can also be useful if it is a task problem.
--- End diff --
Also worth recommending the use of `jmap -histo:live <pid>` to see how many live instances of each type there are.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #122: info on debugging memory usage
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/122#discussion_r87770697
--- Diff: guide/ops/troubleshooting/memory-usage.md ---
@@ -0,0 +1,138 @@
+---
+layout: website-normal
+title: "Troubleshooting: Monitoring Memory Usage"
+toc: /guide/toc.json
+---
+
+## Memory Usage
+
+Brooklyn tries to keep in memory as much history of its activity as possible,
+for displaying through the UI, so it is normal for it to consume as much memory
+as it can. It uses "soft references" so these objects will be cleared if needed,
+but **it is not a sign of anything unusual if Brooklyn is using all its available memory**.
+
+The number of active tasks, CPU usage, thread counts, and
+retention of soft reference objects are a much better indication of load.
+This information can be found by looking in the log for lines containing
+`brooklyn gc`, such as:
+
+ 2016-09-16 16:19:43,337 DEBUG o.a.b.c.m.i.BrooklynGarbageCollector [brooklyn-gc]: brooklyn gc (before) - using 910 MB / 3.76 GB memory; 98% soft-reference maybe retention (of 362); 35 threads; tasks: 0 active, 2 unfinished; 31 remembered, 1013 total submitted)
+
+The soft-reference figure is indicative, but the lower this is, the more
+the JVM has decided to get rid of items that were desired to be kept but optional.
+It only tracks some soft-references (those wrapped in `Maybe`),
+and of course if there are many many such items the JVM will have to get rid
+of some, so a lower figure does not necessarily mean a problem.
+Typically however if there's no `OutOfMemoryError` (OOME) reported,
+there's no problem.
+
+
+## Problem Indicators and Resolutions
+
+Two things that *do* normally indicate a problem with memory are:
+
+* `OutOfMemoryError` exceptions being thrown
+* Memory usage high *and* CPU high, where the CPU is spent doing full garbage collection
+
+One possible cause is the JVM doing a poorly-selected GC strategy,
+as described in [Oracle Java bug 6912889](http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6912889).
+This can be confirmed by running the "analyzing soft reference usage" technique below;
+memory should shrink dramatically then increase until the problem recurs.
+This can be fixed by passing `-XX:SoftRefLRUPolicyMSPerMB=1` to the JVM,
+as described in [Brooklyn issue 375](https://issues.apache.org/jira/browse/BROOKLYN-375).
+
+Other common JVM options include `-Xms256m -Xmx1g -XX:MaxPermSize=256m`
+(depending on JVM provider and version) to set the right balance of memory allocation.
+In some cases a larger `-Xmx` value may simply be the fix
+(but this should not be the case unless many or large blueprints are being used).
+
+If the problem is not with soft references but with real memory usage,
+the culprit is likely a memory leak, typically in blueprint design.
+An early warning of this situation is the "soft-reference maybe retention" level decreasing.
+In these situations, follow the steps as described below for "Investigating Leaks".
+
+
+## Analyzing Soft Reference Usage
+
+If you are concerned about memory usage, or doing evaluation on test environments,
+the following method (in the Groovy console) can be invoked to force the system to
+reclaim as much memory as possible, including *all* soft references:
+
+ org.apache.brooklyn.util.javalang.MemoryUsageTracker.forceClearSoftReferences()
+
+In good situations, memory usage should return to a small level.
+This call can be disruptive to the system however so use with care.
+
+The above method can also be configured to run automatically when memory usage
+is detected to hit a certain level. That can be useful if external policies are
+being used to warn on high memory usage, and you want to keep some headroom.
+Many JVM authorities discourage interfering with its garbage collector, however,
+so use with care and study the particular JVM you are using.
+See the class `BrooklynGarbageCollector` for more information.
+
+
+## Investigating Leaks
+
+If a memory leak is found, the first place to look should be the WARN/ERROR logs.
+Many common causes of leaks, including as runaway tasks and cyclic dependent configuration,
+will show their own log errors prior to the memory error.
+
+You should also note the task counts in the `brooklyn gc` messages described above,
+and if there are an exceptional number of tasks or tasks are not clearing,
+other log messages will describe what is happening, and the in-product task
+view can indicate issues.
+
+Sometimes slow leaks can occur if blueprints do not clean up entities or locations.
+These can be diagnosed by noting the number of files written to the persistence location,
+if persistence is being used. Deploying then destroying a blueprint should not leave
+anything behind in the persistence directory.
+
+Where problems have been encountered in the past, we have resolved them and/or
+worked to improve logging and early identification.
+Please report any issues so that we can improve this further.
+In many cases we can also give advice on what other log `grep` patterns can be useful.
+
+
+### Standard Java Techniques
+
+Useful standard Java techniques for tracking memory leaks include:
+
+* `jstack <pid>` to see what tasks are running
+* `jmap -histo:live <pid>` to see what objects are using memory (see below)
+* Memory profilers such as VisualVM or Eclipse MAT, either connected to a running system or
+ against a heap dump generated on an OOME
+
+More information is available on [the Oracle Java web site](https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/memleaks.html).
+
+Note that some of the above techniques will often include soft and weak references that are irrelevant
+to the problem (and will be cleared on an OOME). Objects that may be cached in that way include:
+
+* `BasicConfigKey` (used for the web server and many blueprints)
+* `DslComponent` and `*Task` (used for Brooklyn activities and dependent configuration)
+* `jclouds` items including `ImageImpl` (to cache data on cloud service providers)
+
+On the other hand any of the above may also indicate a leak.
+Taking snapshots after a `forceClearSoftReferences()` (above) invocation and comparing those
+is one technique to filter out noise. Another is to wait until there is an OOME
+and look just after, because that will clear all non-essential data from memory.
+(The `forceClearSoftReferences()` actually works by triggering an OOME, in as safe
+a way as possible.)
+
+If leaked items are found, a profiler will normally let you see their content
+and walk backwards along their references to find out why they are being retained.
+
+
+### Summary of Techniques
+
+The following sequence of techniques is a common approach to investigating and fixing memory issues:
+
+* Note the log lines about `brooklyn gc`, including memory and tasks
+* Do not assume high memory usage alone is an error, as soft reference caches are deliberate;
+ use `forceClearSoftReferences()` to clear these
--- End diff --
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs issue #122: info on debugging memory usage
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on the issue:
https://github.com/apache/brooklyn-docs/pull/122
Perhaps worth adding links to generic resources, so it's clearer we're telling people to use "normal" tools/approaches - e.g. https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/memleaks.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---