You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by aledsage <gi...@git.apache.org> on 2016/11/10 15:33:52 UTC
[GitHub] brooklyn-server pull request #426: Improve robustness of MemoryUsageTrackerT...
GitHub user aledsage opened a pull request:
https://github.com/apache/brooklyn-server/pull/426
Improve robustness of MemoryUsageTrackerTest
Motivated by discussion in https://github.com/apache/brooklyn-server/pull/412 with @neykov (cc @ahgittin).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aledsage/brooklyn-server fix-MemoryUsageTrackerTest
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/brooklyn-server/pull/426.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #426
----
commit e6b40af13e572fbb509410eded6ee094c465d6e1
Author: Aled Sage <al...@gmail.com>
Date: 2016-11-10T15:32:45Z
Improve robustness of MemoryUsageTrackerTest
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server issue #426: Improve robustness of MemoryUsageTrackerTest
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on the issue:
https://github.com/apache/brooklyn-server/pull/426
@Tibor17 not sure what you mean by "The prefix `SUREFIRE-859` does not exist in the console."
I see the output that @neykov pasted when I follow the link to https://builds.apache.org/view/Brooklyn/job/brooklyn-integration-tests/52/console.
When I run locally (e.g. with `mvn test -Dtest=MemoryUsageTrackerTest#testSoftUsageAndClearance`) then I see the same `SUREFIRE` prefixes for the memory usage. This is because of the `-verbose:gc` at https://github.com/apache/brooklyn-server/blob/master/parent/pom.xml#L691
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server issue #426: Improve robustness of MemoryUsageTrackerTest
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the issue:
https://github.com/apache/brooklyn-server/pull/426
The `SUREFIRE-859` prefix lines may depend on which JVM is being used?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server issue #426: Improve robustness of MemoryUsageTrackerTest
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on the issue:
https://github.com/apache/brooklyn-server/pull/426
@neykov Merging this as soon as jenkins builds, so we can then see what this does in the apache infra integration tests. Unless anyone spots any problems!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request #426: Improve robustness of MemoryUsageTrackerT...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/brooklyn-server/pull/426
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server issue #426: Improve robustness of MemoryUsageTrackerTest
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the issue:
https://github.com/apache/brooklyn-server/pull/426
Kind of related discussion on the prefix in a [previous PR](https://github.com/apache/incubator-brooklyn/pull/1068#issuecomment-159588763).
According to the link the prefix doesn't exist any more in surefire 2.19+.
I believe that's fine for the Brooklyn project - we don't rely on it anywhere. Still the GC output is useful for tests so better keep it enabled. I upgraded surefire to latest version in https://github.com/apache/brooklyn-server/pull/434 and indeed the prefix is nowhere to be seen.
Thanks for checking with us @Tibor17.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server issue #426: Improve robustness of MemoryUsageTrackerTest
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the issue:
https://github.com/apache/brooklyn-server/pull/426
Looks good and works much better, see the [new build](https://builds.apache.org/view/Brooklyn/job/brooklyn-integration-tests/52/console)
```
2016-11-10 15:47:31,205 INFO TESTNG INVOKING: "Surefire test" - org.apache.brooklyn.util.javalang.MemoryUsageTrackerTest.testSoftUsageAndClearance()
2016-11-10 15:47:31,206 INFO Memory usage at start of test: MemoryUsageSummary{total=772 MB, free=652 MB, used=120 MB, usedFraction=0.1555049937704335}
SUREFIRE-859: [GC (Allocation Failure) 265638K->264541K(753664K), 0.0252673 secs]
SUREFIRE-859: [GC (Allocation Failure) 460429K->459760K(753152K), 0.0260455 secs]
SUREFIRE-859: [Full GC (Ergonomics) 459760K->351817K(753152K), 0.0518254 secs]
SUREFIRE-859: [Full GC (Ergonomics) 548263K->548101K(753152K), 0.1852436 secs]
SUREFIRE-859: [Full GC (Ergonomics) 719375K->718997K(753152K), 0.0680970 secs]
SUREFIRE-859: [Full GC (Ergonomics) 720275K->719974K(753152K), 0.0546380 secs]
SUREFIRE-859: [Full GC (Allocation Failure) 719974K->8048K(753152K), 0.0267869 secs]
2016-11-10 15:47:31,782 INFO First soft reference cleared after 730 1M blocks created; 729 of them cleared; memory just before collected is MemoryUsageSummary{total=771 MB, free=33.7 MB, used=738 MB, usedFraction=0.9563477890253229}
SUREFIRE-859: [Full GC (Ergonomics) 704594K->702492K(753152K), 0.2032216 secs]
SUREFIRE-859: [Full GC (Ergonomics) 712258K->711185K(753152K), 0.0347821 secs]
SUREFIRE-859: [Full GC (Allocation Failure) 711185K->711185K(753152K), 0.0322254 secs]
SUREFIRE-859: [Full GC (System.gc()) 711185K->8055K(753152K), 0.0285412 secs]
SUREFIRE-859: [GC (System.gc()) 8055K->8055K(645632K), 0.0029759 secs]
SUREFIRE-859: [Full GC (System.gc()) 8055K->8055K(645632K), 0.0225672 secs]
2016-11-10 15:47:32,244 INFO Forcing memory eviction: allocated 725 MB +- 5 MB really free memory in 10 MB chunks; memory used from 11.5 MB -> 8.25 MB / 661 MB
SUREFIRE-859: [GC (System.gc()) 9315K->8215K(699392K), 0.0011906 secs]
SUREFIRE-859: [Full GC (System.gc()) 8215K->8063K(699392K), 0.0333181 secs]
SUREFIRE-859: [GC (System.gc()) 8063K->8063K(699392K), 0.0021073 secs]
SUREFIRE-859: [Full GC (System.gc()) 8063K->8063K(699392K), 0.0225468 secs]
2016-11-10 15:47:32,305 INFO Final memory usage (after forcing clear, and GC): MemoryUsageSummary{total=716 MB, free=708 MB, used=8.26 MB, usedFraction=0.01152881940671326}
2016-11-10 15:47:32,305 INFO TESTNG PASSED: "Surefire test" - org.apache.brooklyn.util.javalang.MemoryUsageTrackerTest.testSoftUsageAndClearance() finished in 1100 ms
```
Merging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server issue #426: Improve robustness of MemoryUsageTrackerTest
Posted by Tibor17 <gi...@git.apache.org>.
Github user Tibor17 commented on the issue:
https://github.com/apache/brooklyn-server/pull/426
@aledsage
The prefix `SUREFIRE-859` does not exist in the console.
Is it fine for you?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request #426: Improve robustness of MemoryUsageTrackerT...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/426#discussion_r87422520
--- Diff: utils/common/src/test/java/org/apache/brooklyn/util/javalang/MemoryUsageTrackerTest.java ---
@@ -84,36 +89,82 @@ private long sizeOfActiveReferences(List<Maybe<byte[]>> references) {
@Test(groups="Integration")
public void testSoftUsageAndClearance() {
- long totalMemory = Runtime.getRuntime().totalMemory();
- long freeMemory = Runtime.getRuntime().freeMemory();
+ MemoryUsageSummary initialMemory = new MemoryUsageSummary();
+ LOG.info("Memory usage at start of test: "+initialMemory);
+
+ MemoryUsageSummary beforeCollectedMemory = null;
List<Maybe<?>> dump = MutableList.of();
- Maybe<byte[]> first = Maybe.soft(new byte[1000*1000]);
- dump.add(first);
for (int i=0; i<1000*1000; i++) {
- totalMemory = Runtime.getRuntime().totalMemory();
- freeMemory = Runtime.getRuntime().freeMemory();
+ beforeCollectedMemory = new MemoryUsageSummary();
dump.add(Maybe.soft(new byte[1000*1000]));
- if (first.isAbsent()) break;
+ if (containsAbsent(dump)) break;
}
- int cleared = 0;
- for (Maybe<?> m: dump) { if (m.isAbsent()) cleared++; }
- LOG.info("First soft reference cleared after "+dump.size()+" 1M blocks created; "+cleared+" of them cleared");
+ int cleared = countAbsents(dump);
+ LOG.info("First soft reference cleared after "+dump.size()+" 1M blocks created; "+cleared+" of them cleared; memory just before collected is "+beforeCollectedMemory);
--- End diff --
`assertTrue(cleared)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---