You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by mxm <gi...@git.apache.org> on 2015/06/22 17:51:50 UTC
[GitHub] flink pull request: [FLINK-2235] fix calculation of free memory fo...
GitHub user mxm opened a pull request:
https://github.com/apache/flink/pull/859
[FLINK-2235] fix calculation of free memory for local execution
The Java runtime may return Long.MAX_VALUE for a call to the maxMemory()
method of the Runtime class. In these cases, we simply use the value
returned by freeMemory().
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mxm/flink too-much-memory
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/859.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #859
----
commit 97a1f697ac2270d6b1c6c4b98ef1eadd10b5a815
Author: Maximilian Michels <mx...@apache.org>
Date: 2015-06-22T15:47:11Z
[FLINK-2235] fix calculation of free memory for local execution
The Java runtime may return Long.MAX_VALUE for a call to the maxMemory()
method of the Runtime class. In these cases, we simply use the value
returned by freeMemory().
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2235] fix calculation of free memory fo...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/859#discussion_r33036608
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java ---
@@ -137,7 +137,13 @@ public static long getSizeOfFreeHeapMemoryWithDefrag() {
*/
public static long getSizeOfFreeHeapMemory() {
Runtime r = Runtime.getRuntime();
- return r.maxMemory() - r.totalMemory() + r.freeMemory();
+ long maxMemory = r.maxMemory();
+ if (maxMemory == Long.MAX_VALUE) {
+ // workaround for some JVM versions
+ return r.freeMemory();
--- End diff --
OK, I think the problem is that Xmx is not set. In this case maxMemory returns Long.MAX_VALUE.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2235] fix calculation of free memory fo...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/859#issuecomment-120942293
Looks good. Merging this with a modification of the tests that they permit the runtime exception in some cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2235] fix calculation of free memory fo...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/859
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2235] fix calculation of free memory fo...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/859#issuecomment-119891011
Okay, from skimming over some Oracle docs, it seems the default max heap is 1/4 of the physical memory. Let's use that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2235] fix calculation of free memory fo...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/859#issuecomment-119883278
Typically, programs can allocate as much memory as they like. We only take a fraction of the free physical memory for the manged memory. We could also take only half of the physical memory. Or, alternatively, fail with an exception that the maximum memory for the JVM is not set (-Xmx is missing). In my opinion, it is ok to take a fraction of the physical memory for local execution.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2235] fix calculation of free memory fo...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/859#issuecomment-119586674
Is this value actually accurate? It assumes that the JVM can grow to occupy the entire available main memory.
Is that the case?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2235] fix calculation of free memory fo...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/859#discussion_r33037235
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java ---
@@ -137,7 +137,13 @@ public static long getSizeOfFreeHeapMemoryWithDefrag() {
*/
public static long getSizeOfFreeHeapMemory() {
Runtime r = Runtime.getRuntime();
- return r.maxMemory() - r.totalMemory() + r.freeMemory();
+ long maxMemory = r.maxMemory();
+ if (maxMemory == Long.MAX_VALUE) {
+ // workaround for some JVM versions
+ return r.freeMemory();
--- End diff --
OK, didn't see it yesterday. Maybe we can play around with [MemoryUsage](https://docs.oracle.com/javase/7/docs/api/java/lang/management/MemoryUsage.html).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2235] fix calculation of free memory fo...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/859#discussion_r33036711
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java ---
@@ -137,7 +137,13 @@ public static long getSizeOfFreeHeapMemoryWithDefrag() {
*/
public static long getSizeOfFreeHeapMemory() {
Runtime r = Runtime.getRuntime();
- return r.maxMemory() - r.totalMemory() + r.freeMemory();
+ long maxMemory = r.maxMemory();
+ if (maxMemory == Long.MAX_VALUE) {
+ // workaround for some JVM versions
+ return r.freeMemory();
--- End diff --
Yes, exactly. Like discussed in the JIRA issue :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2235] fix calculation of free memory fo...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/859#issuecomment-115594116
I found a fix for the Oracle JVM (the only known JVM with this problem). We call a proprietary method and fail on other JDKs with a nice exception message. @uce What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2235] fix calculation of free memory fo...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/859#discussion_r33020328
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java ---
@@ -137,7 +137,13 @@ public static long getSizeOfFreeHeapMemoryWithDefrag() {
*/
public static long getSizeOfFreeHeapMemory() {
Runtime r = Runtime.getRuntime();
- return r.maxMemory() - r.totalMemory() + r.freeMemory();
+ long maxMemory = r.maxMemory();
+ if (maxMemory == Long.MAX_VALUE) {
+ // workaround for some JVM versions
+ return r.freeMemory();
--- End diff --
We could fall back to a conservative value and print a warning that the memory has to be set manually. To set the memory manually, the user has to go through the LocalExecutor which is not a nice out of the box experience.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2235] fix calculation of free memory fo...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/859#discussion_r32957111
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java ---
@@ -137,7 +137,13 @@ public static long getSizeOfFreeHeapMemoryWithDefrag() {
*/
public static long getSizeOfFreeHeapMemory() {
Runtime r = Runtime.getRuntime();
- return r.maxMemory() - r.totalMemory() + r.freeMemory();
+ long maxMemory = r.maxMemory();
+ if (maxMemory == Long.MAX_VALUE) {
+ // workaround for some JVM versions
+ return r.freeMemory();
--- End diff --
I haven't tested this yet, but I think this [will not work](http://stackoverflow.com/questions/12807797/java-get-available-memory), because `freeMemory` returns to the free memory from the currently allocated memory. The JVM allocates memory in chunks. Free memory refers to how much of these allocated chunks is free.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2235] fix calculation of free memory fo...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/859#issuecomment-119987554
1/4 of the physical memory seems sensible and indeed returns almost the same memory size as the maximum memory setting on my machine. I've adapted the pull request.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2235] fix calculation of free memory fo...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/859#discussion_r33019549
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java ---
@@ -137,7 +137,13 @@ public static long getSizeOfFreeHeapMemoryWithDefrag() {
*/
public static long getSizeOfFreeHeapMemory() {
Runtime r = Runtime.getRuntime();
- return r.maxMemory() - r.totalMemory() + r.freeMemory();
+ long maxMemory = r.maxMemory();
+ if (maxMemory == Long.MAX_VALUE) {
+ // workaround for some JVM versions
+ return r.freeMemory();
--- End diff --
You're right. For a simple WordCount it works but for anything advanced, this will fail with "Too few memory segments provided".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---