You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by mx...@apache.org on 2015/10/06 10:47:44 UTC
[1/3] flink git commit: [FLINK-2810] Warn user if bc not installed
Repository: flink
Updated Branches:
refs/heads/master 9513f0e33 -> 19076b2df
[FLINK-2810] Warn user if bc not installed
This closes #1228.
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/19076b2d
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/19076b2d
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/19076b2d
Branch: refs/heads/master
Commit: 19076b2df677f64ec25413dbd59c25c76208bd3b
Parents: f8fa337
Author: Greg Hogan <co...@greghogan.com>
Authored: Mon Oct 5 15:11:59 2015 +0000
Committer: Maximilian Michels <mx...@apache.org>
Committed: Tue Oct 6 10:46:18 2015 +0200
----------------------------------------------------------------------
flink-dist/src/main/flink-bin/bin/taskmanager.sh | 6 ++++++
1 file changed, 6 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/19076b2d/flink-dist/src/main/flink-bin/bin/taskmanager.sh
----------------------------------------------------------------------
diff --git a/flink-dist/src/main/flink-bin/bin/taskmanager.sh b/flink-dist/src/main/flink-bin/bin/taskmanager.sh
index f63285e..fd29a93 100755
--- a/flink-dist/src/main/flink-bin/bin/taskmanager.sh
+++ b/flink-dist/src/main/flink-bin/bin/taskmanager.sh
@@ -71,6 +71,12 @@ if [[ $STARTSTOP == "start" ]]; then
fi
TM_HEAP_SIZE=$((FLINK_TM_HEAP - FLINK_TM_MEM_MANAGED_SIZE))
else
+ # Bash only performs integer arithmetic so floating point computation is performed using bc
+ command -v bc >/dev/null 2>&1
+ if [[ $? -ne 0 ]]; then
+ echo "[ERROR] Program 'bc' not found. Please install bc or define '${KEY_TASKM_MEM_MANAGED_SIZE}' instead of '${KEY_TASKM_MEM_MANAGED_FRACTION}' in ${FLINK_CONF_FILE}"
+ exit 1
+ fi
# We calculate the memory using a fraction of the total memory
if [[ `bc -l <<< "${FLINK_TM_MEM_MANAGED_FRACTION} >= 1.0"` != "0" ]] || [[ `bc -l <<< "${FLINK_TM_MEM_MANAGED_FRACTION} <= 0.0"` != "0" ]]; then
echo "[ERROR] Configured TaskManager managed memory fraction is not a valid value. Please set '${KEY_TASKM_MEM_MANAGED_FRACTION}' in ${FLINK_CONF_FILE}"
[3/3] flink git commit: [FLINK-1610] fix building of java doc for
Java 8
Posted by mx...@apache.org.
[FLINK-1610] fix building of java doc for Java 8
This closes #1225.
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/352b041b
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/352b041b
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/352b041b
Branch: refs/heads/master
Commit: 352b041bbc51f6df838549cb5508c11bf6a54196
Parents: 9513f0e
Author: Maximilian Michels <mx...@apache.org>
Authored: Mon Oct 5 15:13:48 2015 +0200
Committer: Maximilian Michels <mx...@apache.org>
Committed: Tue Oct 6 10:46:18 2015 +0200
----------------------------------------------------------------------
pom.xml | 15 +++++++++++++++
1 file changed, 15 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/352b041b/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index cfb94e1..2a5ff01 100644
--- a/pom.xml
+++ b/pom.xml
@@ -582,6 +582,21 @@ under the License.
</plugin>
</plugins>
</build>
+ <dependencies>
+ <dependency>
+ <!--
+ This is necessary for building the java docs using Java 8. Otherwise the javadoc
+ plugin will fail with "javadoc: error -
+ com.sun.tools.doclets.internal.toolkit.util.DocletAbortException:
+ com.sun.tools.javac.code.Symbol$CompletionFailure:
+ class file for akka.testkit.TestKit not found"
+ -->
+ <groupId>com.typesafe.akka</groupId>
+ <artifactId>akka-testkit_${scala.binary.version}</artifactId>
+ <version>${akka.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
</profile>
<profile>
[2/3] flink git commit: [FLINK-2813] document parameter for off-heap
memory
Posted by mx...@apache.org.
[FLINK-2813] document parameter for off-heap memory
This closes #1226.
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/f8fa3377
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/f8fa3377
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/f8fa3377
Branch: refs/heads/master
Commit: f8fa3377493359b911a98a6cb278e2c3bb94d9db
Parents: 352b041
Author: Maximilian Michels <mx...@apache.org>
Authored: Mon Oct 5 17:27:57 2015 +0200
Committer: Maximilian Michels <mx...@apache.org>
Committed: Tue Oct 6 10:46:18 2015 +0200
----------------------------------------------------------------------
docs/setup/config.md | 51 +++++++++++++++++++++++++++++++++++------------
1 file changed, 38 insertions(+), 13 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/f8fa3377/docs/setup/config.md
----------------------------------------------------------------------
diff --git a/docs/setup/config.md b/docs/setup/config.md
index 35edf7a..4cfe428 100644
--- a/docs/setup/config.md
+++ b/docs/setup/config.md
@@ -106,6 +106,44 @@ and replication factors. Flink will look for the "core-site.xml" and
## Advanced Options
+### Managed Memory
+
+By default, Flink allocates a fraction of 0.7 of the total memory configured via
+`taskmanager.heap.mb` for its managed memory. Managed memory helps Flink to run
+the operators efficiently. It prevents OutOfMemoryExceptions because Flink knows
+how much memory it can use to execute operations. If Flink runs out of managed
+memory, it utilizes disk space. Using managed memory, some operations can be
+performed directly on the raw data without having to deserialize the data to
+convert it into Java objects. All in all, managed memory improves the robustness
+and speed of the system.
+
+The default fraction for managed memory can be adjusted using the
+`taskmanager.memory.fraction` parameter. An absolute value may be set using
+`taskmanager.memory.size` (overrides the fraction parameter). If desired, the
+managed memory may be allocated outside the JVM heap. This may improve
+performance in setups with large memory sizes.
+
+- `taskmanager.memory.size`: The amount of memory (in megabytes) that the task
+manager reserves on the JVM's heap space for sorting, hash tables, and caching
+of intermediate results. If unspecified (-1), the memory manager will take a fixed
+ratio of the heap memory available to the JVM, as specified by
+`taskmanager.memory.fraction`. (DEFAULT: -1)
+
+- `taskmanager.memory.fraction`: The relative amount of memory that the task
+manager reserves for sorting, hash tables, and caching of intermediate results.
+For example, a value of 0.8 means that TaskManagers reserve 80% of the
+JVM's heap space for internal data buffers, leaving 20% of the JVM's heap space
+free for objects created by user-defined functions. (DEFAULT: 0.7)
+This parameter is only evaluated, if `taskmanager.memory.size` is not set.
+
+- `taskmanager.memory.off-heap`: If set to `true`, the task manager allocates
+memory which is used for sorting, hash tables, and caching of intermediate
+results outside of the JVM heap. For setups with larger quantities of memory,
+this can improve the efficiency of the operations performed on the memory
+(DEFAULT: false).
+
+### Other
+
- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of
directories separated by the systems directory delimiter (for example ':'
(colon) on Linux/Unix). If multiple directories are specified, then the temporary
@@ -135,19 +173,6 @@ a TaskManager can have at the same time and how well buffered the channels are.
If a job is rejected or you get a warning that the system has not enough buffers
available, increase this value (DEFAULT: 2048).
-- `taskmanager.memory.size`: The amount of memory (in megabytes) that the task
-manager reserves on the JVM's heap space for sorting, hash tables, and caching
-of intermediate results. If unspecified (-1), the memory manager will take a fixed
-ratio of the heap memory available to the JVM, as specified by
-`taskmanager.memory.fraction`. (DEFAULT: -1)
-
-- `taskmanager.memory.fraction`: The relative amount of memory that the task
-manager reserves for sorting, hash tables, and caching of intermediate results.
-For example, a value of 0.8 means that TaskManagers reserve 80% of the
-JVM's heap space for internal data buffers, leaving 20% of the JVM's heap space
-free for objects created by user-defined functions. (DEFAULT: 0.7)
-This parameter is only evaluated, if `taskmanager.memory.size` is not set.
-
- `env.java.opts`: Set custom JVM options. This value is respected by Flink's start scripts
and Flink's YARN client.
This can be used to set different garbage collectors or to include remote debuggers into