You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by mx...@apache.org on 2015/03/23 09:09:29 UTC
[9/9] flink git commit: [FLINK-1679] extend faq and programming guide
to clarify parallelism
[FLINK-1679] extend faq and programming guide to clarify parallelism
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/126f9f79
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/126f9f79
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/126f9f79
Branch: refs/heads/master
Commit: 126f9f799071688fe80955a7e7cfa991f53c95af
Parents: 013ed82
Author: Maximilian Michels <mx...@apache.org>
Authored: Wed Mar 18 10:44:45 2015 +0100
Committer: Maximilian Michels <mx...@apache.org>
Committed: Mon Mar 23 09:03:56 2015 +0100
----------------------------------------------------------------------
docs/config.md | 16 +++++++++----
docs/faq.md | 21 ++++++++++++++++
docs/programming_guide.md | 54 ++++++++++++++++++++++++++++++++++++++++++
3 files changed, 86 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/126f9f79/docs/config.md
----------------------------------------------------------------------
diff --git a/docs/config.md b/docs/config.md
index 09a01d9..3524a60 100644
--- a/docs/config.md
+++ b/docs/config.md
@@ -80,11 +80,17 @@ This value is typically proportional to the number of physical CPU cores that
the TaskManager's machine has (e.g., equal to the number of cores, or half the
number of cores). [More about task slots](config.html#configuring-taskmanager-processing-slots).
-- `parallelism.default`: The default parallelism to use for
-programs that have no parallelism specified. (DEFAULT: 1). For
-setups that have no concurrent jobs running, setting this value to
-NumTaskManagers * NumSlotsPerTaskManager will cause the system to use all
-available execution resources for the program's execution.
+- `parallelism.default`: The default parallelism to use for programs that have
+no parallelism specified. (DEFAULT: 1). For setups that have no concurrent jobs
+running, setting this value to NumTaskManagers * NumSlotsPerTaskManager will
+cause the system to use all available execution resources for the program's
+execution. **Note**: The default parallelism can be overwriten for an entire
+job by calling `setParallelism(int parallelism)` on the `ExecutionEnvironment`
+or by passing `-p <parallelism>` to the Flink Command-line frontend. It can be
+overwritten for single transformations by calling `setParallelism(int
+parallelism)` on an operator. See the [programming
+guide](programming_guide.html#parallel-execution) for more information about the
+parallelism.
- `fs.hdfs.hadoopconf`: The absolute path to the Hadoop File System's (HDFS)
configuration directory (OPTIONAL VALUE).
http://git-wip-us.apache.org/repos/asf/flink/blob/126f9f79/docs/faq.md
----------------------------------------------------------------------
diff --git a/docs/faq.md b/docs/faq.md
index 2c3fe53..5c31bc6 100644
--- a/docs/faq.md
+++ b/docs/faq.md
@@ -85,6 +85,27 @@ tracking the behavior of the parallel execution. They allow you to gather
information inside the program's operations and show them after the program
execution.
+### What is the parallelism? How do I set it?
+
+In Flink programs, the parallelism determines how operations are split into
+individual tasks which are assigned to task slots. Each node in a cluster has at
+least one task slot. The total number of task slots is the number of all task slots
+on all machines. If the parallelism is set to `N`, Flink tries to divide an
+operation into `N` parallel tasks which can be computed concurrently using the
+available task slots. The number of task slots should be equal to the
+parallelism to ensure that all tasks can be computed in a task slot concurrently.
+
+**Note**: Not all operations can be divided into multiple tasks. For example, a
+`GroupReduce` operation without a grouping has to be performed with a
+parallelism of 1 because the entire group needs to be present at exactly one
+node to perform the reduce operation. Flink will determine whether the
+parallelism has to be 1 and set it accordingly.
+
+The parallelism can be set in numerous ways to ensure a fine-grained control
+over the execution of a Flink program. See
+[Configuration](config.html#common-options) for detailed instructions on how to
+set the parallelism.
+
## Errors
### Why am I getting a "NonSerializableException" ?
http://git-wip-us.apache.org/repos/asf/flink/blob/126f9f79/docs/programming_guide.md
----------------------------------------------------------------------
diff --git a/docs/programming_guide.md b/docs/programming_guide.md
index 1750a12..4e15b91 100644
--- a/docs/programming_guide.md
+++ b/docs/programming_guide.md
@@ -2789,6 +2789,60 @@ env.execute("Word Count Example")
</div>
</div>
+### Client Level
+
+The parallelism can be set at the Client when submitting jobs to Flink. The
+Client can either be a Java or a Scala program. One example of such a Client is
+Flink's Command-line Interface (CLI).
+
+For the CLI client, the parallelism parameter can be specified with `-p`. For
+exampple:
+
+ ./bin/flink run -p 10 ../examples/*WordCount-java*.jar
+
+
+In a Java/Scala program, the parallelism is set as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+
+try {
+ PackagedProgram program = new PackagedProgram(file, args);
+ InetSocketAddress jobManagerAddress = RemoteExecutor.getInetFromHostport("localhost:6123");
+ Configuration config = new Configuration();
+
+ Client client = new Client(jobManagerAddress, config, program.getUserCodeClassLoader());
+
+ // set the parallelism to 10 here
+ client.run(program, 10, true);
+
+} catch (ProgramInvocationException e) {
+ e.printStackTrace();
+}
+
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+try {
+ PackagedProgram program = new PackagedProgram(file, args)
+ InetSocketAddress jobManagerAddress = RemoteExecutor.getInetFromHostport("localhost:6123")
+ Configuration config = new Configuration()
+
+ Client client = new Client(jobManagerAddress, new Configuration(), program.getUserCodeClassLoader())
+
+ // set the parallelism to 10 here
+ client.run(program, 10, true)
+
+} catch {
+ case e: Exception => e.printStackTrace
+}
+{% endhighlight %}
+</div>
+</div>
+
+
### System Level
A system-wide default parallelism for all execution environments can be defined by setting the