You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by fh...@apache.org on 2016/12/16 15:47:12 UTC
[43/47] flink git commit: [FLINK-5008] [docs] Update IDE setup and
quickstart documentation.
[FLINK-5008] [docs] Update IDE setup and quickstart documentation.
This closes #2764.
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/67c4be64
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/67c4be64
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/67c4be64
Branch: refs/heads/master
Commit: 67c4be648b1f51ceadae3a9e3dd41807802a89ef
Parents: 22af6cf
Author: Nico Kruber <ni...@data-artisans.com>
Authored: Thu Nov 3 16:19:49 2016 +0100
Committer: Fabian Hueske <fh...@apache.org>
Committed: Fri Dec 16 16:41:21 2016 +0100
----------------------------------------------------------------------
README.md | 24 +---
docs/internals/ide_setup.md | 129 +++++++------------
docs/page/img/quickstart-setup/jobmanager-1.png | Bin 118647 -> 66956 bytes
docs/page/img/quickstart-setup/jobmanager-2.png | Bin 136935 -> 77840 bytes
docs/page/img/quickstart-setup/jobmanager-3.png | Bin 184257 -> 97567 bytes
docs/page/img/quickstart-setup/setup.gif | Bin 495692 -> 0 bytes
docs/quickstart/java_api_quickstart.md | 69 ++++++++--
docs/quickstart/run_example_quickstart.md | 34 ++---
docs/quickstart/scala_api_quickstart.md | 71 ++++++++--
docs/quickstart/setup_quickstart.md | 89 ++++++++-----
.../examples/socket/SocketWindowWordCount.java | 4 +-
.../examples/socket/SocketWindowWordCount.scala | 2 +-
12 files changed, 240 insertions(+), 182 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/67c4be64/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 8fd533b..025dfb9 100644
--- a/README.md
+++ b/README.md
@@ -85,7 +85,7 @@ To build unit tests with Java 8, use Java 8u51 or above to prevent failures in u
## Developing Flink
-The Flink committers use IntelliJ IDEA and Eclipse IDE to develop the Flink codebase.
+The Flink committers use IntelliJ IDEA to develop the Flink codebase.
We recommend IntelliJ IDEA for developing projects that involve Scala code.
Minimal requirements for an IDE are:
@@ -104,25 +104,11 @@ Check out our [Setting up IntelliJ](https://github.com/apache/flink/blob/master/
### Eclipse Scala IDE
-For Eclipse users, we recommend using Scala IDE 3.0.3, based on Eclipse Kepler. While this is a slightly older version,
-we found it to be the version that works most robustly for a complex project like Flink.
-
-Further details, and a guide to newer Scala IDE versions can be found in the
-[How to setup Eclipse](https://github.com/apache/flink/blob/master/docs/internals/ide_setup.md#eclipse) docs.
-
-**Note:** Before following this setup, make sure to run the build from the command line once
-(`mvn clean install -DskipTests`, see above)
-
-1. Download the Scala IDE (preferred) or install the plugin to Eclipse Kepler. See
- [How to setup Eclipse](https://github.com/apache/flink/blob/master/docs/internals/ide_setup.md#eclipse) for download links and instructions.
-2. Add the "macroparadise" compiler plugin to the Scala compiler.
- Open "Window" -> "Preferences" -> "Scala" -> "Compiler" -> "Advanced" and put into the "Xplugin" field the path to
- the *macroparadise* jar file (typically "/home/*-your-user-*/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar").
- Note: If you do not have the jar file, you probably did not run the command line build.
-3. Import the Flink Maven projects ("File" -> "Import" -> "Maven" -> "Existing Maven Projects")
-4. During the import, Eclipse will ask to automatically install additional Maven build helper plugins.
-5. Close the "flink-java8" project. Since Eclipse Kepler does not support Java 8, you cannot develop this project.
+**NOTE:** From our experience, this setup does not work with Flink
+due to deficiencies of the old Eclipse version bundled with Scala IDE 3.0.3 or
+due to version incompatibilities with the bundled Scala version in Scala IDE 4.4.1.
+**We recommend to use IntelliJ instead (see above)**
## Support
http://git-wip-us.apache.org/repos/asf/flink/blob/67c4be64/docs/internals/ide_setup.md
----------------------------------------------------------------------
diff --git a/docs/internals/ide_setup.md b/docs/internals/ide_setup.md
index 1c514d0..f170588 100644
--- a/docs/internals/ide_setup.md
+++ b/docs/internals/ide_setup.md
@@ -25,105 +25,66 @@ under the License.
* Replaced by the TOC
{:toc}
+The sections below describe how to import the Flink project into an IDE
+for the development of Flink itself. For writing Flink programs, please
+refer to the [Java API]({{ site.baseurl }}/quickstart/java_api_quickstart.html)
+and the [Scala API]({{ site.baseurl }}/quickstart/scala_api_quickstart.html)
+quickstart guides.
+
+**NOTE:** Whenever something is not working in your IDE, try with the Maven
+command line first (`mvn clean package -DskipTests`) as it might be your IDE
+that has a bug or is not properly set up.
+
+## Preparation
+
+To get started, please first checkout the Flink sources from one of our
+[repositories](https://flink.apache.org/community.html#source-code),
+e.g.
+{% highlight bash %}
+git clone https://github.com/apache/flink.git
+{% endhighlight %}
+
## IntelliJ IDEA
A brief guide on how to set up IntelliJ IDEA IDE for development of the Flink core.
As Eclipse is known to have issues with mixed Scala and Java projects, more and more contributers are migrating to IntelliJ IDEA.
-The following documentation describes the steps to setup IntelliJ IDEA 14.0.3 (https://www.jetbrains.com/idea/download/) with the Flink sources.
-
-Prior to doing anything, make sure that the Flink project is built at least once from the terminal:
-`mvn clean package -DskipTests`
+The following documentation describes the steps to setup IntelliJ IDEA 2016.2.5
+([https://www.jetbrains.com/idea/download/](https://www.jetbrains.com/idea/download/))
+with the Flink sources.
### Installing the Scala plugin
-1. Go to IntelliJ plugins settings (File -> Settings -> Plugins) and click on "Install Jetbrains plugin...".
+
+The IntelliJ installation setup offers to install the Scala plugin.
+If it is not installed, follow these instructions before importing Flink
+to enable support for Scala projects and files:
+
+1. Go to IntelliJ plugins settings (File -> Settings -> Plugins) and
+ click on "Install Jetbrains plugin...".
2. Select and install the "Scala" plugin.
3. Restart IntelliJ
-### Installing the Scala compiler plugin
-1. Go to IntelliJ scala compiler settings (File -> Settings -> Build, Execution, Deployment -> Compiler -> Scala Compiler) and click on "Install Jetbrains plugin...".
-2. Click on the green plus icon on the right to add a compiler plugin
-3. Point to the paradise jar: ~/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar If there is no such file, this means that you should build Flink from the terminal as explained above.
-
### Importing Flink
+
1. Start IntelliJ IDEA and choose "Import Project"
2. Select the root folder of the Flink repository
3. Choose "Import project from external model" and select "Maven"
-4. Leave the default options and finish the import.
+4. Leave the default options and click on "Next" until you hit the SDK section.
+5. If there is no SDK, create a one with the "+" sign top left,
+ then click "JDK", select your JDK home directory and click "OK".
+ Otherwise simply select your SDK.
+6. Continue by clicking "Next" again and finish the import.
+7. Right-click on the imported Flink project -> Maven -> Generate Sources and Update Folders.
+ Note that this will install Flink libraries in your local Maven repository,
+ i.e. "/home/*-your-user-*/.m2/repository/org/apache/flink/".
+ Alternatively, `mvn clean package -DskipTests` also creates the necessary
+ files for the IDE to work with but without installing libraries.
+8. Build the Project (Build -> Make Project)
## Eclipse
-A brief guide how to set up Eclipse for development of the Flink core.
-Flink uses mixed Scala/Java projects, which pose a challenge to some IDEs.
-Below is the setup guide that works best from our personal experience.
-
-For Eclipse users, we currently recomment the Scala IDE 3.0.3, as the most robust solution.
-
-
-### Eclipse Scala IDE 3.0.3
-
-**NOTE:** While this version of the Scala IDE is not the newest, we have found it to be the most reliably working
-version for complex projects like Flink. One restriction is, though, that it works only with Java 7, not with Java 8.
-
-**Note:** Before following this setup, make sure to run the build from the command line once
-(`mvn clean package -DskipTests`)
-
-1. Download the Scala IDE (preferred) or install the plugin to Eclipse Kepler. See section below for download links
- and instructions.
-2. Add the "macroparadise" compiler plugin to the Scala compiler.
- Open "Window" -> "Preferences" -> "Scala" -> "Compiler" -> "Advanced" and put into the "Xplugin" field the path to
- the *macroparadise* jar file (typically "/home/*-your-user-*/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar").
- Note: If you do not have the jar file, you probably did not ran the command line build.
-3. Import the Flink Maven projects ("File" -> "Import" -> "Maven" -> "Existing Maven Projects")
-4. During the import, Eclipse will ask to automatically install additional Maven build helper plugins.
-5. Close the "flink-java8" project. Since Eclipse Kepler does not support Java 8, you cannot develop this project.
-
-
-#### Download links for Scala IDE 3.0.3
-
-The Scala IDE 3.0.3 is a previous stable release, and download links are a bit hidden.
-
-The pre-packaged Scala IDE can be downloaded from the following links:
-
-* [Linux (64 bit)](http://downloads.typesafe.com/scalaide-pack/3.0.3.vfinal-210-20140327/scala-SDK-3.0.3-2.10-linux.gtk.x86_64.tar.gz)
-* [Linux (32 bit)](http://downloads.typesafe.com/scalaide-pack/3.0.3.vfinal-210-20140327/scala-SDK-3.0.3-2.10-linux.gtk.x86.tar.gz)
-* [MaxOS X Cocoa (64 bit)](http://downloads.typesafe.com/scalaide-pack/3.0.3.vfinal-210-20140327/scala-SDK-3.0.3-2.10-macosx.cocoa.x86_64.zip)
-* [MaxOS X Cocoa (32 bit)](http://downloads.typesafe.com/scalaide-pack/3.0.3.vfinal-210-20140327/scala-SDK-3.0.3-2.10-macosx.cocoa.x86.zip)
-* [Windows (64 bit)](http://downloads.typesafe.com/scalaide-pack/3.0.3.vfinal-210-20140327/scala-SDK-3.0.3-2.10-win32.win32.x86_64.zip)
-* [Windows (32 bit)](http://downloads.typesafe.com/scalaide-pack/3.0.3.vfinal-210-20140327/scala-SDK-3.0.3-2.10-win32.win32.x86.zip)
-
-Alternatively, you can download Eclipse Kepler from [https://eclipse.org/downloads/packages/release/Kepler/SR2](https://eclipse.org/downloads/packages/release/Kepler/SR2)
-and manually add the Scala and Maven plugins by plugin site at [http://scala-ide.org/download/prev-stable.html](http://scala-ide.org/download/prev-stable.html).
-
-* Either use the update site to install the plugin ("Help" -> "Install new Software")
-* Or download the [zip file](http://download.scala-ide.org/sdk/helium/e38/scala211/stable/update-site.zip), unpack it, and move the contents of the
- "plugins" and "features" folders into the equally named folders of the Eclipse root directory
-
-**NOTE:** It might happen that some modules do not build in Eclipse correctly (even if the maven build succeeds).
-To fix this, right-click in the corresponding Eclipse project and choose "Properties" and than "Maven".
-Uncheck the box labeled "Resolve dependencies from Workspace projects", click "Apply" and then "OK". "
-
-
-### Eclipse Scala IDE 4.0.0
-
-**NOTE: From personal experience, the use of the Scala IDE 4.0.0 performs worse than previous versions for complex projects like Flink.**
-**Version 4.0.0 does not handle mixed Java/Scala projects as robustly and it frequently raises incorrect import and type errors.**
-
-*Note:* Before following this setup, make sure to run the build from the command line once
-(`mvn clean package -DskipTests`)
-
-1. Download the Scala IDE: [http://scala-ide.org/download/sdk.html](http://scala-ide.org/download/sdk.html)
-2. Import the Flink Maven projects (File -> Import -> Maven -> Existing Maven Projects)
-3. While importing the Flink project, the IDE may ask you to install an additional maven build helper plugin.
-4. After the import, you need to set the Scala version of your projects to Scala 2.10 (from the default 2.11).
- To do that, select all projects that contain Scala code (marked by the small *S* on the project icon),
- right click and select "Scala -> Set the Scala Installation" and pick "2.10.4".
- Currently, the project to which that is relevant are "flink-runtime", "flink-scala", "flink-scala-examples",
- "flink-streaming-example", "flink-streaming-scala", "flink-tests", "flink-test-utils", and "flink-yarn".
-5. Depending on your version of the Scala IDE, you may need to add the "macroparadise" compiler plugin to the
- Scala compiler. Open "Window" -> "Preferences" -> "Scala" -> "Compiler" -> "Advanced" and put into the "Xplugin" field
- the path to the *macroparadise* jar file (typically "/home/*-your-user-*/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar")
-6. In order to compile the "flink-java-8" project, you may need to add a Java 8 execution environment.
- See [this post](http://stackoverflow.com/questions/25391207/how-do-i-add-execution-environment-1-8-to-eclipse-luna)
- for details.
+**NOTE:** From our experience, this setup does not work with Flink
+due to deficiencies of the old Eclipse version bundled with Scala IDE 3.0.3 or
+due to version incompatibilities with the bundled Scala version in Scala IDE 4.4.1.
+**We recommend to use IntelliJ instead (see [above](#intellij-idea))**
http://git-wip-us.apache.org/repos/asf/flink/blob/67c4be64/docs/page/img/quickstart-setup/jobmanager-1.png
----------------------------------------------------------------------
diff --git a/docs/page/img/quickstart-setup/jobmanager-1.png b/docs/page/img/quickstart-setup/jobmanager-1.png
index c6ecfb1..41243de 100644
Binary files a/docs/page/img/quickstart-setup/jobmanager-1.png and b/docs/page/img/quickstart-setup/jobmanager-1.png differ
http://git-wip-us.apache.org/repos/asf/flink/blob/67c4be64/docs/page/img/quickstart-setup/jobmanager-2.png
----------------------------------------------------------------------
diff --git a/docs/page/img/quickstart-setup/jobmanager-2.png b/docs/page/img/quickstart-setup/jobmanager-2.png
index 724f21e..09e7169 100644
Binary files a/docs/page/img/quickstart-setup/jobmanager-2.png and b/docs/page/img/quickstart-setup/jobmanager-2.png differ
http://git-wip-us.apache.org/repos/asf/flink/blob/67c4be64/docs/page/img/quickstart-setup/jobmanager-3.png
----------------------------------------------------------------------
diff --git a/docs/page/img/quickstart-setup/jobmanager-3.png b/docs/page/img/quickstart-setup/jobmanager-3.png
index 8572cf1..ac9e1ef 100644
Binary files a/docs/page/img/quickstart-setup/jobmanager-3.png and b/docs/page/img/quickstart-setup/jobmanager-3.png differ
http://git-wip-us.apache.org/repos/asf/flink/blob/67c4be64/docs/page/img/quickstart-setup/setup.gif
----------------------------------------------------------------------
diff --git a/docs/page/img/quickstart-setup/setup.gif b/docs/page/img/quickstart-setup/setup.gif
deleted file mode 100644
index bc9aad6..0000000
Binary files a/docs/page/img/quickstart-setup/setup.gif and /dev/null differ
http://git-wip-us.apache.org/repos/asf/flink/blob/67c4be64/docs/quickstart/java_api_quickstart.md
----------------------------------------------------------------------
diff --git a/docs/quickstart/java_api_quickstart.md b/docs/quickstart/java_api_quickstart.md
index c8f21d8..7384d91 100644
--- a/docs/quickstart/java_api_quickstart.md
+++ b/docs/quickstart/java_api_quickstart.md
@@ -46,38 +46,80 @@ Use one of the following commands to __create a project__:
{% highlight bash %}
$ mvn archetype:generate \
-DarchetypeGroupId=org.apache.flink \
- -DarchetypeArtifactId=flink-quickstart-java \
+ -DarchetypeArtifactId=flink-quickstart-java \{% unless site.is_stable %}
+ -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %}
-DarchetypeVersion={{site.version}}
{% endhighlight %}
This allows you to <strong>name your newly created project</strong>. It will interactively ask you for the groupId, artifactId, and package name.
</div>
<div class="tab-pane" id="quickstart-script">
{% highlight bash %}
+{% if site.is_stable %}
$ curl https://flink.apache.org/q/quickstart.sh | bash
+{% else %}
+ $ curl https://flink.apache.org/q/quickstart-SNAPSHOT.sh | bash
+{% endif %}
{% endhighlight %}
</div>
</div>
## Inspect Project
-There will be a new directory in your working directory. If you've used the _curl_ approach, the directory is called `quickstart`. Otherwise, it has the name of your artifactId.
+There will be a new directory in your working directory. If you've used
+the _curl_ approach, the directory is called `quickstart`. Otherwise,
+it has the name of your `artifactId`:
+
+{% highlight bash %}
+$ tree quickstart/
+quickstart/
+\u251c\u2500\u2500 pom.xml
+\u2514\u2500\u2500 src
+ \u2514\u2500\u2500 main
+ \u251c\u2500\u2500 java
+ \u2502�� \u2514\u2500\u2500 org
+ \u2502�� \u2514\u2500\u2500 myorg
+ \u2502�� \u2514\u2500\u2500 quickstart
+ \u2502�� \u251c\u2500\u2500 BatchJob.java
+ \u2502�� \u251c\u2500\u2500 SocketTextStreamWordCount.java
+ \u2502�� \u251c\u2500\u2500 StreamingJob.java
+ \u2502�� \u2514\u2500\u2500 WordCount.java
+ \u2514\u2500\u2500 resources
+ \u2514\u2500\u2500 log4j.properties
+{% endhighlight %}
The sample project is a __Maven project__, which contains four classes. _StreamingJob_ and _BatchJob_ are basic skeleton programs, _SocketTextStreamWordCount_ is a working streaming example and _WordCountJob_ is a working batch example. Please note that the _main_ method of all classes allow you to start Flink in a development/testing mode.
-We recommend you [import this project into your IDE]({{ site.baseurl }}/internals/ide_setup) to develop and test it.
+We recommend you __import this project into your IDE__ to develop and
+test it. If you use Eclipse, the [m2e plugin](http://www.eclipse.org/m2e/)
+allows to [import Maven projects](http://books.sonatype.com/m2eclipse-book/reference/creating-sect-importing-projects.html#fig-creating-import).
+Some Eclipse bundles include that plugin by default, others require you
+to install it manually. The IntelliJ IDE supports Maven projects out of
+the box.
-A note to Mac OS X users: The default JVM heapsize for Java is too small for Flink. You have to manually increase it. Choose "Run Configurations" -> Arguments and write into the "VM Arguments" box: "-Xmx800m" in Eclipse.
+
+*A note to Mac OS X users*: The default JVM heapsize for Java is too
+small for Flink. You have to manually increase it. In Eclipse, choose
+`Run Configurations -> Arguments` and write into the `VM Arguments`
+box: `-Xmx800m`.
## Build Project
-If you want to __build your project__, go to your project directory and issue the `mvn clean install -Pbuild-jar` command. You will __find a jar__ that runs on every Flink cluster in __target/your-artifact-id-{{ site.version }}.jar__. There is also a fat-jar, __target/your-artifact-id-{{ site.version }}-flink-fat-jar.jar__. This
-also contains all dependencies that get added to the maven project.
+If you want to __build your project__, go to your project directory and
+issue the `mvn clean install -Pbuild-jar` command. You will
+__find a jar__ that runs on every Flink cluster with a compatible
+version, __target/original-your-artifact-id-your-version.jar__. There
+is also a fat-jar in __target/your-artifact-id-your-version.jar__ which,
+additionally, contains all dependencies that were added to the Maven
+project.
## Next Steps
Write your application!
-The quickstart project contains a WordCount implementation, the "Hello World" of Big Data processing systems. The goal of WordCount is to determine the frequencies of words in a text, e.g., how often do the terms "the" or "house" occur in all Wikipedia texts.
+The quickstart project contains a `WordCount` implementation, the
+"Hello World" of Big Data processing systems. The goal of `WordCount`
+is to determine the frequencies of words in a text, e.g., how often do
+the terms "the" or "house" occur in all Wikipedia texts.
__Sample Input__:
@@ -93,7 +135,10 @@ data 1
is 1
~~~
-The following code shows the WordCount implementation from the Quickstart which processes some text lines with two operators (FlatMap and Reduce), and prints the resulting words and counts to std-out.
+The following code shows the `WordCount` implementation from the
+Quickstart which processes some text lines with two operators (a FlatMap
+and a Reduce operation via aggregating a sum), and prints the resulting
+words and counts to std-out.
~~~java
public class WordCount {
@@ -116,9 +161,9 @@ public class WordCount {
text.flatMap(new LineSplitter())
// group by the tuple field "0" and sum up tuple field "1"
.groupBy(0)
- .aggregate(Aggregations.SUM, 1);
+ .sum(1);
- // emit result
+ // execute and print result
counts.print();
}
}
@@ -127,11 +172,11 @@ public class WordCount {
The operations are defined by specialized classes, here the LineSplitter class.
~~~java
-public class LineSplitter implements FlatMapFunction<String, Tuple2<String, Integer>> {
+public static final class LineSplitter implements FlatMapFunction<String, Tuple2<String, Integer>> {
@Override
public void flatMap(String value, Collector<Tuple2<String, Integer>> out) {
- // normalize and split the line into words
+ // normalize and split the line
String[] tokens = value.toLowerCase().split("\\W+");
// emit the pairs
http://git-wip-us.apache.org/repos/asf/flink/blob/67c4be64/docs/quickstart/run_example_quickstart.md
----------------------------------------------------------------------
diff --git a/docs/quickstart/run_example_quickstart.md b/docs/quickstart/run_example_quickstart.md
index 9ce4c84..d68a8a5 100644
--- a/docs/quickstart/run_example_quickstart.md
+++ b/docs/quickstart/run_example_quickstart.md
@@ -41,15 +41,16 @@ see [Java API Quickstart]({{ site.baseurl }}/quickstart/java_api_quickstart.html
about this. For our purposes, the command to run is this:
{% highlight bash %}
-$ mvn archetype:generate\
- -DarchetypeGroupId=org.apache.flink\
- -DarchetypeArtifactId=flink-quickstart-java\
- -DarchetypeVersion={{ site.version }}\
- -DgroupId=wiki-edits\
- -DartifactId=wiki-edits\
- -Dversion=0.1\
- -Dpackage=wikiedits\
- -DinteractiveMode=false\
+$ mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.flink \
+ -DarchetypeArtifactId=flink-quickstart-java \{% unless site.is_stable %}
+ -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %}
+ -DarchetypeVersion={{ site.version }} \
+ -DgroupId=wiki-edits \
+ -DartifactId=wiki-edits \
+ -Dversion=0.1 \
+ -Dpackage=wikiedits \
+ -DinteractiveMode=false
{% endhighlight %}
You can edit the `groupId`, `artifactId` and `package` if you like. With the above parameters,
@@ -63,8 +64,9 @@ wiki-edits/
\u2514\u2500\u2500 main
\u251c\u2500\u2500 java
\u2502�� \u2514\u2500\u2500 wikiedits
- \u2502�� \u251c\u2500\u2500 Job.java
+ \u2502�� \u251c\u2500\u2500 BatchJob.java
\u2502�� \u251c\u2500\u2500 SocketTextStreamWordCount.java
+ \u2502�� \u251c\u2500\u2500 StreamingJob.java
\u2502�� \u2514\u2500\u2500 WordCount.java
\u2514\u2500\u2500 resources
\u2514\u2500\u2500 log4j.properties
@@ -79,7 +81,7 @@ $ rm wiki-edits/src/main/java/wikiedits/*.java
{% endhighlight %}
As a last step we need to add the Flink Wikipedia connector as a dependency so that we can
-use it in our program. Edit the `dependencies` section so that it looks like this:
+use it in our program. Edit the `dependencies` section of the `pom.xml` so that it looks like this:
{% highlight xml %}
<dependencies>
@@ -90,23 +92,23 @@ use it in our program. Edit the `dependencies` section so that it looks like thi
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_2.10</artifactId>
+ <artifactId>flink-streaming-java_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-clients_2.10</artifactId>
+ <artifactId>flink-clients_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-wikiedits_2.10</artifactId>
+ <artifactId>flink-connector-wikiedits_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
</dependencies>
{% endhighlight %}
-Notice the `flink-connector-wikiedits_2.10` dependency that was added. (This example and
+Notice the `flink-connector-wikiedits_2.11` dependency that was added. (This example and
the Wikipedia connector were inspired by the *Hello Samza* example of Apache Samza.)
## Writing a Flink Program
@@ -295,7 +297,7 @@ use the Kafka sink. Add this to the `pom.xml` file in the dependencies section:
{% highlight xml %}
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-kafka-0.8_2.10</artifactId>
+ <artifactId>flink-connector-kafka-0.8_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
{% endhighlight %}
http://git-wip-us.apache.org/repos/asf/flink/blob/67c4be64/docs/quickstart/scala_api_quickstart.md
----------------------------------------------------------------------
diff --git a/docs/quickstart/scala_api_quickstart.md b/docs/quickstart/scala_api_quickstart.md
index 388f0a7..8ee1e66 100644
--- a/docs/quickstart/scala_api_quickstart.md
+++ b/docs/quickstart/scala_api_quickstart.md
@@ -136,14 +136,19 @@ Use one of the following commands to __create a project__:
{% highlight bash %}
$ mvn archetype:generate \
-DarchetypeGroupId=org.apache.flink \
- -DarchetypeArtifactId=flink-quickstart-scala \
+ -DarchetypeArtifactId=flink-quickstart-scala \{% unless site.is_stable %}
+ -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %}
-DarchetypeVersion={{site.version}}
{% endhighlight %}
This allows you to <strong>name your newly created project</strong>. It will interactively ask you for the groupId, artifactId, and package name.
</div>
<div class="tab-pane" id="quickstart-script">
{% highlight bash %}
-$ curl https://flink.apache.org/q/quickstart-scala.sh | bash
+{% if site.is_stable %}
+ $ curl https://flink.apache.org/q/quickstart-scala.sh | bash
+{% else %}
+ $ curl https://flink.apache.org/q/quickstart-scala-SNAPSHOT.sh | bash
+{% endif %}
{% endhighlight %}
</div>
</div>
@@ -151,34 +156,63 @@ $ curl https://flink.apache.org/q/quickstart-scala.sh | bash
### Inspect Project
-There will be a new directory in your working directory. If you've used the _curl_ approach, the directory is called `quickstart`. Otherwise, it has the name of your artifactId.
+There will be a new directory in your working directory. If you've used
+the _curl_ approach, the directory is called `quickstart`. Otherwise,
+it has the name of your `artifactId`:
+
+{% highlight bash %}
+$ tree quickstart/
+quickstart/
+\u251c\u2500\u2500 pom.xml
+\u2514\u2500\u2500 src
+ \u2514\u2500\u2500 main
+ \u251c\u2500\u2500 resources
+ \u2502�� \u2514\u2500\u2500 log4j.properties
+ \u2514\u2500\u2500 scala
+ \u2514\u2500\u2500 org
+ \u2514\u2500\u2500 myorg
+ \u2514\u2500\u2500 quickstart
+ \u251c\u2500\u2500 BatchJob.scala
+ \u251c\u2500\u2500 SocketTextStreamWordCount.scala
+ \u251c\u2500\u2500 StreamingJob.scala
+ \u2514\u2500\u2500 WordCount.scala
+{% endhighlight %}
The sample project is a __Maven project__, which contains four classes. _StreamingJob_ and _BatchJob_ are basic skeleton programs, _SocketTextStreamWordCount_ is a working streaming example and _WordCountJob_ is a working batch example. Please note that the _main_ method of all classes allow you to start Flink in a development/testing mode.
We recommend you __import this project into your IDE__. For Eclipse, you need the following plugins, which you can install from the provided Eclipse Update Sites:
* _Eclipse 4.x_
- * [Scala IDE](http://download.scala-ide.org/sdk/e38/scala210/stable/site)
+ * [Scala IDE](http://download.scala-ide.org/sdk/lithium/e44/scala211/stable/site)
* [m2eclipse-scala](http://alchim31.free.fr/m2e-scala/update-site)
- * [Build Helper Maven Plugin](https://repository.sonatype.org/content/repositories/forge-sites/m2e-extras/0.15.0/N/0.15.0.201206251206/)
-* _Eclipse 3.7_
- * [Scala IDE](http://download.scala-ide.org/sdk/e37/scala210/stable/site)
+ * [Build Helper Maven Plugin](https://repo1.maven.org/maven2/.m2e/connectors/m2eclipse-buildhelper/0.15.0/N/0.15.0.201207090124/)
+* _Eclipse 3.8_
+ * [Scala IDE for Scala 2.11](http://download.scala-ide.org/sdk/helium/e38/scala211/stable/site) or [Scala IDE for Scala 2.10](http://download.scala-ide.org/sdk/helium/e38/scala210/stable/site)
* [m2eclipse-scala](http://alchim31.free.fr/m2e-scala/update-site)
* [Build Helper Maven Plugin](https://repository.sonatype.org/content/repositories/forge-sites/m2e-extras/0.14.0/N/0.14.0.201109282148/)
-The IntelliJ IDE also supports Maven and offers a plugin for Scala development.
+The IntelliJ IDE supports Maven out of the box and offers a plugin for
+Scala development.
### Build Project
-If you want to __build your project__, go to your project directory and issue the `mvn clean package -Pbuild-jar` command. You will __find a jar__ that runs on every Flink cluster in __target/your-artifact-id-{{ site.version }}.jar__. There is also a fat-jar, __target/your-artifact-id-{{ site.version }}-flink-fat-jar.jar__. This
-also contains all dependencies that get added to the maven project.
+If you want to __build your project__, go to your project directory and
+issue the `mvn clean package -Pbuild-jar` command. You will
+__find a jar__ that runs on every Flink cluster with a compatible
+version, __target/original-your-artifact-id-your-version.jar__. There
+is also a fat-jar in __target/your-artifact-id-your-version.jar__ which,
+additionally, contains all dependencies that were added to the Maven
+project.
## Next Steps
Write your application!
-The quickstart project contains a WordCount implementation, the "Hello World" of Big Data processing systems. The goal of WordCount is to determine the frequencies of words in a text, e.g., how often do the terms "the" or "house" occur in all Wikipedia texts.
+The quickstart project contains a `WordCount` implementation, the
+"Hello World" of Big Data processing systems. The goal of `WordCount`
+is to determine the frequencies of words in a text, e.g., how often do
+the terms "the" or "house" occur in all Wikipedia texts.
__Sample Input__:
@@ -194,7 +228,10 @@ data 1
is 1
~~~
-The following code shows the WordCount implementation from the Quickstart which processes some text lines with two operators (FlatMap and Reduce), and prints the resulting words and counts to std-out.
+The following code shows the `WordCount` implementation from the
+Quickstart which processes some text lines with two operators (a FlatMap
+and a Reduce operation via aggregating a sum), and prints the resulting
+words and counts to std-out.
~~~scala
object WordCountJob {
@@ -213,7 +250,7 @@ object WordCountJob {
.groupBy(0)
.sum(1)
- // emit result
+ // emit result and print result
counts.print()
}
}
@@ -221,4 +258,10 @@ object WordCountJob {
{% gh_link flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala/wordcount/WordCount.scala "Check GitHub" %} for the full example code.
-For a complete overview over our API, have a look at the [DataStream API]({{ site.baseurl }}/dev/datastream_api.html) and [DataSet API]({{ site.baseurl }}/dev/batch/index.html) sections. If you have any trouble, ask on our [Mailing List](http://mail-archives.apache.org/mod_mbox/flink-dev/). We are happy to provide help.
+For a complete overview over our API, have a look at the
+[DataStream API]({{ site.baseurl }}/dev/datastream_api.html),
+[DataSet API]({{ site.baseurl }}/dev/batch/index.html), and
+[Scala API Extensions]({{ site.baseurl }}/dev/scala_api_extensions.html)
+sections. If you have any trouble, ask on our
+[Mailing List](http://mail-archives.apache.org/mod_mbox/flink-dev/).
+We are happy to provide help.
http://git-wip-us.apache.org/repos/asf/flink/blob/67c4be64/docs/quickstart/setup_quickstart.md
----------------------------------------------------------------------
diff --git a/docs/quickstart/setup_quickstart.md b/docs/quickstart/setup_quickstart.md
index 6aab686..0628176 100644
--- a/docs/quickstart/setup_quickstart.md
+++ b/docs/quickstart/setup_quickstart.md
@@ -48,21 +48,34 @@ Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
### Download
-Download a binary from the [downloads page](http://flink.apache.org/downloads.html). You can pick
-any Hadoop/Scala combination you like. If you plan to just use the local file system, any Hadoop
-version will work fine.
-
-### Start a Local Flink Cluster
-
-1. Go to the download directory.
-2. Unpack the downloaded archive.
-3. Start Flink.
+{% if site.is_stable %}
+1. Download a binary from the [downloads page](http://flink.apache.org/downloads.html). You can pick
+ any Hadoop/Scala combination you like. If you plan to just use the local file system, any Hadoop
+ version will work fine.
+2. Go to the download directory.
+3. Unpack the downloaded archive.
~~~bash
$ cd ~/Downloads # Go to download directory
$ tar xzf flink-*.tgz # Unpack the downloaded archive
$ cd flink-{{site.version}}
-$ bin/start-local.sh # Start Flink
+~~~
+{% else %}
+Clone the source code from one of our
+[repositories](http://flink.apache.org/community.html#source-code), e.g.:
+
+~~~bash
+$ git clone https://github.com/apache/flink.git
+$ cd flink
+$ mvn clean package -DskipTests # this will take up to 10 minutes
+$ cd build-target # this is where Flink is installed to
+~~~
+{% endif %}
+
+### Start a Local Flink Cluster
+
+~~~bash
+$ ./bin/start-local.sh # Start Flink
~~~
Check the __JobManager's web frontend__ at [http://localhost:8081](http://localhost:8081) and make sure everything is up and running. The web frontend should report a single available TaskManager instance.
@@ -89,7 +102,7 @@ You can find the complete source code for this SocketWindowWordCount example in
object SocketWindowWordCount {
def main(args: Array[String]) : Unit = {
-
+
// the port to connect to
val port: Int = try {
ParameterTool.fromArgs(args).getInt("port")
@@ -102,11 +115,11 @@ object SocketWindowWordCount {
// get the execution environment
val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment
-
+
// get input data by connecting to the socket
val text = env.socketTextStream("localhost", port, '\n')
- // parse the data, group it, window it, and aggregate the counts
+ // parse the data, group it, window it, and aggregate the counts
val windowCounts = text
.flatMap { w => w.split("\\s") }
.map { w => WordWithCount(w, 1) }
@@ -197,7 +210,10 @@ public class SocketWindowWordCount {
## Run the Example
-Now, we are going to run this Flink application. It will read text from a socket and once a second print the number of occurances of each distinct word during the previous 5 seconds.
+Now, we are going to run this Flink application. It will read text from
+a socket and once every 5 seconds print the number of occurrences of
+each distinct word during the previous 5 seconds, i.e. a tumbling
+window of processing time, as long as words are floating in.
* First of all, we use **netcat** to start local server via
@@ -208,15 +224,21 @@ Now, we are going to run this Flink application. It will read text from a socket
* Submit the Flink program:
~~~bash
- $ bin/flink run examples/streaming/SocketWindowWordCount.jar --port 9000
-
- 03/08/2016 17:21:56 Job execution switched to status RUNNING.
- 03/08/2016 17:21:56 Source: Socket Stream -> Flat Map(1/1) switched to SCHEDULED
- 03/08/2016 17:21:56 Source: Socket Stream -> Flat Map(1/1) switched to DEPLOYING
- 03/08/2016 17:21:56 Keyed Aggregation -> Sink: Unnamed(1/1) switched to SCHEDULED
- 03/08/2016 17:21:56 Keyed Aggregation -> Sink: Unnamed(1/1) switched to DEPLOYING
- 03/08/2016 17:21:56 Source: Socket Stream -> Flat Map(1/1) switched to RUNNING
- 03/08/2016 17:21:56 Keyed Aggregation -> Sink: Unnamed(1/1) switched to RUNNING
+ $ ./bin/flink run examples/streaming/SocketWindowWordCount.jar --port 9000
+
+ Cluster configuration: Standalone cluster with JobManager at /127.0.0.1:6123
+ Using address 127.0.0.1:6123 to connect to JobManager.
+ JobManager web interface address http://127.0.0.1:8081
+ Starting execution of program
+ Submitting job with JobID: 574a10c8debda3dccd0c78a3bde55e1b. Waiting for job completion.
+ Connected to JobManager at Actor[akka.tcp://flink@127.0.0.1:6123/user/jobmanager#297388688]
+ 11/04/2016 14:04:50 Job execution switched to status RUNNING.
+ 11/04/2016 14:04:50 Source: Socket Stream -> Flat Map(1/1) switched to SCHEDULED
+ 11/04/2016 14:04:50 Source: Socket Stream -> Flat Map(1/1) switched to DEPLOYING
+ 11/04/2016 14:04:50 Fast TumblingProcessingTimeWindows(5000) of WindowedStream.main(SocketWindowWordCount.java:79) -> Sink: Unnamed(1/1) switched to SCHEDULED
+ 11/04/2016 14:04:51 Fast TumblingProcessingTimeWindows(5000) of WindowedStream.main(SocketWindowWordCount.java:79) -> Sink: Unnamed(1/1) switched to DEPLOYING
+ 11/04/2016 14:04:51 Fast TumblingProcessingTimeWindows(5000) of WindowedStream.main(SocketWindowWordCount.java:79) -> Sink: Unnamed(1/1) switched to RUNNING
+ 11/04/2016 14:04:51 Source: Socket Stream -> Flat Map(1/1) switched to RUNNING
~~~
The program connects to the socket and waits for input. You can check the web interface to verify that the job is running as expected:
@@ -230,7 +252,10 @@ Now, we are going to run this Flink application. It will read text from a socket
</div>
</div>
-* Counts are printed to `stdout`. Monitor the JobManager's output file and write some text in `nc`:
+* Words are counted in time windows of 5 seconds (processing time, tumbling
+ windows) and are printed to `stdout`. Monitor the JobManager's output file
+ and write some text in `nc` (input is sent to Flink line by line after
+ hitting <RETURN>):
~~~bash
$ nc -l 9000
@@ -239,26 +264,22 @@ Now, we are going to run this Flink application. It will read text from a socket
bye
~~~
- The `.out` file will print the counts immediately:
+ The `.out` file will print the counts at the end of each time window as long
+ as words are floating in, e.g.:
~~~bash
$ tail -f log/flink-*-jobmanager-*.out
- (lorem,1)
- (ipsum,1)
- (ipsum,2)
- (ipsum,3)
- (ipsum,4)
- (bye,1)
+ lorem : 1
+ bye : 1
+ ipsum : 4
~~~~
To **stop** Flink when you're done type:
~~~bash
- $ bin/stop-local.sh
+ $ ./bin/stop-local.sh
~~~
- <a href="{{ site.baseurl }}/page/img/quickstart-setup/setup.gif" ><img class="img-responsive" src="{{ site.baseurl }}/page/img/quickstart-setup/setup.gif" alt="Quickstart: Setup"/></a>
-
## Next Steps
Check out some more [examples]({{ site.baseurl }}/examples) to get a better feel for Flink's programming APIs. When you are done with that, go ahead and read the [streaming guide]({{ site.baseurl }}/dev/datastream_api.html).
http://git-wip-us.apache.org/repos/asf/flink/blob/67c4be64/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/socket/SocketWindowWordCount.java
----------------------------------------------------------------------
diff --git a/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/socket/SocketWindowWordCount.java b/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/socket/SocketWindowWordCount.java
index fe1b6e7..dd2e061 100644
--- a/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/socket/SocketWindowWordCount.java
+++ b/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/socket/SocketWindowWordCount.java
@@ -74,7 +74,7 @@ public class SocketWindowWordCount {
})
.keyBy("word")
- .timeWindow(Time.seconds(5), Time.seconds(1))
+ .timeWindow(Time.seconds(5))
.reduce(new ReduceFunction<WordWithCount>() {
@Override
@@ -111,4 +111,4 @@ public class SocketWindowWordCount {
return word + " : " + count;
}
}
-}
\ No newline at end of file
+}
http://git-wip-us.apache.org/repos/asf/flink/blob/67c4be64/flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples/socket/SocketWindowWordCount.scala
----------------------------------------------------------------------
diff --git a/flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples/socket/SocketWindowWordCount.scala b/flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples/socket/SocketWindowWordCount.scala
index 3b432ec..1761b84 100644
--- a/flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples/socket/SocketWindowWordCount.scala
+++ b/flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples/socket/SocketWindowWordCount.scala
@@ -62,7 +62,7 @@ object SocketWindowWordCount {
.flatMap { w => w.split("\\s") }
.map { w => WordWithCount(w, 1) }
.keyBy("word")
- .timeWindow(Time.seconds(5), Time.seconds(1))
+ .timeWindow(Time.seconds(5))
.sum("count")
// print the results with a single thread, rather than in parallel