You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by rm...@apache.org on 2014/07/14 16:10:07 UTC
svn commit: r1610413 [1/4] - in /incubator/flink: ./ _layouts/ site/
site/docs/0.6-SNAPSHOT/ site/docs/0.6-SNAPSHOT/img/
Author: rmetzger
Date: Mon Jul 14 14:10:06 2014
New Revision: 1610413
URL: http://svn.apache.org/r1610413
Log:
Add eclipse instructions to 'how to contrib', update documentation (it now reflects the new name)
Added:
incubator/flink/site/docs/0.6-SNAPSHOT/building.html
incubator/flink/site/docs/0.6-SNAPSHOT/img/FlinkOnYarn.svg (with props)
Modified:
incubator/flink/_layouts/docs.html
incubator/flink/how-to-contribute.md
incubator/flink/site/docs/0.6-SNAPSHOT/build_docs.sh
incubator/flink/site/docs/0.6-SNAPSHOT/cli.html
incubator/flink/site/docs/0.6-SNAPSHOT/cluster_execution.html
incubator/flink/site/docs/0.6-SNAPSHOT/cluster_setup.html
incubator/flink/site/docs/0.6-SNAPSHOT/coding_guidelines.html
incubator/flink/site/docs/0.6-SNAPSHOT/config.html
incubator/flink/site/docs/0.6-SNAPSHOT/faq.html
incubator/flink/site/docs/0.6-SNAPSHOT/hadoop_compatability.html
incubator/flink/site/docs/0.6-SNAPSHOT/how_to_contribute.html
incubator/flink/site/docs/0.6-SNAPSHOT/index.html
incubator/flink/site/docs/0.6-SNAPSHOT/internal_add_operator.html
incubator/flink/site/docs/0.6-SNAPSHOT/internal_distributed_runtime.html
incubator/flink/site/docs/0.6-SNAPSHOT/internal_general_arch.html
incubator/flink/site/docs/0.6-SNAPSHOT/internal_job_scheduling.html
incubator/flink/site/docs/0.6-SNAPSHOT/internal_operators_and_memory.html
incubator/flink/site/docs/0.6-SNAPSHOT/internal_optimizer.html
incubator/flink/site/docs/0.6-SNAPSHOT/internal_overview.html
incubator/flink/site/docs/0.6-SNAPSHOT/internal_program_life_cycle.html
incubator/flink/site/docs/0.6-SNAPSHOT/iterations.html
incubator/flink/site/docs/0.6-SNAPSHOT/java_api_examples.html
incubator/flink/site/docs/0.6-SNAPSHOT/java_api_guide.html
incubator/flink/site/docs/0.6-SNAPSHOT/java_api_quickstart.html
incubator/flink/site/docs/0.6-SNAPSHOT/local_execution.html
incubator/flink/site/docs/0.6-SNAPSHOT/local_setup.html
incubator/flink/site/docs/0.6-SNAPSHOT/run_example_quickstart.html
incubator/flink/site/docs/0.6-SNAPSHOT/scala_api_examples.html
incubator/flink/site/docs/0.6-SNAPSHOT/scala_api_guide.html
incubator/flink/site/docs/0.6-SNAPSHOT/scala_api_quickstart.html
incubator/flink/site/docs/0.6-SNAPSHOT/setup_quickstart.html
incubator/flink/site/docs/0.6-SNAPSHOT/spargel_guide.html
incubator/flink/site/docs/0.6-SNAPSHOT/web_client.html
incubator/flink/site/docs/0.6-SNAPSHOT/yarn_setup.html
incubator/flink/site/how-to-contribute.html
Modified: incubator/flink/_layouts/docs.html
URL: http://svn.apache.org/viewvc/incubator/flink/_layouts/docs.html?rev=1610413&r1=1610412&r2=1610413&view=diff
==============================================================================
--- incubator/flink/_layouts/docs.html (original)
+++ incubator/flink/_layouts/docs.html Mon Jul 14 14:10:06 2014
@@ -18,6 +18,7 @@ layout: default
<li>Setup & Configuration
<ul>
<li><a href="local_setup.html">Local Setup</a></li>
+ <li><a href="building.html">Build Flink</a></li>
<li><a href="cluster_setup.html">Cluster Setup</a></li>
<li><a href="yarn_setup.html">YARN Setup</a></li>
<li><a href="config.html">Configuration</a></li>
Modified: incubator/flink/how-to-contribute.md
URL: http://svn.apache.org/viewvc/incubator/flink/how-to-contribute.md?rev=1610413&r1=1610412&r2=1610413&view=diff
==============================================================================
--- incubator/flink/how-to-contribute.md (original)
+++ incubator/flink/how-to-contribute.md Mon Jul 14 14:10:06 2014
@@ -38,7 +38,26 @@ git clone https://github.com/<your-user-
git checkout -b myBranch master
```
-4. Now you can create your changes, compile the code, and validate the changes. Here are some pointers on how to [set up the Eclipse IDE for development](https://github.com/apache/incubator-flink/#eclipse-setup-and-debugging), and how to [build the code](https://github.com/apache/incubator-flink/#build-stratosphere).
+4. Now you can create your changes, compile the code, and validate the changes. Here are some pointers on how to [build the code](https://github.com/apache/incubator-flink/#build-apache-flink).
+In addition to that, we recommend setting up Eclipse (or IntelliJ) using the "Import Maven Project" feature. If you want to work on the scala code you will need the following plugins:
+
+ Eclipse 4.x:
+ * scala-ide: http://download.scala-ide.org/sdk/e38/scala210/stable/site
+ * m2eclipse-scala: http://alchim31.free.fr/m2e-scala/update-site
+ * build-helper-maven-plugin: https://repository.sonatype.org/content/repositories/forge-sites/m2e-extras/0.15.0/N/0.15.0.201206251206/
+
+ Eclipse 3.7:
+ * scala-ide: http://download.scala-ide.org/sdk/e37/scala210/stable/site
+ * m2eclipse-scala: http://alchim31.free.fr/m2e-scala/update-site
+ * build-helper-maven-plugin: https://repository.sonatype.org/content/repositories/forge-sites/m2e-extras/0.14.0/N/0.14.0.201109282148/
+
+ When you don't have the plugins your project will have build errors, you can just close the scala projects and ignore them.
+
+ Import the Flink source code using Maven's Import tool:
+ * Select "Import" from the "File"-menu.
+ * Expand "Maven" node, select "Existing Maven Projects", and click "next" button
+ * Select the root directory by clicking on the "Browse" button and navigate to the top folder of the cloned Flink git repository.
+ * Ensure that all projects are selected and click the "Finish" button.
5. After you have finalized your contribution, verify the compliance with the contribution guidelines (see below), and commit them. To make the changes easily mergeable, please rebase them to the latest version of the main repositories master branch. Assuming you created a topic branch (step 3), you can follow this sequence of commands to do that:
Switch to the master branch, update it to the latest revision, switch back to your topic branch, and rebase it on top of the master branch.
@@ -96,5 +115,7 @@ Only the infrastructure team of the ASF
**ASF git web interface**: [https://git-wip-us.apache.org/repos/asf?p=incubator-flink.git;a=summary](https://git-wip-us.apache.org/repos/asf?p=incubator-flink.git;a=summary)
+**ASF svn for the website**: [https://svn.apache.org/repos/asf/incubator/flink/](https://svn.apache.org/repos/asf/incubator/flink/).
+
Details on how to set the credentials for the ASF git repostiory are [linked here](https://git-wip-us.apache.org/).
To merge pull requests from our GitHub mirror, there is a script in the source `./tools/merge_pull_request.sh.template`. Rename it to `merge_pull_request.sh` with the appropriate settings and use it for merging.
Modified: incubator/flink/site/docs/0.6-SNAPSHOT/build_docs.sh
URL: http://svn.apache.org/viewvc/incubator/flink/site/docs/0.6-SNAPSHOT/build_docs.sh?rev=1610413&r1=1610412&r2=1610413&view=diff
==============================================================================
--- incubator/flink/site/docs/0.6-SNAPSHOT/build_docs.sh (original)
+++ incubator/flink/site/docs/0.6-SNAPSHOT/build_docs.sh Mon Jul 14 14:10:06 2014
@@ -1,16 +1,22 @@
#!/bin/bash
-########################################################################################################################
-# Copyright (C) 2010-2014 by the Stratos phere project (http://stratosphere.eu)
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
-# an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations under the License.
-########################################################################################################################
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
HAS_JEKYLL=true
Added: incubator/flink/site/docs/0.6-SNAPSHOT/building.html
URL: http://svn.apache.org/viewvc/incubator/flink/site/docs/0.6-SNAPSHOT/building.html?rev=1610413&view=auto
==============================================================================
--- incubator/flink/site/docs/0.6-SNAPSHOT/building.html (added)
+++ incubator/flink/site/docs/0.6-SNAPSHOT/building.html Mon Jul 14 14:10:06 2014
@@ -0,0 +1,270 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Apache Flink (incubating): Build Flink</title>
+ <link rel="stylesheet" href="/css/bootstrap.css">
+ <link rel="stylesheet" href="/css/bootstrap-lumen-custom.css">
+ <link href="//maxcdn.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.min.css" rel="stylesheet">
+ </head>
+ <body>
+
+<nav class="navbar navbar-default navbar-static-top" role="navigation">
+ <div class="container">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a class="navbar-brand" href="/index.html">Apache Flink</a>
+ </div>
+
+ <div class="collapse navbar-collapse" id="navbar-collapse-1">
+ <ul class="nav navbar-nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Quickstart <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="/docs/0.6-SNAPSHOT/setup_quickstart.html">Setup Flink</a></li>
+ <li><a href="/docs/0.6-SNAPSHOT/java_api_quickstart.html">Java API</a></li>
+ <li><a href="/docs/0.6-SNAPSHOT/scala_api_quickstart.html">Scala API</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="/downloads.html" class="">Downloads</a>
+ </li>
+
+ <li>
+ <a href="/docs/0.6-SNAPSHOT/faq.html" class="">FAQ</a>
+ </li>
+
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="/docs/0.6-SNAPSHOT/">0.6-SNAPSHOT</a></li>
+ <li><a href="http://stratosphere-javadocs.github.io/">0.6-SNAPSHOT Javadocs</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="/community.html#mailing-lists">Mailing Lists</a></li>
+ <li><a href="/community.html#issues">Issues</a></li>
+ <li><a href="/community.html#team">Team</a></li>
+ <li class="divider"></li>
+ <li><a href="/how-to-contribute.html">How To Contribute</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/">Apache Software Foundation</a>
+ <li><a href="http://www.apache.org/foundation/how-it-works.html">How it works</a>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Become a Sponsor</a>
+ <li><a href="http://incubator.apache.org/projects/flink.html">Incubation Status page</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <!--<li><a href="/project.html#history">History</a></li> -->
+ <li><a href="https://wiki.apache.org/incubator/StratosphereProposal">Incubator Proposal (external)</a></li>
+ <li><a href="http://www.apache.org/licenses/LICENSE-2.0">License</a></li>
+ <li><a href="https://github.com/apache/incubator-flink">Source Code</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="/blog/index.html" class="">Blog</a>
+ </li>
+
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container">
+
+<div class="row">
+ <div class="col-md-3">
+ <ul>
+ <li><a href="faq.html">FAQ</a></li>
+ <li>Quickstart
+ <ul>
+ <li><a href="setup_quickstart.html">Setup</a></li>
+ <li><a href="run_example_quickstart.html">Run Example</a></li>
+ <li><a href="java_api_quickstart.html">Java API</a></li>
+ <li><a href="scala_api_quickstart.html">Scala API</a></li>
+ </ul>
+ </li>
+
+ <li>Setup & Configuration
+ <ul>
+ <li><a href="local_setup.html">Local Setup</a></li>
+ <li><a href="building.html">Build Flink</a></li>
+ <li><a href="cluster_setup.html">Cluster Setup</a></li>
+ <li><a href="yarn_setup.html">YARN Setup</a></li>
+ <li><a href="config.html">Configuration</a></li>
+ </ul>
+ </li>
+
+ <li>Programming Guides
+ <ul>
+ <li><a href="java_api_guide.html">Java API</a></li>
+ <li><a href="scala_api_guide.html">Scala API</a></li>
+ <li><a href="hadoop_compatability.html">Hadoop Compatability</a></li>
+ <li><a href="iterations.html">Iterations</a></li>
+ <li><a href="spargel_guide.html">Spargel Graph API</a></li>
+ </ul>
+ </li>
+
+ <li>Examples
+ <ul>
+ <li><a href="java_api_examples.html">Java API</a></li>
+ <li><a href="scala_api_examples.html">Scala API</a></li>
+ </ul>
+ </li>
+
+ <li>Execution
+ <ul>
+ <li><a href="local_execution.html">Local/Debugging</a></li>
+ <li><a href="cluster_execution.html">Cluster</a></li>
+ <li><a href="cli.html">Command-Line Interface</a></li>
+ <li><a href="web_client.html">Web Interface</a></li>
+ </ul>
+ </li>
+
+ <li>Internals
+ <ul>
+ <li><a href="internal_overview.html">Overview</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+ <div class="col-md-9">
+ <h1>Build Flink</h1>
+
+ <ul>
+<li>
+<a href="#build-flink-for-a-specific-hadoop-version">Build Flink for a specific Hadoop Version</a>
+<ul>
+<li>
+<a href="#background">Background</a>
+</li>
+<li>
+<a href="#example-for-cloudera-hadoop-5-beta-2">Example for Cloudera Hadoop 5 Beta 2</a>
+</li>
+</ul>
+</li>
+</ul>
+
+
+ <p>In order to build Flink, you need the source code. Either download the source of a release or clone the git repository. In addition to that, you need Maven 3 and a JDK. Note that you can not build Flink with Oracle JDK 6 due to a unresolved bug in the Java compiler. It works well with OpenJDK 6 and all Java 7 and 8 compilers.</p>
+
+<p>To clone from git, enter:</p>
+<div class="highlight"><pre><code class="language-text" data-lang="text">git clone https://github.com/apache/incubator-flink
+</code></pre></div>
+<p>The simplest way of building Flink is by running:</p>
+<div class="highlight"><pre><code class="language-text" data-lang="text">mvn clean package -DskipTests
+</code></pre></div>
+<p>This instructs Maven (<code>mvn</code>) to first remove all existing builds (<code>clean</code>) and then create a new Flink binary (<code>package</code>). The <code>-DskipTests</code> command prevents Maven from executing the unit tests.</p>
+
+<h2 id="build-flink-for-a-specific-hadoop-version">Build Flink for a specific Hadoop Version</h2>
+
+<p>This section covers building Flink for a specific Hadoop version. Most users do not need to do this manually.
+The problem is that Flink uses HDFS and YARN which are both from Apache Hadoop. There exist many different builds of Hadoop (from both the upstream project and the different Hadoop distributions). Typically errors arise with the RPC services. An error could look like this:</p>
+<div class="highlight"><pre><code class="language-text" data-lang="text">ERROR: The job was not successfully submitted to the nephele job manager:
+ org.apache.flink.nephele.executiongraph.GraphConversionException: Cannot compute input splits for TSV:
+ java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException:
+ Protocol message contained an invalid tag (zero).; Host Details :
+</code></pre></div>
+<h3 id="background">Background</h3>
+
+<p>The builds with Maven are controlled by <a href="http://maven.apache.org/pom.html#Properties">properties</a> and <a href="http://maven.apache.org/guides/introduction/introduction-to-profiles.html">build profiles</a>.
+There are two profiles, one for hadoop1 and one for hadoop2. When the hadoop2 profile is enabled, the system will also build the YARN client.
+The hadoop1 profile is used by default. To enable the hadoop2 profile, set <code>-Dhadoop.profile=2</code> when building.
+Depending on the profile, there are two Hadoop versions, set via properties. For "hadoop1", we use 1.2.1 by default, for "hadoop2" it is 2.2.0.</p>
+
+<p>You can change these versions with the <code>hadoop-two.version</code> (or <code>hadoop-one.version</code>) property. For example <code>-Dhadoop-two.version=2.4.0</code>.</p>
+
+<h3 id="example-for-cloudera-hadoop-5-beta-2">Example for Cloudera Hadoop 5 Beta 2</h3>
+<div class="highlight"><pre><code class="language-text" data-lang="text">mvn -Dhadoop.profile=2 -Pvendor-repos -Dhadoop.version=2.2.0-cdh5.0.0-beta-2 -DskipTests package
+</code></pre></div>
+<p>The commands in detail:</p>
+
+<ul>
+<li> <code>-Dhadoop.profile=2</code> activates the Hadoop YARN profile of Flink. This will enable all components of Flink that are compatible with Hadoop 2.2</li>
+<li> <code>-Pvendor-repos</code> is adding the Maven repositories of MapR, Cloudera and Hortonworks into your Maven build.</li>
+<li><code>-Dhadoop.version=2.2.0-cdh5.0.0-beta-2</code> sets a special version of the Hadoop dependencies. Make sure that the specified Hadoop version is compatible with the profile you activated.</li>
+</ul>
+
+<p>If you want to build HDFS for Hadoop 2 without YARN, use the following parameter:</p>
+<div class="highlight"><pre><code class="language-text" data-lang="text">-P!include-yarn
+</code></pre></div>
+<p>Some Cloudera versions (such as <code>2.0.0-cdh4.2.0</code>) require this, since they have a new HDFS version with the old YARN API.</p>
+
+<p>Please post to the <em>Flink mailinglist</em>(<a href="mailto:dev@flink.incubator.apache.org">dev@flink.incubator.apache.org</a>) or create an issue on <a href="https://issues.apache.org/jira/browse/FLINK">Jira</a>, if you have issues with your YARN setup and Flink.</p>
+
+
+ <div style="padding-top:30px" id="disqus_thread"></div>
+<script type="text/javascript">
+ /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
+ var disqus_shortname = 'stratosphere-eu'; // required: replace example with your forum shortname
+
+ /* * * DON'T EDIT BELOW THIS LINE * * */
+ (function() {
+ var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
+ dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
+ (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
+ })();
+</script>
+<noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>
+<a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a>
+
+ </div>
+</div>
+
+ <div class="footer">
+
+<hr class="divider">
+
+<p><small>Apache Flink is an effort undergoing incubation at The Apache Software
+Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is
+required of all newly accepted projects until a further review indicates that
+the infrastructure, communications, and decision making process have
+stabilized in a manner consistent with other successful ASF projects. While
+incubation status is not necessarily a reflection of the completeness or
+stability of the code, it does indicate that the project has yet to be fully
+endorsed by the ASF.</small></p>
+
+<p><a href="http://incubator.apache.org/"><img src="/img/apache-incubator-logo.png" alt="Incubator Logo"></a></p>
+
+<p class="text-center"><a href="/privacy-policy.html">Privacy Policy<a></p>
+
+ </div>
+ </div>
+
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-52545728-1', 'auto');
+ ga('send', 'pageview');
+
+ </script>
+
+ </body>
+</html>
Modified: incubator/flink/site/docs/0.6-SNAPSHOT/cli.html
URL: http://svn.apache.org/viewvc/incubator/flink/site/docs/0.6-SNAPSHOT/cli.html?rev=1610413&r1=1610412&r2=1610413&view=diff
==============================================================================
--- incubator/flink/site/docs/0.6-SNAPSHOT/cli.html (original)
+++ incubator/flink/site/docs/0.6-SNAPSHOT/cli.html Mon Jul 14 14:10:06 2014
@@ -109,6 +109,7 @@
<li>Setup & Configuration
<ul>
<li><a href="local_setup.html">Local Setup</a></li>
+ <li><a href="building.html">Build Flink</a></li>
<li><a href="cluster_setup.html">Cluster Setup</a></li>
<li><a href="yarn_setup.html">YARN Setup</a></li>
<li><a href="config.html">Configuration</a></li>
@@ -161,16 +162,16 @@
</ul>
- <p>Stratosphere provides a command-line interface to run programs that are packaged
+ <p>Flink provides a command-line interface to run programs that are packaged
as JAR files, and control their execution. The command line interface is part
-of any Stratosphere setup, available in local single node setups and in
-distributed setups. It is located under <code><stratosphere-home>/bin/stratosphere</code>
-and connects by default to the running Stratosphere master (JobManager) that was
+of any Flink setup, available in local single node setups and in
+distributed setups. It is located under <code><flink-home>/bin/flink</code>
+and connects by default to the running Flink master (JobManager) that was
started from the same installation directory.</p>
-<p>A prerequisite to using the command line interface is that the Stratosphere
-master (JobManager) has been started (via <code><stratosphere-home>/bin/start-
-local.sh</code> or <code><stratosphere-home>/bin/start-cluster.sh</code>).</p>
+<p>A prerequisite to using the command line interface is that the Flink
+master (JobManager) has been started (via <code><flink-home>/bin/start-
+local.sh</code> or <code><flink-home>/bin/start-cluster.sh</code>).</p>
<p>The command line can be used to</p>
@@ -185,56 +186,56 @@ local.sh</code> or <code><stratospher
<ul>
<li><p>Run example program with no arguments.</p>
-<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/stratosphere run ./examples/stratosphere-java-examples-0.5.1-WordCount.jar
+<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/flink run ./examples/flink-java-examples-0.5.1-WordCount.jar
</code></pre></div></li>
<li><p>Run example program with arguments for input and result files</p>
-<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/stratosphere run ./examples/stratosphere-java-examples-0.5.1-WordCount.jar \
+<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/flink run ./examples/flink-java-examples-0.5.1-WordCount.jar \
file:///home/user/hamlet.txt file:///home/user/wordcount_out
</code></pre></div></li>
<li><p>Run example program with parallelism 16 and arguments for input and result files</p>
-<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/stratosphere run -p 16 ./examples/stratosphere-java-examples-0.5.1-WordCount.jar \
+<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/flink run -p 16 ./examples/flink-java-examples-0.5.1-WordCount.jar \
file:///home/user/hamlet.txt file:///home/user/wordcount_out
</code></pre></div></li>
<li><p>Run example program on a specific JobManager:</p>
-<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/stratosphere run -m myJMHost:6123 \
- ./examples/stratosphere-java-examples-0.5.1-WordCount.jar \
+<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/flink run -m myJMHost:6123 \
+ ./examples/flink-java-examples-0.5.1-WordCount.jar \
-file:///home/user/hamlet.txt file:///home/user/wordcount_out
</code></pre></div></li>
<li><p>Display the expected arguments for the WordCount example program:</p>
-<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/stratosphere info -d ./examples/stratosphere-java-examples-0.5.1-WordCount.jar
+<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/flink info -d ./examples/flink-java-examples-0.5.1-WordCount.jar
</code></pre></div></li>
<li><p>Display the optimized execution plan for the WordCount example program as JSON:</p>
-<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/stratosphere info -e
- ./examples/stratosphere-java-examples-0.5.1-WordCount.jar \
+<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/flink info -e
+ ./examples/flink-java-examples-0.5.1-WordCount.jar \
file:///home/user/hamlet.txt file:///home/user/wordcount_out
</code></pre></div></li>
<li><p>List scheduled and running jobs (including their JobIDs):</p>
-<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/stratosphere list -s -r
+<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/flink list -s -r
</code></pre></div></li>
<li><p>Cancel a job:</p>
-<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/stratosphere cancel -i <jobID>
+<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/flink cancel -i <jobID>
</code></pre></div></li>
</ul>
<h1 id="usage">Usage</h1>
<p>The command line syntax is as follows:</p>
-<div class="highlight"><pre><code class="language-text" data-lang="text">./stratosphere <ACTION> [OPTIONS] [ARGUMENTS]
+<div class="highlight"><pre><code class="language-text" data-lang="text">./flink <ACTION> [OPTIONS] [ARGUMENTS]
General options:
-h,--help Show the help for the CLI Frontend, or a specific action.
-v,--verbose Print more detailed error messages.
-Action "run" - compiles and submits a Stratosphere program that is given in the form of a JAR file.
+Action "run" - compiles and submits a Flink program that is given in the form of a JAR file.
"run" options:
- -p,--parallelism <parallelism> The degree of parallelism for the execution. This value is used unless the program overrides the degree of parallelism on the execution environment or program plan. If this option is not set, then the execution will use the default parallelism specified in the stratosphere-conf.yaml file.
+ -p,--parallelism <parallelism> The degree of parallelism for the execution. This value is used unless the program overrides the degree of parallelism on the execution environment or program plan. If this option is not set, then the execution will use the default parallelism specified in the flink-conf.yaml file.
-c,--class <classname> The class with the entry point (main method, or getPlan() method). Needs only be specified if the JAR file has no manifest pointing to that class. See program packaging instructions for details.
- -m,--jobmanager <host:port> Option to submit the program to a different Stratosphere master (JobManager).
+ -m,--jobmanager <host:port> Option to submit the program to a different Flink master (JobManager).
"run" arguments:
@@ -242,7 +243,7 @@ Action "run" - compiles and su
- All successive arguments are passed to the program's main method (or getPlan() method).
-Action "info" - displays information about a Stratosphere program.
+Action "info" - displays information about a Flink program.
"info" action arguments:
-d,--description Show description of the program, if the main class implements the 'ProgramDescription' interface.
@@ -253,7 +254,7 @@ Action "info" - displays infor
-c,--class <classname> The class with the entry point (main method, or getPlan() method). Needs only be specified if the JAR file has no manifest pointing to that class. See program packaging instructions for details.
- -m,--jobmanager <host:port> Option to connect to a different Stratosphere master (JobManager). Connecting to a master is relevant to compile the execution plan. The option is only evaluated if used together with the -e option.
+ -m,--jobmanager <host:port> Option to connect to a different Flink master (JobManager). Connecting to a master is relevant to compile the execution plan. The option is only evaluated if used together with the -e option.
"info" arguments:
@@ -261,7 +262,7 @@ Action "info" - displays infor
- All successive arguments are passed to the program's main method (or getPlan() method).
-Action "list" lists submitted Stratosphere programs.
+Action "list" lists submitted Flink programs.
"list" action arguments:
@@ -269,16 +270,16 @@ Action "list" lists submitted
-s,--scheduled Show scheduled programs and their JobIDs
- -m,--jobmanager <host:port> Option to connect to a different Stratosphere master (JobManager).
+ -m,--jobmanager <host:port> Option to connect to a different Flink master (JobManager).
-Action "cancel" cancels a submitted Stratosphere program.
+Action "cancel" cancels a submitted Flink program.
"cancel" action arguments:
-i,--jobid <jobID> JobID of program to cancel
- -m,--jobmanager <host:port> Option to connect to a different Stratosphere master (JobManager).
+ -m,--jobmanager <host:port> Option to connect to a different Flink master (JobManager).
</code></pre></div>
<div style="padding-top:30px" id="disqus_thread"></div>
Modified: incubator/flink/site/docs/0.6-SNAPSHOT/cluster_execution.html
URL: http://svn.apache.org/viewvc/incubator/flink/site/docs/0.6-SNAPSHOT/cluster_execution.html?rev=1610413&r1=1610412&r2=1610413&view=diff
==============================================================================
--- incubator/flink/site/docs/0.6-SNAPSHOT/cluster_execution.html (original)
+++ incubator/flink/site/docs/0.6-SNAPSHOT/cluster_execution.html Mon Jul 14 14:10:06 2014
@@ -109,6 +109,7 @@
<li>Setup & Configuration
<ul>
<li><a href="local_setup.html">Local Setup</a></li>
+ <li><a href="building.html">Build Flink</a></li>
<li><a href="cluster_setup.html">Cluster Setup</a></li>
<li><a href="yarn_setup.html">YARN Setup</a></li>
<li><a href="config.html">Configuration</a></li>
@@ -180,7 +181,7 @@
</ul>
- <p>Stratosphere programs can run distributed on clusters of many machines. There
+ <p>Flink programs can run distributed on clusters of many machines. There
are two ways to send a program to a cluster for execution:</p>
<h1 id="command-line-interface">Command Line Interface</h1>
@@ -193,17 +194,17 @@ details.</p>
<h1 id="remote-environment">Remote Environment</h1>
-<p>The remote environment lets you execute Stratosphere Java programs on a cluster
+<p>The remote environment lets you execute Flink Java programs on a cluster
directly. The remote environment points to the cluster on which you want to
execute the program.</p>
<h2 id="maven-dependency">Maven Dependency</h2>
<p>If you are developing your program as a Maven project, you have to add the
-<code>stratosphere-clients</code> module using this dependency:</p>
+<code>flink-clients</code> module using this dependency:</p>
<div class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>eu.stratosphere<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>stratosphere-clients<span class="nt"></artifactId></span>
+ <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
+ <span class="nt"><artifactId></span>flink-clients<span class="nt"></artifactId></span>
<span class="nt"><version></span>0.5.1<span class="nt"></version></span>
<span class="nt"></dependency></span>
</code></pre></div>
@@ -234,16 +235,16 @@ takes the path(s) to the JAR file(s).</p
<h1 id="remote-executor">Remote Executor</h1>
<p>Similar to the RemoteEnvironment, the RemoteExecutor lets you execute
-Stratosphere programs on a cluster directly. The remote executor accepts a
+Flink programs on a cluster directly. The remote executor accepts a
<em>Plan</em> object, which describes the program as a single executable unit.</p>
<h2 id="maven-dependency">Maven Dependency</h2>
<p>If you are developing your program in a Maven project, you have to add the
-<code>stratosphere-clients</code> module using this dependency:</p>
+<code>flink-clients</code> module using this dependency:</p>
<div class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt"><dependency></span>
- <span class="nt"><groupId></span>eu.stratosphere<span class="nt"></groupId></span>
- <span class="nt"><artifactId></span>stratosphere-clients<span class="nt"></artifactId></span>
+ <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span>
+ <span class="nt"><artifactId></span>flink-clients<span class="nt"></artifactId></span>
<span class="nt"><version></span>0.5.1<span class="nt"></version></span>
<span class="nt"></dependency></span>
</code></pre></div>
Modified: incubator/flink/site/docs/0.6-SNAPSHOT/cluster_setup.html
URL: http://svn.apache.org/viewvc/incubator/flink/site/docs/0.6-SNAPSHOT/cluster_setup.html?rev=1610413&r1=1610412&r2=1610413&view=diff
==============================================================================
--- incubator/flink/site/docs/0.6-SNAPSHOT/cluster_setup.html (original)
+++ incubator/flink/site/docs/0.6-SNAPSHOT/cluster_setup.html Mon Jul 14 14:10:06 2014
@@ -109,6 +109,7 @@
<li>Setup & Configuration
<ul>
<li><a href="local_setup.html">Local Setup</a></li>
+ <li><a href="building.html">Build Flink</a></li>
<li><a href="cluster_setup.html">Cluster Setup</a></li>
<li><a href="yarn_setup.html">YARN Setup</a></li>
<li><a href="config.html">Configuration</a></li>
@@ -178,7 +179,7 @@
</ul>
</li>
<li>
-<a href="#stratosphere-setup">Stratosphere Setup</a>
+<a href="#flink-setup">Flink Setup</a>
<ul>
<li>
<a href="#configuring-the-cluster">Configuring the Cluster</a>
@@ -190,7 +191,7 @@
<a href="#configuring-temporary-i/o-directories">Configuring Temporary I/O Directories</a>
</li>
<li>
-<a href="#starting-stratosphere">Starting Stratosphere</a>
+<a href="#starting-flink">Starting Flink</a>
</li>
</ul>
</li>
@@ -198,10 +199,10 @@
<p>This documentation is intended to provide instructions on how to run
-Stratosphere in a fully distributed fashion on a static (but possibly
+Flink in a fully distributed fashion on a static (but possibly
heterogeneous) cluster.</p>
-<p>This involves two steps. First, installing and configuring Stratosphere and
+<p>This involves two steps. First, installing and configuring Flink and
second installing and configuring the <a href="http://hadoop.apache.org/">Hadoop Distributed
Filesystem</a> (HDFS).</p>
@@ -209,14 +210,14 @@ Filesystem</a> (HDFS).</p>
<h2 id="software-requirements">Software Requirements</h2>
-<p>Stratosphere runs on all <em>UNIX-like environments</em>, e.g. <strong>Linux</strong>, <strong>Mac OS X</strong>,
+<p>Flink runs on all <em>UNIX-like environments</em>, e.g. <strong>Linux</strong>, <strong>Mac OS X</strong>,
and <strong>Cygwin</strong> (for Windows) and expects the cluster to consist of <strong>one master
node</strong> and <strong>one or more worker nodes</strong>. Before you start to setup the system,
make sure you have the following software installed <strong>on each node</strong>:</p>
<ul>
<li><strong>Java 1.6.x</strong> or higher,</li>
-<li><strong>ssh</strong> (sshd must be running to use the Stratosphere scripts that manage
+<li><strong>ssh</strong> (sshd must be running to use the Flink scripts that manage
remote components)</li>
</ul>
@@ -249,16 +250,16 @@ of the command on every host of your clu
<p>In order to start/stop the remote processes, the master node requires access via
ssh to the worker nodes. It is most convenient to use ssh's public key
authentication for this. To setup public key authentication, log on to the
-master as the user who will later execute all the Stratosphere components. <strong>The
+master as the user who will later execute all the Flink components. <strong>The
same user (i.e. a user with the same user name) must also exist on all worker
nodes</strong>. For the remainder of this instruction we will refer to this user as
-<em>stratosphere</em>. Using the super user <em>root</em> is highly discouraged for security
+<em>flink</em>. Using the super user <em>root</em> is highly discouraged for security
reasons.</p>
<p>Once you logged in to the master node as the desired user, you must generate a
new public/private key pair. The following command will create a new
public/private key pair into the <em>.ssh</em> directory inside the home directory of
-the user <em>stratosphere</em>. See the ssh-keygen man page for more details. Note that
+the user <em>flink</em>. See the ssh-keygen man page for more details. Note that
the private key is not protected by a passphrase.</p>
<div class="highlight"><pre><code class="language-text" data-lang="text">ssh-keygen -b 2048 -P '' -f ~/.ssh/id_rsa
</code></pre></div>
@@ -283,11 +284,11 @@ worker node from your master node via ss
<h2 id="setting-java_home-on-each-node">Setting JAVA_HOME on each Node</h2>
-<p>Stratosphere requires the <code>JAVA_HOME</code> environment variable to be set on the
+<p>Flink requires the <code>JAVA_HOME</code> environment variable to be set on the
master and all worker nodes and point to the directory of your Java
installation.</p>
-<p>You can set this variable in <code>conf/stratosphere-conf.yaml</code> via the
+<p>You can set this variable in <code>conf/flink-conf.yaml</code> via the
<code>env.java.home</code> key.</p>
<p>Alternatively, add the following line to your shell profile. If you use the
@@ -303,7 +304,7 @@ environments with the following commands
</code></pre></div>
<h1 id="hadoop-distributed-filesystem-(hdfs)-setup">Hadoop Distributed Filesystem (HDFS) Setup</h1>
-<p>The Stratosphere system currently uses the Hadoop Distributed Filesystem (HDFS)
+<p>The Flink system currently uses the Hadoop Distributed Filesystem (HDFS)
to read and write data in a distributed fashion.</p>
<p>Make sure to have a running HDFS installation. The following instructions are
@@ -315,7 +316,7 @@ many installation guides available onlin
<h2 id="downloading,-installing,-and-configuring-hdfs">Downloading, Installing, and Configuring HDFS</h2>
-<p>Similar to the Stratosphere system HDFS runs in a distributed fashion. HDFS
+<p>Similar to the Flink system HDFS runs in a distributed fashion. HDFS
consists of a <strong>NameNode</strong> which manages the distributed file system's meta
data. The actual data is stored by one or more <strong>DataNodes</strong>. For the remainder
of this instruction we assume the HDFS's NameNode component runs on the master
@@ -356,7 +357,7 @@ Guide</a> guide.</p>
<p>Replace <em>MASTER</em> with the IP/host name of your master node which runs the
<em>NameNode</em>. <em>DATAPATH</em> must be replaced with path to the directory in which the
actual HDFS data shall be stored on each worker node. Make sure that the
-<em>stratosphere</em> user has sufficient permissions to read and write in that
+<em>flink</em> user has sufficient permissions to read and write in that
directory.</p>
<p>After having saved the HDFS configuration file, open the file <em>conf/slaves</em> and
@@ -396,29 +397,29 @@ like to point you to the <a href="http:/
Start</a>
guide.</p>
-<h1 id="stratosphere-setup">Stratosphere Setup</h1>
+<h1 id="flink-setup">Flink Setup</h1>
<p>Go to the <a href="downloads/">downloads page</a> and get the ready to run
-package. Make sure to pick the Stratosphere package <strong>matching your Hadoop
+package. Make sure to pick the Flink package <strong>matching your Hadoop
version</strong>.</p>
<p>After downloading the latest release, copy the archive to your master node and
extract it:</p>
-<div class="highlight"><pre><code class="language-text" data-lang="text">tar xzf stratosphere-*.tgz
-cd stratosphere-*
+<div class="highlight"><pre><code class="language-text" data-lang="text">tar xzf flink-*.tgz
+cd flink-*
</code></pre></div>
<h2 id="configuring-the-cluster">Configuring the Cluster</h2>
-<p>After having extracted the system files, you need to configure Stratosphere for
-the cluster by editing <em>conf/stratosphere-conf.yaml</em>.</p>
+<p>After having extracted the system files, you need to configure Flink for
+the cluster by editing <em>conf/flink-conf.yaml</em>.</p>
<p>Set the <code>jobmanager.rpc.address</code> key to point to your master node. Furthermode
define the maximum amount of main memory the JVM is allowed to allocate on each
node by setting the <code>jobmanager.heap.mb</code> and <code>taskmanager.heap.mb</code> keys.</p>
<p>The value is given in MB. If some worker nodes have more main memory which you
-want to allocate to the Stratosphere system you can overwrite the default value
-by setting an environment variable <code>STRATOSPHERE_TM_HEAP</code> on the respective
+want to allocate to the Flink system you can overwrite the default value
+by setting an environment variable <code>FLINK_TM_HEAP</code> on the respective
node.</p>
<p>Finally you must provide a list of all nodes in your cluster which shall be used
@@ -434,9 +435,9 @@ will later run a TaskManager.</p>
.
192.168.0.150
</code></pre></div>
-<p>The Stratosphere directory must be available on every worker under the same
+<p>The Flink directory must be available on every worker under the same
path. Similarly as for HDFS, you can use a shared NSF directory, or copy the
-entire Stratosphere directory to every worker node.</p>
+entire Flink directory to every worker node.</p>
<h2 id="configuring-the-network-buffers">Configuring the Network Buffers</h2>
@@ -476,35 +477,35 @@ parameters:</p>
<h2 id="configuring-temporary-i/o-directories">Configuring Temporary I/O Directories</h2>
-<p>Although Stratosphere aims to process as much data in main memory as possible,
+<p>Although Flink aims to process as much data in main memory as possible,
it is not uncommon that more data needs to be processed than memory is
-available. Stratosphere's runtime is designed to write temporary data to disk
+available. Flink's runtime is designed to write temporary data to disk
to handle these situations.</p>
<p>The <code>taskmanager.tmp.dirs</code> parameter specifies a list of directories into which
-Stratosphere writes temporary files. The paths of the directories need to be
-separated by ':' (colon character). Stratosphere will concurrently write (or
+Flink writes temporary files. The paths of the directories need to be
+separated by ':' (colon character). Flink will concurrently write (or
read) one temporary file to (from) each configured directory. This way,
temporary I/O can be evenly distributed over multiple independent I/O devices
such as hard disks to improve performance. To leverage fast I/O devices (e.g.,
SSD, RAID, NAS), it is possible to specify a directory multiple times.</p>
<p>If the <code>taskmanager.tmp.dirs</code> parameter is not explicitly specified,
-Stratosphere writes temporary data to the temporary directory of the operating
+Flink writes temporary data to the temporary directory of the operating
system, such as <em>/tmp</em> in Linux systems.</p>
<p>Please see the <a href="config.html">configuration page</a> for details and additional
configuration options.</p>
-<h2 id="starting-stratosphere">Starting Stratosphere</h2>
+<h2 id="starting-flink">Starting Flink</h2>
<p>The following script starts a JobManager on the local node and connects via
SSH to all worker nodes listed in the <em>slaves</em> file to start the
-TaskManager on each node. Now your Stratosphere system is up and
+TaskManager on each node. Now your Flink system is up and
running. The JobManager running on the local node will now accept jobs
at the configured RPC port.</p>
-<p>Assuming that you are on the master node and inside the Stratosphere directory:</p>
+<p>Assuming that you are on the master node and inside the Flink directory:</p>
<div class="highlight"><pre><code class="language-text" data-lang="text">bin/start-cluster.sh
</code></pre></div>
Modified: incubator/flink/site/docs/0.6-SNAPSHOT/coding_guidelines.html
URL: http://svn.apache.org/viewvc/incubator/flink/site/docs/0.6-SNAPSHOT/coding_guidelines.html?rev=1610413&r1=1610412&r2=1610413&view=diff
==============================================================================
--- incubator/flink/site/docs/0.6-SNAPSHOT/coding_guidelines.html (original)
+++ incubator/flink/site/docs/0.6-SNAPSHOT/coding_guidelines.html Mon Jul 14 14:10:06 2014
@@ -109,6 +109,7 @@
<li>Setup & Configuration
<ul>
<li><a href="local_setup.html">Local Setup</a></li>
+ <li><a href="building.html">Build Flink</a></li>
<li><a href="cluster_setup.html">Cluster Setup</a></li>
<li><a href="yarn_setup.html">YARN Setup</a></li>
<li><a href="config.html">Configuration</a></li>
Modified: incubator/flink/site/docs/0.6-SNAPSHOT/config.html
URL: http://svn.apache.org/viewvc/incubator/flink/site/docs/0.6-SNAPSHOT/config.html?rev=1610413&r1=1610412&r2=1610413&view=diff
==============================================================================
--- incubator/flink/site/docs/0.6-SNAPSHOT/config.html (original)
+++ incubator/flink/site/docs/0.6-SNAPSHOT/config.html Mon Jul 14 14:10:06 2014
@@ -109,6 +109,7 @@
<li>Setup & Configuration
<ul>
<li><a href="local_setup.html">Local Setup</a></li>
+ <li><a href="building.html">Build Flink</a></li>
<li><a href="cluster_setup.html">Cluster Setup</a></li>
<li><a href="yarn_setup.html">YARN Setup</a></li>
<li><a href="config.html">Configuration</a></li>
Modified: incubator/flink/site/docs/0.6-SNAPSHOT/faq.html
URL: http://svn.apache.org/viewvc/incubator/flink/site/docs/0.6-SNAPSHOT/faq.html?rev=1610413&r1=1610412&r2=1610413&view=diff
==============================================================================
--- incubator/flink/site/docs/0.6-SNAPSHOT/faq.html (original)
+++ incubator/flink/site/docs/0.6-SNAPSHOT/faq.html Mon Jul 14 14:10:06 2014
@@ -109,6 +109,7 @@
<li>Setup & Configuration
<ul>
<li><a href="local_setup.html">Local Setup</a></li>
+ <li><a href="building.html">Build Flink</a></li>
<li><a href="cluster_setup.html">Cluster Setup</a></li>
<li><a href="yarn_setup.html">YARN Setup</a></li>
<li><a href="config.html">Configuration</a></li>
@@ -156,10 +157,10 @@
<a href="#general">General</a>
<ul>
<li>
-<a href="#is-stratosphere-a-hadoop-project?">Is Stratosphere a Hadoop Project?</a>
+<a href="#is-flink-a-hadoop-project?">Is Flink a Hadoop Project?</a>
</li>
<li>
-<a href="#do-i-have-to-install-apache-hadoop-to-use-stratosphere?">Do I have to install Apache Hadoop to use Stratosphere?</a>
+<a href="#do-i-have-to-install-apache-hadoop-to-use-flink?">Do I have to install Apache Hadoop to use Flink?</a>
</li>
</ul>
</li>
@@ -167,13 +168,13 @@
<a href="#usage">Usage</a>
<ul>
<li>
-<a href="#how-do-i-assess-the-progress-of-a-stratosphere-program?">How do I assess the progress of a Stratosphere program?</a>
+<a href="#how-do-i-assess-the-progress-of-a-flink-program?">How do I assess the progress of a Flink program?</a>
</li>
<li>
<a href="#how-can-i-figure-out-why-a-program-failed?">How can I figure out why a program failed?</a>
</li>
<li>
-<a href="#how-do-i-debug-stratosphere-programs?">How do I debug Stratosphere programs?</a>
+<a href="#how-do-i-debug-flink-programs?">How do I debug Flink programs?</a>
</li>
</ul>
</li>
@@ -193,7 +194,7 @@
<a href="#i-get-a-java.lang.instantiationexception-for-my-data-type,-what-is-wrong?">I get a java.lang.InstantiationException for my data type, what is wrong?</a>
</li>
<li>
-<a href="#i-can't-stop-stratosphere-with-the-provided-stop-scripts.-what-can-i-do?">I can't stop Stratosphere with the provided stop-scripts. What can I do?</a>
+<a href="#i-can't-stop-flink-with-the-provided-stop-scripts.-what-can-i-do?">I can't stop Flink with the provided stop-scripts. What can I do?</a>
</li>
<li>
<a href="#i-got-an-outofmemoryexception.-what-can-i-do?">I got an OutOfMemoryException. What can I do?</a>
@@ -218,7 +219,7 @@
<a href="#features">Features</a>
<ul>
<li>
-<a href="#what-kind-of-fault-tolerance-does-stratosphere-provide?">What kind of fault-tolerance does Stratosphere provide?</a>
+<a href="#what-kind-of-fault-tolerance-does-flink-provide?">What kind of fault-tolerance does Flink provide?</a>
</li>
<li>
<a href="#are-hadoop-like-utilities,-such-as-counters-and-the-distributedcache-supported?">Are Hadoop-like utilities, such as Counters and the DistributedCache supported?</a>
@@ -230,25 +231,25 @@
<h1 id="general">General</h1>
-<h2 id="is-stratosphere-a-hadoop-project?">Is Stratosphere a Hadoop Project?</h2>
+<h2 id="is-flink-a-hadoop-project?">Is Flink a Hadoop Project?</h2>
-<p>Stratosphere is a data processing system and an alternative to Hadoop's
+<p>Flink is a data processing system and an alternative to Hadoop's
MapReduce component. It comes with its own runtime, rather than building on top
of MapReduce. As such, it can work completely independently of the Hadoop
-ecosystem. However, Stratosphere can also access Hadoop's distributed file
+ecosystem. However, Flink can also access Hadoop's distributed file
system (HDFS) to read and write data, and Hadoop's next-generation resource
-manager (YARN) to provision cluster resources. Since most Stratosphere users are
+manager (YARN) to provision cluster resources. Since most Flink users are
using Hadoop HDFS to store their data, we ship already the required libraries to
access HDFS.</p>
-<h2 id="do-i-have-to-install-apache-hadoop-to-use-stratosphere?">Do I have to install Apache Hadoop to use Stratosphere?</h2>
+<h2 id="do-i-have-to-install-apache-hadoop-to-use-flink?">Do I have to install Apache Hadoop to use Flink?</h2>
-<p>No. Stratosphere can run without a Hadoop installation. However, a very common
-setup is to use Stratosphere to analyze data stored in the Hadoop Distributed
+<p>No. Flink can run without a Hadoop installation. However, a very common
+setup is to use Flink to analyze data stored in the Hadoop Distributed
File System (HDFS). To make these setups work out of the box, we bundle the
-Hadoop client libraries with Stratosphere by default.</p>
+Hadoop client libraries with Flink by default.</p>
-<p>Additionally, we provide a special YARN Enabled download of Stratosphere for
+<p>Additionally, we provide a special YARN Enabled download of Flink for
users with an existing Hadoop YARN cluster. <a href="http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/YARN.html">Apache Hadoop
YARN</a>
is Hadoop's cluster resource manager that allows to use
@@ -256,14 +257,14 @@ different execution engines next to each
<h1 id="usage">Usage</h1>
-<h2 id="how-do-i-assess-the-progress-of-a-stratosphere-program?">How do I assess the progress of a Stratosphere program?</h2>
+<h2 id="how-do-i-assess-the-progress-of-a-flink-program?">How do I assess the progress of a Flink program?</h2>
-<p>There are a multiple of ways to track the progress of a Stratosphere program:</p>
+<p>There are a multiple of ways to track the progress of a Flink program:</p>
<ul>
<li>The JobManager (the master of the distributed system) starts a web interface
to observe program execution. In runs on port 8081 by default (configured in
-<code>conf/stratosphere-config.yml</code>).</li>
+<code>conf/flink-config.yml</code>).</li>
<li>When you start a program from the command line, it will print the status
changes of all operators as the program progresses through the operations.</li>
<li>All status changes are also logged to the JobManager's log file.</li>
@@ -280,11 +281,11 @@ the standard error stream and shown on t
parallel task first failed and caused the other tasks to cancel the execution.</li>
<li>Failing tasks and the corresponding exceptions are reported in the log files
of the master and the worker where the exception occurred
-(<code>log/stratosphere-<user>-jobmanager-<host>.log</code> and
-<code>log/stratosphere-<user>-taskmanager-<host>.log</code>).</li>
+(<code>log/flink-<user>-jobmanager-<host>.log</code> and
+<code>log/flink-<user>-taskmanager-<host>.log</code>).</li>
</ul>
-<h2 id="how-do-i-debug-stratosphere-programs?">How do I debug Stratosphere programs?</h2>
+<h2 id="how-do-i-debug-flink-programs?">How do I debug Flink programs?</h2>
<ul>
<li>When you start a program locally with the <a href="local_execution.html">LocalExecutor</a>,
@@ -300,7 +301,7 @@ execution.</li>
<h2 id="i-get-an-error-message-saying-that-not-enough-buffers-are-available.-how-do-i-fix-this?">I get an error message saying that not enough buffers are available. How do I fix this?</h2>
-<p>If you run Stratosphere in a massively parallel setting (100+ parallel threads),
+<p>If you run Flink in a massively parallel setting (100+ parallel threads),
you need to adapt the number of network buffers via the config parameter
<code>taskmanager.network.numberOfBuffers</code>.
As a rule-of-thumb, the number of buffers should be at least
@@ -312,7 +313,7 @@ As a rule-of-thumb, the number of buffer
<p>Note: In version <em>0.4</em>, the delta iterations limit the solution set to
records with fixed-length data types. We will in the next version.</p>
-<p>The most common case for these exception is when Stratosphere is set up with the
+<p>The most common case for these exception is when Flink is set up with the
wrong HDFS version. Because different HDFS versions are often not compatible
with each other, the connection between the filesystem master and the client
breaks.</p>
@@ -326,11 +327,11 @@ breaks.</p>
at org.apache.hadoop.hdfs.DFSClient.<init><span class="o">(</span>DFSClient.java:207<span class="o">)</span>
at org.apache.hadoop.hdfs.DFSClient.<init><span class="o">(</span>DFSClient.java:170<span class="o">)</span>
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize<span class="o">(</span>DistributedFileSystem.java:82<span class="o">)</span>
- at eu.stratosphere.runtime.fs.hdfs.DistributedFileSystem.initialize<span class="o">(</span>DistributedFileSystem.java:276
+ at org.apache.flinkruntime.fs.hdfs.DistributedFileSystem.initialize<span class="o">(</span>DistributedFileSystem.java:276
</code></pre></div>
-<p>Please refer to the <a href="http://stratosphere.eu/downloads/#maven">download page</a> and
+<p>Please refer to the <a href="/downloads.html#maven">download page</a> and
the <a href=https://github.com/apache/incubator-flink/blob/master/README.md>build instructions</a>
-for details on how to set up Stratosphere for different Hadoop and HDFS versions.</p>
+for details on how to set up Flink for different Hadoop and HDFS versions.</p>
<h2 id="my-program-does-not-compute-the-correct-result.-why-are-my-custom-key-types">My program does not compute the correct result. Why are my custom key types</h2>
@@ -348,7 +349,7 @@ inadequate. Therefore, all keys must ove
or interfaces. If the classes are internal classes, they must be public and
static.</p>
-<h2 id="i-can't-stop-stratosphere-with-the-provided-stop-scripts.-what-can-i-do?">I can't stop Stratosphere with the provided stop-scripts. What can I do?</h2>
+<h2 id="i-can't-stop-flink-with-the-provided-stop-scripts.-what-can-i-do?">I can't stop Flink with the provided stop-scripts. What can I do?</h2>
<p>Stopping the processes sometimes takes a few seconds, because the shutdown may
do some cleanup work.</p>
@@ -382,7 +383,7 @@ provided.</p>
<ol>
<li><p>See whether you can use less memory inside the functions. For example, use
arrays of primitive types instead of object types.</p></li>
-<li><p>Reduce the memory that Stratosphere reserves for its own processing. The
+<li><p>Reduce the memory that Flink reserves for its own processing. The
TaskManager reserves a certain portion of the available memory for sorting,
hashing, caching, network buffering, etc. That part of the memory is unavailable
to the user-defined functions. By reserving it, the system can guarantee to not
@@ -409,13 +410,13 @@ inefficient and disk space consuming if
open. In some error cases however, the script immediately stops running. The
output looks like this:</p>
<div class="highlight"><pre><code class="language-text" data-lang="text">07:34:27,004 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1395604279745_273123 to ResourceManager at jobtracker-host
-Stratosphere JobManager is now running on worker1:6123
+Flink JobManager is now running on worker1:6123
JobManager Web Interface: http://jobtracker-host:54311/proxy/application_1295604279745_273123/
-07:34:51,528 INFO eu.stratosphere.yarn.Client - Application application_1295604279745_273123 finished with state FINISHED at 1398152089553
-07:34:51,529 INFO eu.stratosphere.yarn.Client - Killing the Stratosphere-YARN application.
+07:34:51,528 INFO org.apache.flinkyarn.Client - Application application_1295604279745_273123 finished with state FINISHED at 1398152089553
+07:34:51,529 INFO org.apache.flinkyarn.Client - Killing the Flink-YARN application.
07:34:51,529 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Killing application application_1295604279745_273123
-07:34:51,534 INFO eu.stratosphere.yarn.Client - Deleting files in hdfs://user/marcus/.stratosphere/application_1295604279745_273123
-07:34:51,559 INFO eu.stratosphere.yarn.Client - YARN Client is shutting down
+07:34:51,534 INFO org.apache.flinkyarn.Client - Deleting files in hdfs://user/marcus/.flink/application_1295604279745_273123
+07:34:51,559 INFO org.apache.flinkyarn.Client - YARN Client is shutting down
</code></pre></div>
<p>The problem here is that the Application Master (AM) is stopping and the YARN client assumes that the application has finished.</p>
@@ -483,30 +484,30 @@ this happened. You see messages from Lin
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2021)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1989)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1954)
- at eu.stratosphere.yarn.Utils.setupLocalResource(Utils.java:176)
- at eu.stratosphere.yarn.Client.run(Client.java:362)
- at eu.stratosphere.yarn.Client.main(Client.java:568)
+ at org.apache.flinkyarn.Utils.setupLocalResource(Utils.java:176)
+ at org.apache.flinkyarn.Client.run(Client.java:362)
+ at org.apache.flinkyarn.Client.main(Client.java:568)
</code></pre></div>
<p>The reason for this error is, that the home directory of the user <strong>in HDFS</strong>
has the wrong permissions. The user (in this case <code>robert</code>) can not create
directories in his own home directory.</p>
-<p>Stratosphere creates a <code>.stratosphere/</code> directory in the users home directory
-where it stores the Stratosphere jar and configuration file.</p>
+<p>Flink creates a <code>.flink/</code> directory in the users home directory
+where it stores the Flink jar and configuration file.</p>
<h1 id="features">Features</h1>
-<h2 id="what-kind-of-fault-tolerance-does-stratosphere-provide?">What kind of fault-tolerance does Stratosphere provide?</h2>
+<h2 id="what-kind-of-fault-tolerance-does-flink-provide?">What kind of fault-tolerance does Flink provide?</h2>
<p>Stratospere can restart failed jobs. Mid-query fault tolerance will go into the
open source project in the next versions.</p>
<h2 id="are-hadoop-like-utilities,-such-as-counters-and-the-distributedcache-supported?">Are Hadoop-like utilities, such as Counters and the DistributedCache supported?</h2>
-<p><a href="java_api_guide.html#accumulators-&-counters">Stratosphere's Accumulators</a> work very similar like
+<p><a href="java_api_guide.html#accumulators-&-counters">Flink's Accumulators</a> work very similar like
[Hadoop's counters, but are more powerful.</p>
-<p>Stratosphere has a <a href=https://github.com/apache/incubator-flink/blob/master//stratosphere-core/src/main/java/eu/stratosphere/api/common/cache/DistributedCache.java>Distributed Cache</a> that is deeply integrated with the APIs. Please refer to the <a href=https://github.com/apache/incubator-flink/blob/master//stratosphere-java/src/main/java/eu/stratosphere/api/java/ExecutionEnvironment.java#L561>JavaDocs</a> for details on how to use it.</p>
+<p>Flink has a <a href=https://github.com/apache/incubator-flink/blob/master//flink-core/src/main/java/org/apache/flink/api/common/cache/DistributedCache.java>Distributed Cache</a> that is deeply integrated with the APIs. Please refer to the <a href=https://github.com/apache/incubator-flink/blob/master//flink-java/src/main/java/org/apache/flink/api/java/ExecutionEnvironment.java#L561>JavaDocs</a> for details on how to use it.</p>
<p>In order to make data sets available on all tasks, we encourage you to use <a href="java_api_guide.html#broadcast_variables">Broadcast Variables</a> instead. They are more efficient and easier to use than the distributed cache.</p>
Modified: incubator/flink/site/docs/0.6-SNAPSHOT/hadoop_compatability.html
URL: http://svn.apache.org/viewvc/incubator/flink/site/docs/0.6-SNAPSHOT/hadoop_compatability.html?rev=1610413&r1=1610412&r2=1610413&view=diff
==============================================================================
--- incubator/flink/site/docs/0.6-SNAPSHOT/hadoop_compatability.html (original)
+++ incubator/flink/site/docs/0.6-SNAPSHOT/hadoop_compatability.html Mon Jul 14 14:10:06 2014
@@ -109,6 +109,7 @@
<li>Setup & Configuration
<ul>
<li><a href="local_setup.html">Local Setup</a></li>
+ <li><a href="building.html">Build Flink</a></li>
<li><a href="cluster_setup.html">Cluster Setup</a></li>
<li><a href="yarn_setup.html">YARN Setup</a></li>
<li><a href="config.html">Configuration</a></li>
Modified: incubator/flink/site/docs/0.6-SNAPSHOT/how_to_contribute.html
URL: http://svn.apache.org/viewvc/incubator/flink/site/docs/0.6-SNAPSHOT/how_to_contribute.html?rev=1610413&r1=1610412&r2=1610413&view=diff
==============================================================================
--- incubator/flink/site/docs/0.6-SNAPSHOT/how_to_contribute.html (original)
+++ incubator/flink/site/docs/0.6-SNAPSHOT/how_to_contribute.html Mon Jul 14 14:10:06 2014
@@ -109,6 +109,7 @@
<li>Setup & Configuration
<ul>
<li><a href="local_setup.html">Local Setup</a></li>
+ <li><a href="building.html">Build Flink</a></li>
<li><a href="cluster_setup.html">Cluster Setup</a></li>
<li><a href="yarn_setup.html">YARN Setup</a></li>
<li><a href="config.html">Configuration</a></li>
@@ -198,7 +199,7 @@
<li><p>It is typically helpful to switch to a <em>topic branch</em> for the changes. To create a dedicated branch based on the current master, use the following command:</p>
<div class="highlight"><pre><code class="language-text" data-lang="text">git checkout -b myBranch master
</code></pre></div></li>
-<li><p>Now you can create your changes, compile the code, and validate the changes. Here are some pointers on how to <a href="https://github.com/apache/incubator-flink/#eclipse-setup-and-debugging">set up the Eclipse IDE for development</a>, and how to <a href="https://github.com/apache/incubator-flink/#build-stratosphere">build the code</a>.</p></li>
+<li><p>Now you can create your changes, compile the code, and validate the changes. Here are some pointers on how to <a href="https://github.com/apache/incubator-flink/#eclipse-setup-and-debugging">set up the Eclipse IDE for development</a>, and how to <a href="https://github.com/apache/incubator-flink/#build-flink">build the code</a>.</p></li>
<li><p>After you have finalized your contribution, verify the compliance with the contribution guidelines (see below), and commit them. To make the changes easily mergeable, please rebase them to the latest version of the main repositories master branch. Assuming you created a topic branch (step 3), you can follow this sequence of commands to do that:
Switch to the master branch, update it to the latest revision, switch back to your topic branch, and rebase it on top of the master branch.</p>
<div class="highlight"><pre><code class="language-text" data-lang="text">git checkout master
Added: incubator/flink/site/docs/0.6-SNAPSHOT/img/FlinkOnYarn.svg
URL: http://svn.apache.org/viewvc/incubator/flink/site/docs/0.6-SNAPSHOT/img/FlinkOnYarn.svg?rev=1610413&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/flink/site/docs/0.6-SNAPSHOT/img/FlinkOnYarn.svg
------------------------------------------------------------------------------
svn:mime-type = image/svg+xml
Modified: incubator/flink/site/docs/0.6-SNAPSHOT/index.html
URL: http://svn.apache.org/viewvc/incubator/flink/site/docs/0.6-SNAPSHOT/index.html?rev=1610413&r1=1610412&r2=1610413&view=diff
==============================================================================
--- incubator/flink/site/docs/0.6-SNAPSHOT/index.html (original)
+++ incubator/flink/site/docs/0.6-SNAPSHOT/index.html Mon Jul 14 14:10:06 2014
@@ -109,6 +109,7 @@
<li>Setup & Configuration
<ul>
<li><a href="local_setup.html">Local Setup</a></li>
+ <li><a href="building.html">Build Flink</a></li>
<li><a href="cluster_setup.html">Cluster Setup</a></li>
<li><a href="yarn_setup.html">YARN Setup</a></li>
<li><a href="config.html">Configuration</a></li>