You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by se...@apache.org on 2016/08/25 18:48:50 UTC
[47/89] [abbrv] [partial] flink git commit: [FLINK-4317,
FLIP-3] [docs] Restructure docs
http://git-wip-us.apache.org/repos/asf/flink/blob/844c874b/docs/apis/batch/iterations.md
----------------------------------------------------------------------
diff --git a/docs/apis/batch/iterations.md b/docs/apis/batch/iterations.md
deleted file mode 100644
index a1bd0e9..0000000
--- a/docs/apis/batch/iterations.md
+++ /dev/null
@@ -1,213 +0,0 @@
----
-title: "Iterations"
-
-# Sub-level navigation
-sub-nav-group: batch
-sub-nav-pos: 3
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-Iterative algorithms occur in many domains of data analysis, such as *machine learning* or *graph analysis*. Such algorithms are crucial in order to realize the promise of Big Data to extract meaningful information out of your data. With increasing interest to run these kinds of algorithms on very large data sets, there is a need to execute iterations in a massively parallel fashion.
-
-Flink programs implement iterative algorithms by defining a **step function** and embedding it into a special iteration operator. There are two variants of this operator: **Iterate** and **Delta Iterate**. Both operators repeatedly invoke the step function on the current iteration state until a certain termination condition is reached.
-
-Here, we provide background on both operator variants and outline their usage. The [programming guide](index.html) explains how to implement the operators in both Scala and Java. We also support **vertex-centric, scatter-gather, and gather-sum-apply iterations** through Flink's graph processing API, [Gelly]({{site.baseurl}}/libs/gelly_guide.html).
-
-The following table provides an overview of both operators:
-
-
-<table class="table table-striped table-hover table-bordered">
- <thead>
- <th></th>
- <th class="text-center">Iterate</th>
- <th class="text-center">Delta Iterate</th>
- </thead>
- <tr>
- <td class="text-center" width="20%"><strong>Iteration Input</strong></td>
- <td class="text-center" width="40%"><strong>Partial Solution</strong></td>
- <td class="text-center" width="40%"><strong>Workset</strong> and <strong>Solution Set</strong></td>
- </tr>
- <tr>
- <td class="text-center"><strong>Step Function</strong></td>
- <td colspan="2" class="text-center">Arbitrary Data Flows</td>
- </tr>
- <tr>
- <td class="text-center"><strong>State Update</strong></td>
- <td class="text-center">Next <strong>partial solution</strong></td>
- <td>
- <ul>
- <li>Next workset</li>
- <li><strong>Changes to solution set</strong></li>
- </ul>
- </td>
- </tr>
- <tr>
- <td class="text-center"><strong>Iteration Result</strong></td>
- <td class="text-center">Last partial solution</td>
- <td class="text-center">Solution set state after last iteration</td>
- </tr>
- <tr>
- <td class="text-center"><strong>Termination</strong></td>
- <td>
- <ul>
- <li><strong>Maximum number of iterations</strong> (default)</li>
- <li>Custom aggregator convergence</li>
- </ul>
- </td>
- <td>
- <ul>
- <li><strong>Maximum number of iterations or empty workset</strong> (default)</li>
- <li>Custom aggregator convergence</li>
- </ul>
- </td>
- </tr>
-</table>
-
-
-* This will be replaced by the TOC
-{:toc}
-
-Iterate Operator
-----------------
-
-The **iterate operator** covers the *simple form of iterations*: in each iteration, the **step function** consumes the **entire input** (the *result of the previous iteration*, or the *initial data set*), and computes the **next version of the partial solution** (e.g. `map`, `reduce`, `join`, etc.).
-
-<p class="text-center">
- <img alt="Iterate Operator" width="60%" src="fig/iterations_iterate_operator.png" />
-</p>
-
- 1. **Iteration Input**: Initial input for the *first iteration* from a *data source* or *previous operators*.
- 2. **Step Function**: The step function will be executed in each iteration. It is an arbitrary data flow consisting of operators like `map`, `reduce`, `join`, etc. and depends on your specific task at hand.
- 3. **Next Partial Solution**: In each iteration, the output of the step function will be fed back into the *next iteration*.
- 4. **Iteration Result**: Output of the *last iteration* is written to a *data sink* or used as input to the *following operators*.
-
-There are multiple options to specify **termination conditions** for an iteration:
-
- - **Maximum number of iterations**: Without any further conditions, the iteration will be executed this many times.
- - **Custom aggregator convergence**: Iterations allow to specify *custom aggregators* and *convergence criteria* like sum aggregate the number of emitted records (aggregator) and terminate if this number is zero (convergence criterion).
-
-You can also think about the iterate operator in pseudo-code:
-
-~~~java
-IterationState state = getInitialState();
-
-while (!terminationCriterion()) {
- state = step(state);
-}
-
-setFinalState(state);
-~~~
-
-<div class="panel panel-default">
- <div class="panel-body">
- See the <strong><a href="index.html">Programming Guide</a> </strong> for details and code examples.</div>
-</div>
-
-### Example: Incrementing Numbers
-
-In the following example, we **iteratively incremenet a set numbers**:
-
-<p class="text-center">
- <img alt="Iterate Operator Example" width="60%" src="fig/iterations_iterate_operator_example.png" />
-</p>
-
- 1. **Iteration Input**: The inital input is read from a data source and consists of five single-field records (integers `1` to `5`).
- 2. **Step function**: The step function is a single `map` operator, which increments the integer field from `i` to `i+1`. It will be applied to every record of the input.
- 3. **Next Partial Solution**: The output of the step function will be the output of the map operator, i.e. records with incremented integers.
- 4. **Iteration Result**: After ten iterations, the initial numbers will have been incremented ten times, resulting in integers `11` to `15`.
-
-~~~
-// 1st 2nd 10th
-map(1) -> 2 map(2) -> 3 ... map(10) -> 11
-map(2) -> 3 map(3) -> 4 ... map(11) -> 12
-map(3) -> 4 map(4) -> 5 ... map(12) -> 13
-map(4) -> 5 map(5) -> 6 ... map(13) -> 14
-map(5) -> 6 map(6) -> 7 ... map(14) -> 15
-~~~
-
-Note that **1**, **2**, and **4** can be arbitrary data flows.
-
-
-Delta Iterate Operator
-----------------------
-
-The **delta iterate operator** covers the case of **incremental iterations**. Incremental iterations **selectively modify elements** of their **solution** and evolve the solution rather than fully recompute it.
-
-Where applicable, this leads to **more efficient algorithms**, because not every element in the solution set changes in each iteration. This allows to **focus on the hot parts** of the solution and leave the **cold parts untouched**. Frequently, the majority of the solution cools down comparatively fast and the later iterations operate only on a small subset of the data.
-
-<p class="text-center">
- <img alt="Delta Iterate Operator" width="60%" src="fig/iterations_delta_iterate_operator.png" />
-</p>
-
- 1. **Iteration Input**: The initial workset and solution set are read from *data sources* or *previous operators* as input to the first iteration.
- 2. **Step Function**: The step function will be executed in each iteration. It is an arbitrary data flow consisting of operators like `map`, `reduce`, `join`, etc. and depends on your specific task at hand.
- 3. **Next Workset/Update Solution Set**: The *next workset* drives the iterative computation and will be fed back into the *next iteration*. Furthermore, the solution set will be updated and implicitly forwarded (it is not required to be rebuild). Both data sets can be updated by different operators of the step function.
- 4. **Iteration Result**: After the *last iteration*, the *solution set* is written to a *data sink* or used as input to the *following operators*.
-
-The default **termination condition** for delta iterations is specified by the **empty workset convergence criterion** and a **maximum number of iterations**. The iteration will terminate when a produced *next workset* is empty or when the maximum number of iterations is reached. It is also possible to specify a **custom aggregator** and **convergence criterion**.
-
-You can also think about the iterate operator in pseudo-code:
-
-~~~java
-IterationState workset = getInitialState();
-IterationState solution = getInitialSolution();
-
-while (!terminationCriterion()) {
- (delta, workset) = step(workset, solution);
-
- solution.update(delta)
-}
-
-setFinalState(solution);
-~~~
-
-<div class="panel panel-default">
- <div class="panel-body">
- See the <strong><a href="index.html">programming guide</a></strong> for details and code examples.</div>
-</div>
-
-### Example: Propagate Minimum in Graph
-
-In the following example, every vertex has an **ID** and a **coloring**. Each vertex will propagate its vertex ID to neighboring vertices. The **goal** is to *assign the minimum ID to every vertex in a subgraph*. If a received ID is smaller then the current one, it changes to the color of the vertex with the received ID. One application of this can be found in *community analysis* or *connected components* computation.
-
-<p class="text-center">
- <img alt="Delta Iterate Operator Example" width="100%" src="fig/iterations_delta_iterate_operator_example.png" />
-</p>
-
-The **initial input** is set as **both workset and solution set.** In the above figure, the colors visualize the **evolution of the solution set**. With each iteration, the color of the minimum ID is spreading in the respective subgraph. At the same time, the amount of work (exchanged and compared vertex IDs) decreases with each iteration. This corresponds to the **decreasing size of the workset**, which goes from all seven vertices to zero after three iterations, at which time the iteration terminates. The **important observation** is that *the lower subgraph converges before the upper half* does and the delta iteration is able to capture this with the workset abstraction.
-
-In the upper subgraph **ID 1** (*orange*) is the **minimum ID**. In the **first iteration**, it will get propagated to vertex 2, which will subsequently change its color to orange. Vertices 3 and 4 will receive **ID 2** (in *yellow*) as their current minimum ID and change to yellow. Because the color of *vertex 1* didn't change in the first iteration, it can be skipped it in the next workset.
-
-In the lower subgraph **ID 5** (*cyan*) is the **minimum ID**. All vertices of the lower subgraph will receive it in the first iteration. Again, we can skip the unchanged vertices (*vertex 5*) for the next workset.
-
-In the **2nd iteration**, the workset size has already decreased from seven to five elements (vertices 2, 3, 4, 6, and 7). These are part of the iteration and further propagate their current minimum IDs. After this iteration, the lower subgraph has already converged (**cold part** of the graph), as it has no elements in the workset, whereas the upper half needs a further iteration (**hot part** of the graph) for the two remaining workset elements (vertices 3 and 4).
-
-The iteration **terminates**, when the workset is empty after the **3rd iteration**.
-
-<a href="#supersteps"></a>
-
-Superstep Synchronization
--------------------------
-
-We referred to each execution of the step function of an iteration operator as *a single iteration*. In parallel setups, **multiple instances of the step function are evaluated in parallel** on different partitions of the iteration state. In many settings, one evaluation of the step function on all parallel instances forms a so called **superstep**, which is also the granularity of synchronization. Therefore, *all* parallel tasks of an iteration need to complete the superstep, before a next superstep will be initialized. **Termination criteria** will also be evaluated at superstep barriers.
-
-<p class="text-center">
- <img alt="Supersteps" width="50%" src="fig/iterations_supersteps.png" />
-</p>
http://git-wip-us.apache.org/repos/asf/flink/blob/844c874b/docs/apis/batch/libs/fig/LICENSE.txt
----------------------------------------------------------------------
diff --git a/docs/apis/batch/libs/fig/LICENSE.txt b/docs/apis/batch/libs/fig/LICENSE.txt
deleted file mode 100644
index 5d0d22b..0000000
--- a/docs/apis/batch/libs/fig/LICENSE.txt
+++ /dev/null
@@ -1,17 +0,0 @@
-All image files in the folder and its subfolders are
-licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
http://git-wip-us.apache.org/repos/asf/flink/blob/844c874b/docs/apis/batch/libs/fig/gelly-example-graph.png
----------------------------------------------------------------------
diff --git a/docs/apis/batch/libs/fig/gelly-example-graph.png b/docs/apis/batch/libs/fig/gelly-example-graph.png
deleted file mode 100644
index abef960..0000000
Binary files a/docs/apis/batch/libs/fig/gelly-example-graph.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/flink/blob/844c874b/docs/apis/batch/libs/fig/gelly-filter.png
----------------------------------------------------------------------
diff --git a/docs/apis/batch/libs/fig/gelly-filter.png b/docs/apis/batch/libs/fig/gelly-filter.png
deleted file mode 100644
index cb09744..0000000
Binary files a/docs/apis/batch/libs/fig/gelly-filter.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/flink/blob/844c874b/docs/apis/batch/libs/fig/gelly-gsa-sssp1.png
----------------------------------------------------------------------
diff --git a/docs/apis/batch/libs/fig/gelly-gsa-sssp1.png b/docs/apis/batch/libs/fig/gelly-gsa-sssp1.png
deleted file mode 100644
index 1eeb1e6..0000000
Binary files a/docs/apis/batch/libs/fig/gelly-gsa-sssp1.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/flink/blob/844c874b/docs/apis/batch/libs/fig/gelly-reduceOnEdges.png
----------------------------------------------------------------------
diff --git a/docs/apis/batch/libs/fig/gelly-reduceOnEdges.png b/docs/apis/batch/libs/fig/gelly-reduceOnEdges.png
deleted file mode 100644
index ffb674d..0000000
Binary files a/docs/apis/batch/libs/fig/gelly-reduceOnEdges.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/flink/blob/844c874b/docs/apis/batch/libs/fig/gelly-reduceOnNeighbors.png
----------------------------------------------------------------------
diff --git a/docs/apis/batch/libs/fig/gelly-reduceOnNeighbors.png b/docs/apis/batch/libs/fig/gelly-reduceOnNeighbors.png
deleted file mode 100644
index 63137b8..0000000
Binary files a/docs/apis/batch/libs/fig/gelly-reduceOnNeighbors.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/flink/blob/844c874b/docs/apis/batch/libs/fig/gelly-union.png
----------------------------------------------------------------------
diff --git a/docs/apis/batch/libs/fig/gelly-union.png b/docs/apis/batch/libs/fig/gelly-union.png
deleted file mode 100644
index b00f831..0000000
Binary files a/docs/apis/batch/libs/fig/gelly-union.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/flink/blob/844c874b/docs/apis/batch/libs/fig/gelly-vc-sssp1.png
----------------------------------------------------------------------
diff --git a/docs/apis/batch/libs/fig/gelly-vc-sssp1.png b/docs/apis/batch/libs/fig/gelly-vc-sssp1.png
deleted file mode 100644
index 9497d98..0000000
Binary files a/docs/apis/batch/libs/fig/gelly-vc-sssp1.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/flink/blob/844c874b/docs/apis/batch/libs/fig/vertex-centric supersteps.png
----------------------------------------------------------------------
diff --git a/docs/apis/batch/libs/fig/vertex-centric supersteps.png b/docs/apis/batch/libs/fig/vertex-centric supersteps.png
deleted file mode 100644
index 6498a25..0000000
Binary files a/docs/apis/batch/libs/fig/vertex-centric supersteps.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/flink/blob/844c874b/docs/apis/batch/libs/gelly.md
----------------------------------------------------------------------
diff --git a/docs/apis/batch/libs/gelly.md b/docs/apis/batch/libs/gelly.md
deleted file mode 100644
index 773642d..0000000
--- a/docs/apis/batch/libs/gelly.md
+++ /dev/null
@@ -1,26 +0,0 @@
----
-title: Gelly
----
-
-<meta http-equiv="refresh" content="1; url={{ site.baseurl }}/apis/batch/libs/gelly/index.html" />
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-The *Gelly guide* has been moved. Redirecting to [{{ site.baseurl }}/apis/batch/libs/gelly/index.html]({{ site.baseurl }}/apis/batch/libs/gelly/index.html) in 1 second.
http://git-wip-us.apache.org/repos/asf/flink/blob/844c874b/docs/apis/batch/libs/gelly/graph_algorithms.md
----------------------------------------------------------------------
diff --git a/docs/apis/batch/libs/gelly/graph_algorithms.md b/docs/apis/batch/libs/gelly/graph_algorithms.md
deleted file mode 100644
index b443a29..0000000
--- a/docs/apis/batch/libs/gelly/graph_algorithms.md
+++ /dev/null
@@ -1,311 +0,0 @@
----
-title: Graph Algorithms
-
-# Sub navigation
-sub-nav-group: batch
-sub-nav-parent: gelly
-sub-nav-title: Graph Algorithms
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-The logic blocks with which the `Graph` API and top-level algorithms are assembled are accessible in Gelly as graph
-algorithms in the `org.apache.flink.graph.asm` package. These algorithms provide optimization and tuning through
-configuration parameters and may provide implicit runtime reuse when processing the same input with a similar
-configuration.
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 20%">Algorithm</th>
- <th class="text-center">Description</th>
- </tr>
- </thead>
-
- <tbody>
- <tr>
- <td>degree.annotate.directed.<br/><strong>VertexInDegree</strong></td>
- <td>
- <p>Annotate vertices of a <a href="#graph-representation">directed graph</a> with the in-degree.</p>
-{% highlight java %}
-DataSet<Vertex<K, LongValue>> inDegree = graph
- .run(new VertexInDegree()
- .setIncludeZeroDegreeVertices(true));
-{% endhighlight %}
- <p>Optional configuration:</p>
- <ul>
- <li><p><strong>setIncludeZeroDegreeVertices</strong>: by default only the edge set is processed for the computation of degree; when this flag is set an additional join is performed against the vertex set in order to output vertices with an in-degree of zero</p></li>
- <li><p><strong>setParallelism</strong>: override the operator parallelism</p></li>
- </ul>
- </td>
- </tr>
-
- <tr>
- <td>degree.annotate.directed.<br/><strong>VertexOutDegree</strong></td>
- <td>
- <p>Annotate vertices of a <a href="#graph-representation">directed graph</a> with the out-degree.</p>
-{% highlight java %}
-DataSet<Vertex<K, LongValue>> outDegree = graph
- .run(new VertexOutDegree()
- .setIncludeZeroDegreeVertices(true));
-{% endhighlight %}
- <p>Optional configuration:</p>
- <ul>
- <li><p><strong>setIncludeZeroDegreeVertices</strong>: by default only the edge set is processed for the computation of degree; when this flag is set an additional join is performed against the vertex set in order to output vertices with an out-degree of zero</p></li>
- <li><p><strong>setParallelism</strong>: override the operator parallelism</p></li>
- </ul>
- </td>
- </tr>
-
- <tr>
- <td>degree.annotate.directed.<br/><strong>VertexDegrees</strong></td>
- <td>
- <p>Annotate vertices of a <a href="#graph-representation">directed graph</a> with the degree, out-degree, and in-degree.</p>
-{% highlight java %}
-DataSet<Vertex<K, Tuple2<LongValue, LongValue>>> degrees = graph
- .run(new VertexDegrees()
- .setIncludeZeroDegreeVertices(true));
-{% endhighlight %}
- <p>Optional configuration:</p>
- <ul>
- <li><p><strong>setIncludeZeroDegreeVertices</strong>: by default only the edge set is processed for the computation of degree; when this flag is set an additional join is performed against the vertex set in order to output vertices with out- and in-degree of zero</p></li>
- <li><p><strong>setParallelism</strong>: override the operator parallelism</p></li>
- </ul>
- </td>
- </tr>
-
- <tr>
- <td>degree.annotate.directed.<br/><strong>EdgeSourceDegrees</strong></td>
- <td>
- <p>Annotate edges of a <a href="#graph-representation">directed graph</a> with the degree, out-degree, and in-degree of the source ID.</p>
-{% highlight java %}
-DataSet<Edge<K, Tuple2<EV, Degrees>>> sourceDegrees = graph
- .run(new EdgeSourceDegrees());
-{% endhighlight %}
- <p>Optional configuration:</p>
- <ul>
- <li><p><strong>setParallelism</strong>: override the operator parallelism</p></li>
- </ul>
- </td>
- </tr>
-
- <tr>
- <td>degree.annotate.directed.<br/><strong>EdgeTargetDegrees</strong></td>
- <td>
- <p>Annotate edges of a <a href="#graph-representation">directed graph</a> with the degree, out-degree, and in-degree of the target ID.</p>
-{% highlight java %}
-DataSet<Edge<K, Tuple2<EV, Degrees>>> targetDegrees = graph
- .run(new EdgeTargetDegrees();
-{% endhighlight %}
- <p>Optional configuration:</p>
- <ul>
- <li><p><strong>setParallelism</strong>: override the operator parallelism</p></li>
- </ul>
- </td>
- </tr>
-
- <tr>
- <td>degree.annotate.directed.<br/><strong>EdgeDegreesPair</strong></td>
- <td>
- <p>Annotate edges of a <a href="#graph-representation">directed graph</a> with the degree, out-degree, and in-degree of both the source and target vertices.</p>
-{% highlight java %}
-DataSet<Edge<K, Tuple2<EV, Degrees>>> degrees = graph
- .run(new EdgeDegreesPair());
-{% endhighlight %}
- <p>Optional configuration:</p>
- <ul>
- <li><p><strong>setParallelism</strong>: override the operator parallelism</p></li>
- </ul>
- </td>
- </tr>
-
- <tr>
- <td>degree.annotate.undirected.<br/><strong>VertexDegree</strong></td>
- <td>
- <p>Annotate vertices of an <a href="#graph-representation">undirected graph</a> with the degree.</p>
-{% highlight java %}
-DataSet<Vertex<K, LongValue>> degree = graph
- .run(new VertexDegree()
- .setIncludeZeroDegreeVertices(true)
- .setReduceOnTargetId(true));
-{% endhighlight %}
- <p>Optional configuration:</p>
- <ul>
- <li><p><strong>setIncludeZeroDegreeVertices</strong>: by default only the edge set is processed for the computation of degree; when this flag is set an additional join is performed against the vertex set in order to output vertices with a degree of zero</p></li>
- <li><p><strong>setParallelism</strong>: override the operator parallelism</p></li>
- <li><p><strong>setReduceOnTargetId</strong>: the degree can be counted from either the edge source or target IDs. By default the source IDs are counted. Reducing on target IDs may optimize the algorithm if the input edge list is sorted by target ID.</p></li>
- </ul>
- </td>
- </tr>
-
- <tr>
- <td>degree.annotate.undirected.<br/><strong>EdgeSourceDegree</strong></td>
- <td>
- <p>Annotate edges of an <a href="#graph-representation">undirected graph</a> with degree of the source ID.</p>
-{% highlight java %}
-DataSet<Edge<K, Tuple2<EV, LongValue>>> sourceDegree = graph
- .run(new EdgeSourceDegree()
- .setReduceOnTargetId(true));
-{% endhighlight %}
- <p>Optional configuration:</p>
- <ul>
- <li><p><strong>setParallelism</strong>: override the operator parallelism</p></li>
- <li><p><strong>setReduceOnTargetId</strong>: the degree can be counted from either the edge source or target IDs. By default the source IDs are counted. Reducing on target IDs may optimize the algorithm if the input edge list is sorted by target ID.</p></li>
- </ul>
- </td>
- </tr>
-
- <tr>
- <td>degree.annotate.undirected.<br/><strong>EdgeTargetDegree</strong></td>
- <td>
- <p>Annotate edges of an <a href="#graph-representation">undirected graph</a> with degree of the target ID.</p>
-{% highlight java %}
-DataSet<Edge<K, Tuple2<EV, LongValue>>> targetDegree = graph
- .run(new EdgeTargetDegree()
- .setReduceOnSourceId(true));
-{% endhighlight %}
- <p>Optional configuration:</p>
- <ul>
- <li><p><strong>setParallelism</strong>: override the operator parallelism</p></li>
- <li><p><strong>setReduceOnSourceId</strong>: the degree can be counted from either the edge source or target IDs. By default the target IDs are counted. Reducing on source IDs may optimize the algorithm if the input edge list is sorted by source ID.</p></li>
- </ul>
- </td>
- </tr>
-
- <tr>
- <td>degree.annotate.undirected.<br/><strong>EdgeDegreePair</strong></td>
- <td>
- <p>Annotate edges of an <a href="#graph-representation">undirected graph</a> with the degree of both the source and target vertices.</p>
-{% highlight java %}
-DataSet<Edge<K, Tuple3<EV, LongValue, LongValue>>> pairDegree = graph
- .run(new EdgeDegreePair()
- .setReduceOnTargetId(true));
-{% endhighlight %}
- <p>Optional configuration:</p>
- <ul>
- <li><p><strong>setParallelism</strong>: override the operator parallelism</p></li>
- <li><p><strong>setReduceOnTargetId</strong>: the degree can be counted from either the edge source or target IDs. By default the source IDs are counted. Reducing on target IDs may optimize the algorithm if the input edge list is sorted by target ID.</p></li>
- </ul>
- </td>
- </tr>
-
- <tr>
- <td>degree.filter.undirected.<br/><strong>MaximumDegree</strong></td>
- <td>
- <p>Filter an <a href="#graph-representation">undirected graph</a> by maximum degree.</p>
-{% highlight java %}
-Graph<K, VV, EV> filteredGraph = graph
- .run(new MaximumDegree(5000)
- .setBroadcastHighDegreeVertices(true)
- .setReduceOnTargetId(true));
-{% endhighlight %}
- <p>Optional configuration:</p>
- <ul>
- <li><p><strong>setBroadcastHighDegreeVertices</strong>: join high-degree vertices using a broadcast-hash to reduce data shuffling when removing a relatively small number of high-degree vertices.</p></li>
- <li><p><strong>setParallelism</strong>: override the operator parallelism</p></li>
- <li><p><strong>setReduceOnTargetId</strong>: the degree can be counted from either the edge source or target IDs. By default the source IDs are counted. Reducing on target IDs may optimize the algorithm if the input edge list is sorted by target ID.</p></li>
- </ul>
- </td>
- </tr>
-
- <tr>
- <td>simple.directed.<br/><strong>Simplify</strong></td>
- <td>
- <p>Remove self-loops and duplicate edges from a <a href="#graph-representation">directed graph</a>.</p>
-{% highlight java %}
-graph.run(new Simplify());
-{% endhighlight %}
- <p>Optional configuration:</p>
- <ul>
- <li><p><strong>setParallelism</strong>: override the operator parallelism</p></li>
- </ul>
- </td>
- </tr>
-
- <tr>
- <td>simple.undirected.<br/><strong>Simplify</strong></td>
- <td>
- <p>Add symmetric edges and remove self-loops and duplicate edges from an <a href="#graph-representation">undirected graph</a>.</p>
-{% highlight java %}
-graph.run(new Simplify());
-{% endhighlight %}
- <p>Optional configuration:</p>
- <ul>
- <li><p><strong>setParallelism</strong>: override the operator parallelism</p></li>
- </ul>
- </td>
- </tr>
-
- <tr>
- <td>translate.<br/><strong>TranslateGraphIds</strong></td>
- <td>
- <p>Translate vertex and edge IDs using the given <code>TranslateFunction</code>.</p>
-{% highlight java %}
-graph.run(new TranslateGraphIds(new LongValueToStringValue()));
-{% endhighlight %}
- <p>Required configuration:</p>
- <ul>
- <li><p><strong>translator</strong>: implements type or value conversion</p></li>
- </ul>
- <p>Optional configuration:</p>
- <ul>
- <li><p><strong>setParallelism</strong>: override the operator parallelism</p></li>
- </ul>
- </td>
- </tr>
-
- <tr>
- <td>translate.<br/><strong>TranslateVertexValues</strong></td>
- <td>
- <p>Translate vertex values using the given <code>TranslateFunction</code>.</p>
-{% highlight java %}
-graph.run(new TranslateVertexValues(new LongValueAddOffset(vertexCount)));
-{% endhighlight %}
- <p>Required configuration:</p>
- <ul>
- <li><p><strong>translator</strong>: implements type or value conversion</p></li>
- </ul>
- <p>Optional configuration:</p>
- <ul>
- <li><p><strong>setParallelism</strong>: override the operator parallelism</p></li>
- </ul>
- </td>
- </tr>
-
- <tr>
- <td>translate.<br/><strong>TranslateEdgeValues</strong></td>
- <td>
- <p>Translate edge values using the given <code>TranslateFunction</code>.</p>
-{% highlight java %}
-graph.run(new TranslateEdgeValues(new Nullify()));
-{% endhighlight %}
- <p>Required configuration:</p>
- <ul>
- <li><p><strong>translator</strong>: implements type or value conversion</p></li>
- </ul>
- <p>Optional configuration:</p>
- <ul>
- <li><p><strong>setParallelism</strong>: override the operator parallelism</p></li>
- </ul>
- </td>
- </tr>
- </tbody>
-</table>
-
-{% top %}
http://git-wip-us.apache.org/repos/asf/flink/blob/844c874b/docs/apis/batch/libs/gelly/graph_api.md
----------------------------------------------------------------------
diff --git a/docs/apis/batch/libs/gelly/graph_api.md b/docs/apis/batch/libs/gelly/graph_api.md
deleted file mode 100644
index 6f30911..0000000
--- a/docs/apis/batch/libs/gelly/graph_api.md
+++ /dev/null
@@ -1,836 +0,0 @@
----
-title: Graph API
-
-# Sub navigation
-sub-nav-group: batch
-sub-nav-parent: gelly
-sub-nav-title: Graph API
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-* This will be replaced by the TOC
-{:toc}
-
-Graph Representation
------------
-
-In Gelly, a `Graph` is represented by a `DataSet` of vertices and a `DataSet` of edges.
-
-The `Graph` nodes are represented by the `Vertex` type. A `Vertex` is defined by a unique ID and a value. `Vertex` IDs should implement the `Comparable` interface. Vertices without value can be represented by setting the value type to `NullValue`.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-// create a new vertex with a Long ID and a String value
-Vertex<Long, String> v = new Vertex<Long, String>(1L, "foo");
-
-// create a new vertex with a Long ID and no value
-Vertex<Long, NullValue> v = new Vertex<Long, NullValue>(1L, NullValue.getInstance());
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// create a new vertex with a Long ID and a String value
-val v = new Vertex(1L, "foo")
-
-// create a new vertex with a Long ID and no value
-val v = new Vertex(1L, NullValue.getInstance())
-{% endhighlight %}
-</div>
-</div>
-
-The graph edges are represented by the `Edge` type. An `Edge` is defined by a source ID (the ID of the source `Vertex`), a target ID (the ID of the target `Vertex`) and an optional value. The source and target IDs should be of the same type as the `Vertex` IDs. Edges with no value have a `NullValue` value type.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-Edge<Long, Double> e = new Edge<Long, Double>(1L, 2L, 0.5);
-
-// reverse the source and target of this edge
-Edge<Long, Double> reversed = e.reverse();
-
-Double weight = e.getValue(); // weight = 0.5
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val e = new Edge(1L, 2L, 0.5)
-
-// reverse the source and target of this edge
-val reversed = e.reverse
-
-val weight = e.getValue // weight = 0.5
-{% endhighlight %}
-</div>
-</div>
-
-In Gelly an `Edge` is always directed from the source vertex to the target vertex. A `Graph` may be undirected if for
-every `Edge` it contains a matching `Edge` from the target vertex to the source vertex.
-
-{% top %}
-
-Graph Creation
------------
-
-You can create a `Graph` in the following ways:
-
-* from a `DataSet` of edges and an optional `DataSet` of vertices:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-DataSet<Vertex<String, Long>> vertices = ...
-
-DataSet<Edge<String, Double>> edges = ...
-
-Graph<String, Long, Double> graph = Graph.fromDataSet(vertices, edges, env);
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = ExecutionEnvironment.getExecutionEnvironment
-
-val vertices: DataSet[Vertex[String, Long]] = ...
-
-val edges: DataSet[Edge[String, Double]] = ...
-
-val graph = Graph.fromDataSet(vertices, edges, env)
-{% endhighlight %}
-</div>
-</div>
-
-* from a `DataSet` of `Tuple2` representing the edges. Gelly will convert each `Tuple2` to an `Edge`, where the first field will be the source ID and the second field will be the target ID. Both vertex and edge values will be set to `NullValue`.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-DataSet<Tuple2<String, String>> edges = ...
-
-Graph<String, NullValue, NullValue> graph = Graph.fromTuple2DataSet(edges, env);
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = ExecutionEnvironment.getExecutionEnvironment
-
-val edges: DataSet[(String, String)] = ...
-
-val graph = Graph.fromTuple2DataSet(edges, env)
-{% endhighlight %}
-</div>
-</div>
-
-* from a `DataSet` of `Tuple3` and an optional `DataSet` of `Tuple2`. In this case, Gelly will convert each `Tuple3` to an `Edge`, where the first field will be the source ID, the second field will be the target ID and the third field will be the edge value. Equivalently, each `Tuple2` will be converted to a `Vertex`, where the first field will be the vertex ID and the second field will be the vertex value:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-DataSet<Tuple2<String, Long>> vertexTuples = env.readCsvFile("path/to/vertex/input").types(String.class, Long.class);
-
-DataSet<Tuple3<String, String, Double>> edgeTuples = env.readCsvFile("path/to/edge/input").types(String.class, String.class, Double.class);
-
-Graph<String, Long, Double> graph = Graph.fromTupleDataSet(vertexTuples, edgeTuples, env);
-{% endhighlight %}
-
-* from a CSV file of Edge data and an optional CSV file of Vertex data. In this case, Gelly will convert each row from the Edge CSV file to an `Edge`, where the first field will be the source ID, the second field will be the target ID and the third field (if present) will be the edge value. Equivalently, each row from the optional Vertex CSV file will be converted to a `Vertex`, where the first field will be the vertex ID and the second field (if present) will be the vertex value. In order to get a `Graph` from a `GraphCsvReader` one has to specify the types, using one of the following methods:
-
-- `types(Class<K> vertexKey, Class<VV> vertexValue,Class<EV> edgeValue)`: both vertex and edge values are present.
-- `edgeTypes(Class<K> vertexKey, Class<EV> edgeValue)`: the Graph has edge values, but no vertex values.
-- `vertexTypes(Class<K> vertexKey, Class<VV> vertexValue)`: the Graph has vertex values, but no edge values.
-- `keyType(Class<K> vertexKey)`: the Graph has no vertex values and no edge values.
-
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-// create a Graph with String Vertex IDs, Long Vertex values and Double Edge values
-Graph<String, Long, Double> graph = Graph.fromCsvReader("path/to/vertex/input", "path/to/edge/input", env)
- .types(String.class, Long.class, Double.class);
-
-
-// create a Graph with neither Vertex nor Edge values
-Graph<Long, NullValue, NullValue> simpleGraph = Graph.fromCsvReader("path/to/edge/input", env).keyType(Long.class);
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = ExecutionEnvironment.getExecutionEnvironment
-
-val vertexTuples = env.readCsvFile[String, Long]("path/to/vertex/input")
-
-val edgeTuples = env.readCsvFile[String, String, Double]("path/to/edge/input")
-
-val graph = Graph.fromTupleDataSet(vertexTuples, edgeTuples, env)
-{% endhighlight %}
-
-* from a CSV file of Edge data and an optional CSV file of Vertex data.
-In this case, Gelly will convert each row from the Edge CSV file to an `Edge`.
-The first field of the each row will be the source ID, the second field will be the target ID and the third field (if present) will be the edge value.
-If the edges have no associated value, set the edge value type parameter (3rd type argument) to `NullValue`.
-You can also specify that the vertices are initialized with a vertex value.
-If you provide a path to a CSV file via `pathVertices`, each row of this file will be converted to a `Vertex`.
-The first field of each row will be the vertex ID and the second field will be the vertex value.
-If you provide a vertex value initializer `MapFunction` via the `vertexValueInitializer` parameter, then this function is used to generate the vertex values.
-The set of vertices will be created automatically from the edges input.
-If the vertices have no associated value, set the vertex value type parameter (2nd type argument) to `NullValue`.
-The vertices will then be automatically created from the edges input with vertex value of type `NullValue`.
-
-{% highlight scala %}
-val env = ExecutionEnvironment.getExecutionEnvironment
-
-// create a Graph with String Vertex IDs, Long Vertex values and Double Edge values
-val graph = Graph.fromCsvReader[String, Long, Double](
- pathVertices = "path/to/vertex/input",
- pathEdges = "path/to/edge/input",
- env = env)
-
-
-// create a Graph with neither Vertex nor Edge values
-val simpleGraph = Graph.fromCsvReader[Long, NullValue, NullValue](
- pathEdges = "path/to/edge/input",
- env = env)
-
-// create a Graph with Double Vertex values generated by a vertex value initializer and no Edge values
-val simpleGraph = Graph.fromCsvReader[Long, Double, NullValue](
- pathEdges = "path/to/edge/input",
- vertexValueInitializer = new MapFunction[Long, Double]() {
- def map(id: Long): Double = {
- id.toDouble
- }
- },
- env = env)
-{% endhighlight %}
-</div>
-</div>
-
-
-* from a `Collection` of edges and an optional `Collection` of vertices:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-List<Vertex<Long, Long>> vertexList = new ArrayList...
-
-List<Edge<Long, String>> edgeList = new ArrayList...
-
-Graph<Long, Long, String> graph = Graph.fromCollection(vertexList, edgeList, env);
-{% endhighlight %}
-
-If no vertex input is provided during Graph creation, Gelly will automatically produce the `Vertex` `DataSet` from the edge input. In this case, the created vertices will have no values. Alternatively, you can provide a `MapFunction` as an argument to the creation method, in order to initialize the `Vertex` values:
-
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-// initialize the vertex value to be equal to the vertex ID
-Graph<Long, Long, String> graph = Graph.fromCollection(edgeList,
- new MapFunction<Long, Long>() {
- public Long map(Long value) {
- return value;
- }
- }, env);
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = ExecutionEnvironment.getExecutionEnvironment
-
-val vertexList = List(...)
-
-val edgeList = List(...)
-
-val graph = Graph.fromCollection(vertexList, edgeList, env)
-{% endhighlight %}
-
-If no vertex input is provided during Graph creation, Gelly will automatically produce the `Vertex` `DataSet` from the edge input. In this case, the created vertices will have no values. Alternatively, you can provide a `MapFunction` as an argument to the creation method, in order to initialize the `Vertex` values:
-
-{% highlight java %}
-val env = ExecutionEnvironment.getExecutionEnvironment
-
-// initialize the vertex value to be equal to the vertex ID
-val graph = Graph.fromCollection(edgeList,
- new MapFunction[Long, Long] {
- def map(id: Long): Long = id
- }, env)
-{% endhighlight %}
-</div>
-</div>
-
-{% top %}
-
-Graph Properties
-------------
-
-Gelly includes the following methods for retrieving various Graph properties and metrics:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-// get the Vertex DataSet
-DataSet<Vertex<K, VV>> getVertices()
-
-// get the Edge DataSet
-DataSet<Edge<K, EV>> getEdges()
-
-// get the IDs of the vertices as a DataSet
-DataSet<K> getVertexIds()
-
-// get the source-target pairs of the edge IDs as a DataSet
-DataSet<Tuple2<K, K>> getEdgeIds()
-
-// get a DataSet of <vertex ID, in-degree> pairs for all vertices
-DataSet<Tuple2<K, LongValue>> inDegrees()
-
-// get a DataSet of <vertex ID, out-degree> pairs for all vertices
-DataSet<Tuple2<K, LongValue>> outDegrees()
-
-// get a DataSet of <vertex ID, degree> pairs for all vertices, where degree is the sum of in- and out- degrees
-DataSet<Tuple2<K, LongValue>> getDegrees()
-
-// get the number of vertices
-long numberOfVertices()
-
-// get the number of edges
-long numberOfEdges()
-
-// get a DataSet of Triplets<srcVertex, trgVertex, edge>
-DataSet<Triplet<K, VV, EV>> getTriplets()
-
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// get the Vertex DataSet
-getVertices: DataSet[Vertex[K, VV]]
-
-// get the Edge DataSet
-getEdges: DataSet[Edge[K, EV]]
-
-// get the IDs of the vertices as a DataSet
-getVertexIds: DataSet[K]
-
-// get the source-target pairs of the edge IDs as a DataSet
-getEdgeIds: DataSet[(K, K)]
-
-// get a DataSet of <vertex ID, in-degree> pairs for all vertices
-inDegrees: DataSet[(K, LongValue)]
-
-// get a DataSet of <vertex ID, out-degree> pairs for all vertices
-outDegrees: DataSet[(K, LongValue)]
-
-// get a DataSet of <vertex ID, degree> pairs for all vertices, where degree is the sum of in- and out- degrees
-getDegrees: DataSet[(K, LongValue)]
-
-// get the number of vertices
-numberOfVertices: Long
-
-// get the number of edges
-numberOfEdges: Long
-
-// get a DataSet of Triplets<srcVertex, trgVertex, edge>
-getTriplets: DataSet[Triplet[K, VV, EV]]
-
-{% endhighlight %}
-</div>
-</div>
-
-{% top %}
-
-Graph Transformations
------------------
-
-* <strong>Map</strong>: Gelly provides specialized methods for applying a map transformation on the vertex values or edge values. `mapVertices` and `mapEdges` return a new `Graph`, where the IDs of the vertices (or edges) remain unchanged, while the values are transformed according to the provided user-defined map function. The map functions also allow changing the type of the vertex or edge values.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-Graph<Long, Long, Long> graph = Graph.fromDataSet(vertices, edges, env);
-
-// increment each vertex value by one
-Graph<Long, Long, Long> updatedGraph = graph.mapVertices(
- new MapFunction<Vertex<Long, Long>, Long>() {
- public Long map(Vertex<Long, Long> value) {
- return value.getValue() + 1;
- }
- });
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = ExecutionEnvironment.getExecutionEnvironment
-val graph = Graph.fromDataSet(vertices, edges, env)
-
-// increment each vertex value by one
-val updatedGraph = graph.mapVertices(v => v.getValue + 1)
-{% endhighlight %}
-</div>
-</div>
-
-* <strong>Translate</strong>: Gelly provides specialized methods for translating the value and/or type of vertex and edge IDs (`translateGraphIDs`), vertex values (`translateVertexValues`), or edge values (`translateEdgeValues`). Translation is performed by the user-defined map function, several of which are provided in the `org.apache.flink.graph.asm.translate` package. The same `MapFunction` can be used for all the three translate methods.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-Graph<Long, Long, Long> graph = Graph.fromDataSet(vertices, edges, env);
-
-// translate each vertex and edge ID to a String
-Graph<String, Long, Long> updatedGraph = graph.translateGraphIds(
- new MapFunction<Long, String>() {
- public String map(Long id) {
- return id.toString();
- }
- });
-
-// translate vertex IDs, edge IDs, vertex values, and edge values to LongValue
-Graph<LongValue, LongValue, LongValue> updatedGraph = graph
- .translateGraphIds(new LongToLongValue())
- .translateVertexValues(new LongToLongValue())
- .translateEdgeValues(new LongToLongValue())
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = ExecutionEnvironment.getExecutionEnvironment
-val graph = Graph.fromDataSet(vertices, edges, env)
-
-// translate each vertex and edge ID to a String
-val updatedGraph = graph.translateGraphIds(id => id.toString)
-{% endhighlight %}
-</div>
-</div>
-
-
-* <strong>Filter</strong>: A filter transformation applies a user-defined filter function on the vertices or edges of the `Graph`. `filterOnEdges` will create a sub-graph of the original graph, keeping only the edges that satisfy the provided predicate. Note that the vertex dataset will not be modified. Respectively, `filterOnVertices` applies a filter on the vertices of the graph. Edges whose source and/or target do not satisfy the vertex predicate are removed from the resulting edge dataset. The `subgraph` method can be used to apply a filter function to the vertices and the edges at the same time.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-Graph<Long, Long, Long> graph = ...
-
-graph.subgraph(
- new FilterFunction<Vertex<Long, Long>>() {
- public boolean filter(Vertex<Long, Long> vertex) {
- // keep only vertices with positive values
- return (vertex.getValue() > 0);
- }
- },
- new FilterFunction<Edge<Long, Long>>() {
- public boolean filter(Edge<Long, Long> edge) {
- // keep only edges with negative values
- return (edge.getValue() < 0);
- }
- })
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val graph: Graph[Long, Long, Long] = ...
-
-// keep only vertices with positive values
-// and only edges with negative values
-graph.subgraph((vertex => vertex.getValue > 0), (edge => edge.getValue < 0))
-{% endhighlight %}
-</div>
-</div>
-
-<p class="text-center">
- <img alt="Filter Transformations" width="80%" src="fig/gelly-filter.png"/>
-</p>
-
-* <strong>Join</strong>: Gelly provides specialized methods for joining the vertex and edge datasets with other input datasets. `joinWithVertices` joins the vertices with a `Tuple2` input data set. The join is performed using the vertex ID and the first field of the `Tuple2` input as the join keys. The method returns a new `Graph` where the vertex values have been updated according to a provided user-defined transformation function.
-Similarly, an input dataset can be joined with the edges, using one of three methods. `joinWithEdges` expects an input `DataSet` of `Tuple3` and joins on the composite key of both source and target vertex IDs. `joinWithEdgesOnSource` expects a `DataSet` of `Tuple2` and joins on the source key of the edges and the first attribute of the input dataset and `joinWithEdgesOnTarget` expects a `DataSet` of `Tuple2` and joins on the target key of the edges and the first attribute of the input dataset. All three methods apply a transformation function on the edge and the input data set values.
-Note that if the input dataset contains a key multiple times, all Gelly join methods will only consider the first value encountered.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-Graph<Long, Double, Double> network = ...
-
-DataSet<Tuple2<Long, LongValue>> vertexOutDegrees = network.outDegrees();
-
-// assign the transition probabilities as the edge weights
-Graph<Long, Double, Double> networkWithWeights = network.joinWithEdgesOnSource(vertexOutDegrees,
- new VertexJoinFunction<Double, LongValue>() {
- public Double vertexJoin(Double vertexValue, LongValue inputValue) {
- return vertexValue / inputValue.getValue();
- }
- });
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val network: Graph[Long, Double, Double] = ...
-
-val vertexOutDegrees: DataSet[(Long, LongValue)] = network.outDegrees
-
-// assign the transition probabilities as the edge weights
-val networkWithWeights = network.joinWithEdgesOnSource(vertexOutDegrees, (v1: Double, v2: LongValue) => v1 / v2.getValue)
-{% endhighlight %}
-</div>
-</div>
-
-* <strong>Reverse</strong>: the `reverse()` method returns a new `Graph` where the direction of all edges has been reversed.
-
-* <strong>Undirected</strong>: In Gelly, a `Graph` is always directed. Undirected graphs can be represented by adding all opposite-direction edges to a graph. For this purpose, Gelly provides the `getUndirected()` method.
-
-* <strong>Union</strong>: Gelly's `union()` method performs a union operation on the vertex and edge sets of the specified graph and the current graph. Duplicate vertices are removed from the resulting `Graph`, while if duplicate edges exist, these will be preserved.
-
-<p class="text-center">
- <img alt="Union Transformation" width="50%" src="fig/gelly-union.png"/>
-</p>
-
-* <strong>Difference</strong>: Gelly's `difference()` method performs a difference on the vertex and edge sets of the current graph and the specified graph.
-
-* <strong>Intersect</strong>: Gelly's `intersect()` method performs an intersect on the edge
- sets of the current graph and the specified graph. The result is a new `Graph` that contains all
- edges that exist in both input graphs. Two edges are considered equal, if they have the same source
- identifier, target identifier and edge value. Vertices in the resulting graph have no
- value. If vertex values are required, one can for example retrieve them from one of the input graphs using
- the `joinWithVertices()` method.
- Depending on the parameter `distinct`, equal edges are either contained once in the resulting
- `Graph` or as often as there are pairs of equal edges in the input graphs.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-// create first graph from edges {(1, 3, 12) (1, 3, 13), (1, 3, 13)}
-List<Edge<Long, Long>> edges1 = ...
-Graph<Long, NullValue, Long> graph1 = Graph.fromCollection(edges1, env);
-
-// create second graph from edges {(1, 3, 13)}
-List<Edge<Long, Long>> edges2 = ...
-Graph<Long, NullValue, Long> graph2 = Graph.fromCollection(edges2, env);
-
-// Using distinct = true results in {(1,3,13)}
-Graph<Long, NullValue, Long> intersect1 = graph1.intersect(graph2, true);
-
-// Using distinct = false results in {(1,3,13),(1,3,13)} as there is one edge pair
-Graph<Long, NullValue, Long> intersect2 = graph1.intersect(graph2, false);
-
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = ExecutionEnvironment.getExecutionEnvironment
-
-// create first graph from edges {(1, 3, 12) (1, 3, 13), (1, 3, 13)}
-val edges1: List[Edge[Long, Long]] = ...
-val graph1 = Graph.fromCollection(edges1, env)
-
-// create second graph from edges {(1, 3, 13)}
-val edges2: List[Edge[Long, Long]] = ...
-val graph2 = Graph.fromCollection(edges2, env)
-
-
-// Using distinct = true results in {(1,3,13)}
-val intersect1 = graph1.intersect(graph2, true)
-
-// Using distinct = false results in {(1,3,13),(1,3,13)} as there is one edge pair
-val intersect2 = graph1.intersect(graph2, false)
-{% endhighlight %}
-</div>
-</div>
-
--{% top %}
-
-Graph Mutations
------------
-
-Gelly includes the following methods for adding and removing vertices and edges from an input `Graph`:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-// adds a Vertex to the Graph. If the Vertex already exists, it will not be added again.
-Graph<K, VV, EV> addVertex(final Vertex<K, VV> vertex)
-
-// adds a list of vertices to the Graph. If the vertices already exist in the graph, they will not be added once more.
-Graph<K, VV, EV> addVertices(List<Vertex<K, VV>> verticesToAdd)
-
-// adds an Edge to the Graph. If the source and target vertices do not exist in the graph, they will also be added.
-Graph<K, VV, EV> addEdge(Vertex<K, VV> source, Vertex<K, VV> target, EV edgeValue)
-
-// adds a list of edges to the Graph. When adding an edge for a non-existing set of vertices, the edge is considered invalid and ignored.
-Graph<K, VV, EV> addEdges(List<Edge<K, EV>> newEdges)
-
-// removes the given Vertex and its edges from the Graph.
-Graph<K, VV, EV> removeVertex(Vertex<K, VV> vertex)
-
-// removes the given list of vertices and their edges from the Graph
-Graph<K, VV, EV> removeVertices(List<Vertex<K, VV>> verticesToBeRemoved)
-
-// removes *all* edges that match the given Edge from the Graph.
-Graph<K, VV, EV> removeEdge(Edge<K, EV> edge)
-
-// removes *all* edges that match the edges in the given list
-Graph<K, VV, EV> removeEdges(List<Edge<K, EV>> edgesToBeRemoved)
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// adds a Vertex to the Graph. If the Vertex already exists, it will not be added again.
-addVertex(vertex: Vertex[K, VV])
-
-// adds a list of vertices to the Graph. If the vertices already exist in the graph, they will not be added once more.
-addVertices(verticesToAdd: List[Vertex[K, VV]])
-
-// adds an Edge to the Graph. If the source and target vertices do not exist in the graph, they will also be added.
-addEdge(source: Vertex[K, VV], target: Vertex[K, VV], edgeValue: EV)
-
-// adds a list of edges to the Graph. When adding an edge for a non-existing set of vertices, the edge is considered invalid and ignored.
-addEdges(edges: List[Edge[K, EV]])
-
-// removes the given Vertex and its edges from the Graph.
-removeVertex(vertex: Vertex[K, VV])
-
-// removes the given list of vertices and their edges from the Graph
-removeVertices(verticesToBeRemoved: List[Vertex[K, VV]])
-
-// removes *all* edges that match the given Edge from the Graph.
-removeEdge(edge: Edge[K, EV])
-
-// removes *all* edges that match the edges in the given list
-removeEdges(edgesToBeRemoved: List[Edge[K, EV]])
-{% endhighlight %}
-</div>
-</div>
-
-Neighborhood Methods
------------
-
-Neighborhood methods allow vertices to perform an aggregation on their first-hop neighborhood.
-`reduceOnEdges()` can be used to compute an aggregation on the values of the neighboring edges of a vertex and `reduceOnNeighbors()` can be used to compute an aggregation on the values of the neighboring vertices. These methods assume associative and commutative aggregations and exploit combiners internally, significantly improving performance.
-The neighborhood scope is defined by the `EdgeDirection` parameter, which takes the values `IN`, `OUT` or `ALL`. `IN` will gather all in-coming edges (neighbors) of a vertex, `OUT` will gather all out-going edges (neighbors), while `ALL` will gather all edges (neighbors).
-
-For example, assume that you want to select the minimum weight of all out-edges for each vertex in the following graph:
-
-<p class="text-center">
- <img alt="reduceOnEdges Example" width="50%" src="fig/gelly-example-graph.png"/>
-</p>
-
-The following code will collect the out-edges for each vertex and apply the `SelectMinWeight()` user-defined function on each of the resulting neighborhoods:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-Graph<Long, Long, Double> graph = ...
-
-DataSet<Tuple2<Long, Double>> minWeights = graph.reduceOnEdges(new SelectMinWeight(), EdgeDirection.OUT);
-
-// user-defined function to select the minimum weight
-static final class SelectMinWeight implements ReduceEdgesFunction<Double> {
-
- @Override
- public Double reduceEdges(Double firstEdgeValue, Double secondEdgeValue) {
- return Math.min(firstEdgeValue, secondEdgeValue);
- }
-}
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val graph: Graph[Long, Long, Double] = ...
-
-val minWeights = graph.reduceOnEdges(new SelectMinWeight, EdgeDirection.OUT)
-
-// user-defined function to select the minimum weight
-final class SelectMinWeight extends ReduceEdgesFunction[Double] {
- override def reduceEdges(firstEdgeValue: Double, secondEdgeValue: Double): Double = {
- Math.min(firstEdgeValue, secondEdgeValue)
- }
- }
-{% endhighlight %}
-</div>
-</div>
-
-<p class="text-center">
- <img alt="reduceOnEdges Example" width="50%" src="fig/gelly-reduceOnEdges.png"/>
-</p>
-
-Similarly, assume that you would like to compute the sum of the values of all in-coming neighbors, for every vertex. The following code will collect the in-coming neighbors for each vertex and apply the `SumValues()` user-defined function on each neighborhood:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-Graph<Long, Long, Double> graph = ...
-
-DataSet<Tuple2<Long, Long>> verticesWithSum = graph.reduceOnNeighbors(new SumValues(), EdgeDirection.IN);
-
-// user-defined function to sum the neighbor values
-static final class SumValues implements ReduceNeighborsFunction<Long> {
-
- @Override
- public Long reduceNeighbors(Long firstNeighbor, Long secondNeighbor) {
- return firstNeighbor + secondNeighbor;
- }
-}
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val graph: Graph[Long, Long, Double] = ...
-
-val verticesWithSum = graph.reduceOnNeighbors(new SumValues, EdgeDirection.IN)
-
-// user-defined function to sum the neighbor values
-final class SumValues extends ReduceNeighborsFunction[Long] {
- override def reduceNeighbors(firstNeighbor: Long, secondNeighbor: Long): Long = {
- firstNeighbor + secondNeighbor
- }
-}
-{% endhighlight %}
-</div>
-</div>
-
-<p class="text-center">
- <img alt="reduceOnNeighbors Example" width="70%" src="fig/gelly-reduceOnNeighbors.png"/>
-</p>
-
-When the aggregation function is not associative and commutative or when it is desirable to return more than one values per vertex, one can use the more general
-`groupReduceOnEdges()` and `groupReduceOnNeighbors()` methods.
-These methods return zero, one or more values per vertex and provide access to the whole neighborhood.
-
-For example, the following code will output all the vertex pairs which are connected with an edge having a weight of 0.5 or more:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-Graph<Long, Long, Double> graph = ...
-
-DataSet<Tuple2<Vertex<Long, Long>, Vertex<Long, Long>>> vertexPairs = graph.groupReduceOnNeighbors(new SelectLargeWeightNeighbors(), EdgeDirection.OUT);
-
-// user-defined function to select the neighbors which have edges with weight > 0.5
-static final class SelectLargeWeightNeighbors implements NeighborsFunctionWithVertexValue<Long, Long, Double,
- Tuple2<Vertex<Long, Long>, Vertex<Long, Long>>> {
-
- @Override
- public void iterateNeighbors(Vertex<Long, Long> vertex,
- Iterable<Tuple2<Edge<Long, Double>, Vertex<Long, Long>>> neighbors,
- Collector<Tuple2<Vertex<Long, Long>, Vertex<Long, Long>>> out) {
-
- for (Tuple2<Edge<Long, Double>, Vertex<Long, Long>> neighbor : neighbors) {
- if (neighbor.f0.f2 > 0.5) {
- out.collect(new Tuple2<Vertex<Long, Long>, Vertex<Long, Long>>(vertex, neighbor.f1));
- }
- }
- }
-}
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val graph: Graph[Long, Long, Double] = ...
-
-val vertexPairs = graph.groupReduceOnNeighbors(new SelectLargeWeightNeighbors, EdgeDirection.OUT)
-
-// user-defined function to select the neighbors which have edges with weight > 0.5
-final class SelectLargeWeightNeighbors extends NeighborsFunctionWithVertexValue[Long, Long, Double,
- (Vertex[Long, Long], Vertex[Long, Long])] {
-
- override def iterateNeighbors(vertex: Vertex[Long, Long],
- neighbors: Iterable[(Edge[Long, Double], Vertex[Long, Long])],
- out: Collector[(Vertex[Long, Long], Vertex[Long, Long])]) = {
-
- for (neighbor <- neighbors) {
- if (neighbor._1.getValue() > 0.5) {
- out.collect(vertex, neighbor._2);
- }
- }
- }
- }
-{% endhighlight %}
-</div>
-</div>
-
-When the aggregation computation does not require access to the vertex value (for which the aggregation is performed), it is advised to use the more efficient `EdgesFunction` and `NeighborsFunction` for the user-defined functions. When access to the vertex value is required, one should use `EdgesFunctionWithVertexValue` and `NeighborsFunctionWithVertexValue` instead.
-
-{% top %}
-
-Graph Validation
------------
-
-Gelly provides a simple utility for performing validation checks on input graphs. Depending on the application context, a graph may or may not be valid according to certain criteria. For example, a user might need to validate whether their graph contains duplicate edges or whether its structure is bipartite. In order to validate a graph, one can define a custom `GraphValidator` and implement its `validate()` method. `InvalidVertexIdsValidator` is Gelly's pre-defined validator. It checks that the edge set contains valid vertex IDs, i.e. that all edge IDs
-also exist in the vertex IDs set.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-// create a list of vertices with IDs = {1, 2, 3, 4, 5}
-List<Vertex<Long, Long>> vertices = ...
-
-// create a list of edges with IDs = {(1, 2) (1, 3), (2, 4), (5, 6)}
-List<Edge<Long, Long>> edges = ...
-
-Graph<Long, Long, Long> graph = Graph.fromCollection(vertices, edges, env);
-
-// will return false: 6 is an invalid ID
-graph.validate(new InvalidVertexIdsValidator<Long, Long, Long>());
-
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = ExecutionEnvironment.getExecutionEnvironment
-
-// create a list of vertices with IDs = {1, 2, 3, 4, 5}
-val vertices: List[Vertex[Long, Long]] = ...
-
-// create a list of edges with IDs = {(1, 2) (1, 3), (2, 4), (5, 6)}
-val edges: List[Edge[Long, Long]] = ...
-
-val graph = Graph.fromCollection(vertices, edges, env)
-
-// will return false: 6 is an invalid ID
-graph.validate(new InvalidVertexIdsValidator[Long, Long, Long])
-
-{% endhighlight %}
-</div>
-</div>
-
-{% top %}
http://git-wip-us.apache.org/repos/asf/flink/blob/844c874b/docs/apis/batch/libs/gelly/graph_generators.md
----------------------------------------------------------------------
diff --git a/docs/apis/batch/libs/gelly/graph_generators.md b/docs/apis/batch/libs/gelly/graph_generators.md
deleted file mode 100644
index 029bede..0000000
--- a/docs/apis/batch/libs/gelly/graph_generators.md
+++ /dev/null
@@ -1,657 +0,0 @@
----
-title: Graph Generators
-
-# Sub navigation
-sub-nav-group: batch
-sub-nav-parent: gelly
-sub-nav-title: Graph Generators
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-* This will be replaced by the TOC
-{:toc}
-
-Gelly provides a collection of scalable graph generators. Each generator is
-
-* parallelizable, in order to create large datasets
-* scale-free, generating the same graph regardless of parallelism
-* thrifty, using as few operators as possible
-
-Graph generators are configured using the builder pattern. The parallelism of generator
-operators can be set explicitly by calling `setParallelism(parallelism)`. Lowering the
-parallelism will reduce the allocation of memory and network buffers.
-
-Graph-specific configuration must be called first, then configuration common to all
-generators, and lastly the call to `generate()`. The following example configures a
-grid graph with two dimensions, configures the parallelism, and generates the graph.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-boolean wrapEndpoints = false;
-
-int parallelism = 4;
-
-Graph<LongValue,NullValue,NullValue> graph = new GridGraph(env)
- .addDimension(2, wrapEndpoints)
- .addDimension(4, wrapEndpoints)
- .setParallelism(parallelism)
- .generate();
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-import org.apache.flink.api.scala._
-import org.apache.flink.graph.generator.GridGraph
-
-val env: ExecutionEnvironment = ExecutionEnvironment.getExecutionEnvironment
-
-wrapEndpoints = false
-
-val parallelism = 4
-
-val graph = new GridGraph(env.getJavaEnv).addDimension(2, wrapEndpoints).addDimension(4, wrapEndpoints).setParallelism(parallelism).generate()
-{% endhighlight %}
-</div>
-</div>
-
-## Complete Graph
-
-An undirected graph connecting every distinct pair of vertices.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-long vertexCount = 5;
-
-Graph<LongValue,NullValue,NullValue> graph = new CompleteGraph(env, vertexCount)
- .generate();
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-import org.apache.flink.api.scala._
-import org.apache.flink.graph.generator.CompleteGraph
-
-val env: ExecutionEnvironment = ExecutionEnvironment.getExecutionEnvironment
-
-val vertexCount = 5
-
-val graph = new CompleteGraph(env.getJavaEnv, vertexCount).generate()
-{% endhighlight %}
-</div>
-</div>
-
-<svg class="graph" width="540" height="540"
- xmlns="http://www.w3.org/2000/svg"
- xmlns:xlink="http://www.w3.org/1999/xlink">
-
- <line x1="270" y1="40" x2="489" y2="199" />
- <line x1="270" y1="40" x2="405" y2="456" />
- <line x1="270" y1="40" x2="135" y2="456" />
- <line x1="270" y1="40" x2="51" y2="199" />
-
- <line x1="489" y1="199" x2="405" y2="456" />
- <line x1="489" y1="199" x2="135" y2="456" />
- <line x1="489" y1="199" x2="51" y2="199" />
-
- <line x1="405" y1="456" x2="135" y2="456" />
- <line x1="405" y1="456" x2="51" y2="199" />
-
- <line x1="135" y1="456" x2="51" y2="199" />
-
- <circle cx="270" cy="40" r="20" />
- <text x="270" y="40">0</text>
-
- <circle cx="489" cy="199" r="20" />
- <text x="489" y="199">1</text>
-
- <circle cx="405" cy="456" r="20" />
- <text x="405" y="456">2</text>
-
- <circle cx="135" cy="456" r="20" />
- <text x="135" y="456">3</text>
-
- <circle cx="51" cy="199" r="20" />
- <text x="51" y="199">4</text>
-</svg>
-
-## Cycle Graph
-
-An undirected graph where all edges form a single cycle.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-long vertexCount = 5;
-
-Graph<LongValue,NullValue,NullValue> graph = new CycleGraph(env, vertexCount)
- .generate();
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-import org.apache.flink.api.scala._
-import org.apache.flink.graph.generator.CycleGraph
-
-val env: ExecutionEnvironment = ExecutionEnvironment.getExecutionEnvironment
-
-val vertexCount = 5
-
-val graph = new CycleGraph(env.getJavaEnv, vertexCount).generate()
-{% endhighlight %}
-</div>
-</div>
-
-<svg class="graph" width="540" height="540"
- xmlns="http://www.w3.org/2000/svg"
- xmlns:xlink="http://www.w3.org/1999/xlink">
-
- <line x1="270" y1="40" x2="489" y2="199" />
- <line x1="489" y1="199" x2="405" y2="456" />
- <line x1="405" y1="456" x2="135" y2="456" />
- <line x1="135" y1="456" x2="51" y2="199" />
- <line x1="51" y1="199" x2="270" y2="40" />
-
- <circle cx="270" cy="40" r="20" />
- <text x="270" y="40">0</text>
-
- <circle cx="489" cy="199" r="20" />
- <text x="489" y="199">1</text>
-
- <circle cx="405" cy="456" r="20" />
- <text x="405" y="456">2</text>
-
- <circle cx="135" cy="456" r="20" />
- <text x="135" y="456">3</text>
-
- <circle cx="51" cy="199" r="20" />
- <text x="51" y="199">4</text>
-</svg>
-
-## Empty Graph
-
-The graph containing no edges.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-long vertexCount = 5;
-
-Graph<LongValue,NullValue,NullValue> graph = new EmptyGraph(env, vertexCount)
- .generate();
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-import org.apache.flink.api.scala._
-import org.apache.flink.graph.generator.EmptyGraph
-
-val env: ExecutionEnvironment = ExecutionEnvironment.getExecutionEnvironment
-
-val vertexCount = 5
-
-val graph = new EmptyGraph(env.getJavaEnv, vertexCount).generate()
-{% endhighlight %}
-</div>
-</div>
-
-<svg class="graph" width="540" height="80"
- xmlns="http://www.w3.org/2000/svg"
- xmlns:xlink="http://www.w3.org/1999/xlink">
-
- <circle cx="30" cy="40" r="20" />
- <text x="30" y="40">0</text>
-
- <circle cx="150" cy="40" r="20" />
- <text x="150" y="40">1</text>
-
- <circle cx="270" cy="40" r="20" />
- <text x="270" y="40">2</text>
-
- <circle cx="390" cy="40" r="20" />
- <text x="390" y="40">3</text>
-
- <circle cx="510" cy="40" r="20" />
- <text x="510" y="40">4</text>
-</svg>
-
-## Grid Graph
-
-An undirected graph connecting vertices in a regular tiling in one or more dimensions.
-Each dimension is configured separately. When the dimension size is at least three the
-endpoints are optionally connected by setting `wrapEndpoints`. Changing the following
-example to `addDimension(4, true)` would connect `0` to `3` and `4` to `7`.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-boolean wrapEndpoints = false;
-
-Graph<LongValue,NullValue,NullValue> graph = new GridGraph(env)
- .addDimension(2, wrapEndpoints)
- .addDimension(4, wrapEndpoints)
- .generate();
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-import org.apache.flink.api.scala._
-import org.apache.flink.graph.generator.GridGraph
-
-val env: ExecutionEnvironment = ExecutionEnvironment.getExecutionEnvironment
-
-val wrapEndpoints = false
-
-val graph = new GridGraph(env.getJavaEnv).addDimension(2, wrapEndpoints).addDimension(4, wrapEndpoints).generate()
-{% endhighlight %}
-</div>
-</div>
-
-<svg class="graph" width="540" height="200"
- xmlns="http://www.w3.org/2000/svg"
- xmlns:xlink="http://www.w3.org/1999/xlink">
-
- <line x1="30" y1="40" x2="510" y2="40" />
- <line x1="30" y1="160" x2="510" y2="160" />
-
- <line x1="30" y1="40" x2="30" y2="160" />
- <line x1="190" y1="40" x2="190" y2="160" />
- <line x1="350" y1="40" x2="350" y2="160" />
- <line x1="510" y1="40" x2="510" y2="160" />
-
- <circle cx="30" cy="40" r="20" />
- <text x="30" y="40">0</text>
-
- <circle cx="190" cy="40" r="20" />
- <text x="190" y="40">1</text>
-
- <circle cx="350" cy="40" r="20" />
- <text x="350" y="40">2</text>
-
- <circle cx="510" cy="40" r="20" />
- <text x="510" y="40">3</text>
-
- <circle cx="30" cy="160" r="20" />
- <text x="30" y="160">4</text>
-
- <circle cx="190" cy="160" r="20" />
- <text x="190" y="160">5</text>
-
- <circle cx="350" cy="160" r="20" />
- <text x="350" y="160">6</text>
-
- <circle cx="510" cy="160" r="20" />
- <text x="510" y="160">7</text>
-</svg>
-
-## Hypercube Graph
-
-An undirected graph where edges form an n-dimensional hypercube. Each vertex
-in a hypercube connects to one other vertex in each dimension.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-long dimensions = 3;
-
-Graph<LongValue,NullValue,NullValue> graph = new HypercubeGraph(env, dimensions)
- .generate();
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-import org.apache.flink.api.scala._
-import org.apache.flink.graph.generator.HypercubeGraph
-
-val env: ExecutionEnvironment = ExecutionEnvironment.getExecutionEnvironment
-
-val dimensions = 3
-
-val graph = new HypercubeGraph(env.getJavaEnv, dimensions).generate()
-{% endhighlight %}
-</div>
-</div>
-
-<svg class="graph" width="540" height="320"
- xmlns="http://www.w3.org/2000/svg"
- xmlns:xlink="http://www.w3.org/1999/xlink">
-
- <line x1="190" y1="120" x2="350" y2="120" />
- <line x1="190" y1="200" x2="350" y2="200" />
- <line x1="190" y1="120" x2="190" y2="200" />
- <line x1="350" y1="120" x2="350" y2="200" />
-
- <line x1="30" y1="40" x2="510" y2="40" />
- <line x1="30" y1="280" x2="510" y2="280" />
- <line x1="30" y1="40" x2="30" y2="280" />
- <line x1="510" y1="40" x2="510" y2="280" />
-
- <line x1="190" y1="120" x2="30" y2="40" />
- <line x1="350" y1="120" x2="510" y2="40" />
- <line x1="190" y1="200" x2="30" y2="280" />
- <line x1="350" y1="200" x2="510" y2="280" />
-
- <circle cx="190" cy="120" r="20" />
- <text x="190" y="120">0</text>
-
- <circle cx="350" cy="120" r="20" />
- <text x="350" y="120">1</text>
-
- <circle cx="190" cy="200" r="20" />
- <text x="190" y="200">2</text>
-
- <circle cx="350" cy="200" r="20" />
- <text x="350" y="200">3</text>
-
- <circle cx="30" cy="40" r="20" />
- <text x="30" y="40">4</text>
-
- <circle cx="510" cy="40" r="20" />
- <text x="510" y="40">5</text>
-
- <circle cx="30" cy="280" r="20" />
- <text x="30" y="280">6</text>
-
- <circle cx="510" cy="280" r="20" />
- <text x="510" y="280">7</text>
-</svg>
-
-## Path Graph
-
-An undirected Graph where all edges form a single path.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-long vertexCount = 5
-
-Graph<LongValue,NullValue,NullValue> graph = new PathGraph(env, vertexCount)
- .generate();
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-import org.apache.flink.api.scala._
-import org.apache.flink.graph.generator.PathGraph
-
-val env: ExecutionEnvironment = ExecutionEnvironment.getExecutionEnvironment
-
-val vertexCount = 5
-
-val graph = new PathGraph(env.getJavaEnv, vertexCount).generate()
-{% endhighlight %}
-</div>
-</div>
-
-<svg class="graph" width="540" height="80"
- xmlns="http://www.w3.org/2000/svg"
- xmlns:xlink="http://www.w3.org/1999/xlink">
-
- <line x1="30" y1="40" x2="510" y2="40" />
-
- <circle cx="30" cy="40" r="20" />
- <text x="30" y="40">0</text>
-
- <circle cx="150" cy="40" r="20" />
- <text x="150" y="40">1</text>
-
- <circle cx="270" cy="40" r="20" />
- <text x="270" y="40">2</text>
-
- <circle cx="390" cy="40" r="20" />
- <text x="390" y="40">3</text>
-
- <circle cx="510" cy="40" r="20" />
- <text x="510" y="40">4</text>
-</svg>
-
-## RMat Graph
-
-A directed or undirected power-law graph generated using the
-[Recursive Matrix (R-Mat)](http://www.cs.cmu.edu/~christos/PUBLICATIONS/siam04.pdf) model.
-
-RMat is a stochastic generator configured with a source of randomness implementing the
-`RandomGenerableFactory` interface. Provided implementations are `JDKRandomGeneratorFactory`
-and `MersenneTwisterFactory`. These generate an initial sequence of random values which are
-then used as seeds for generating the edges.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-RandomGenerableFactory<JDKRandomGenerator> rnd = new JDKRandomGeneratorFactory();
-
-int vertexCount = 1 << scale;
-int edgeCount = edgeFactor * vertexCount;
-
-Graph<LongValue,NullValue,NullValue> graph = new RMatGraph<>(env, rnd, vertexCount, edgeCount)
- .generate();
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-import org.apache.flink.api.scala._
-import org.apache.flink.graph.generator.RMatGraph
-
-val env = ExecutionEnvironment.getExecutionEnvironment
-
-val vertexCount = 1 << scale
-val edgeCount = edgeFactor * vertexCount
-
-val graph = new RMatGraph(env.getJavaEnv, rnd, vertexCount, edgeCount).generate()
-{% endhighlight %}
-</div>
-</div>
-
-The default RMat contants can be overridden as shown in the following example.
-The contants define the interdependence of bits from each generated edge's source
-and target labels. The RMat noise can be enabled and progressively perturbs the
-contants while generating each edge.
-
-The RMat generator can be configured to produce a simple graph by removing self-loops
-and duplicate edges. Symmetrization is performed either by a "clip-and-flip" throwing away
-the half matrix above the diagonal or a full "flip" preserving and mirroring all edges.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-RandomGenerableFactory<JDKRandomGenerator> rnd = new JDKRandomGeneratorFactory();
-
-int vertexCount = 1 << scale;
-int edgeCount = edgeFactor * vertexCount;
-
-boolean clipAndFlip = false;
-
-Graph<LongValue,NullValue,NullValue> graph = new RMatGraph<>(env, rnd, vertexCount, edgeCount)
- .setConstants(0.57f, 0.19f, 0.19f)
- .setNoise(true, 0.10f)
- .generate();
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-import org.apache.flink.api.scala._
-import org.apache.flink.graph.generator.RMatGraph
-
-val env = ExecutionEnvironment.getExecutionEnvironment
-
-val vertexCount = 1 << scale
-val edgeCount = edgeFactor * vertexCount
-
-clipAndFlip = false
-
-val graph = new RMatGraph(env.getJavaEnv, rnd, vertexCount, edgeCount).setConstants(0.57f, 0.19f, 0.19f).setNoise(true, 0.10f).generate()
-{% endhighlight %}
-</div>
-</div>
-
-## Singleton Edge Graph
-
-An undirected graph containing isolated two-paths.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-long vertexPairCount = 4
-
-// note: configured with the number of vertex pairs
-Graph<LongValue,NullValue,NullValue> graph = new SingletonEdgeGraph(env, vertexPairCount)
- .generate();
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-import org.apache.flink.api.scala._
-import org.apache.flink.graph.generator.SingletonEdgeGraph
-
-val env: ExecutionEnvironment = ExecutionEnvironment.getExecutionEnvironment
-
-val vertexPairCount = 4
-
-// note: configured with the number of vertex pairs
-val graph = new SingletonEdgeGraph(env.getJavaEnv, vertexPairCount).generate()
-{% endhighlight %}
-</div>
-</div>
-
-<svg class="graph" width="540" height="200"
- xmlns="http://www.w3.org/2000/svg"
- xmlns:xlink="http://www.w3.org/1999/xlink">
-
- <line x1="30" y1="40" x2="190" y2="40" />
- <line x1="350" y1="40" x2="510" y2="40" />
- <line x1="30" y1="160" x2="190" y2="160" />
- <line x1="350" y1="160" x2="510" y2="160" />
-
- <circle cx="30" cy="40" r="20" />
- <text x="30" y="40">0</text>
-
- <circle cx="190" cy="40" r="20" />
- <text x="190" y="40">1</text>
-
- <circle cx="350" cy="40" r="20" />
- <text x="350" y="40">2</text>
-
- <circle cx="510" cy="40" r="20" />
- <text x="510" y="40">3</text>
-
- <circle cx="30" cy="160" r="20" />
- <text x="30" y="160">4</text>
-
- <circle cx="190" cy="160" r="20" />
- <text x="190" y="160">5</text>
-
- <circle cx="350" cy="160" r="20" />
- <text x="350" y="160">6</text>
-
- <circle cx="510" cy="160" r="20" />
- <text x="510" y="160">7</text>
-</svg>
-
-## Star Graph
-
-An undirected graph containing a single central vertex connected to all other leaf vertices.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-long vertexCount = 6;
-
-Graph<LongValue,NullValue,NullValue> graph = new StarGraph(env, vertexCount)
- .generate();
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-import org.apache.flink.api.scala._
-import org.apache.flink.graph.generator.StarGraph
-
-val env: ExecutionEnvironment = ExecutionEnvironment.getExecutionEnvironment
-
-val vertexCount = 6
-
-val graph = new StarGraph(env.getJavaEnv, vertexCount).generate()
-{% endhighlight %}
-</div>
-</div>
-
-<svg class="graph" width="540" height="540"
- xmlns="http://www.w3.org/2000/svg"
- xmlns:xlink="http://www.w3.org/1999/xlink">
-
- <line x1="270" y1="270" x2="270" y2="40" />
- <line x1="270" y1="270" x2="489" y2="199" />
- <line x1="270" y1="270" x2="405" y2="456" />
- <line x1="270" y1="270" x2="135" y2="456" />
- <line x1="270" y1="270" x2="51" y2="199" />
-
- <circle cx="270" cy="270" r="20" />
- <text x="270" y="270">0</text>
-
- <circle cx="270" cy="40" r="20" />
- <text x="270" y="40">1</text>
-
- <circle cx="489" cy="199" r="20" />
- <text x="489" y="199">2</text>
-
- <circle cx="405" cy="456" r="20" />
- <text x="405" y="456">3</text>
-
- <circle cx="135" cy="456" r="20" />
- <text x="135" y="456">4</text>
-
- <circle cx="51" cy="199" r="20" />
- <text x="51" y="199">5</text>
-</svg>
-
-{% top %}