You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by me...@apache.org on 2018/07/22 23:09:47 UTC
[beam-site] 01/03: Add Flink instructions to portability contrib
page.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit bd6b56430b834ef6fc0b90740766d076d5df4306
Author: Thomas Weise <th...@apache.org>
AuthorDate: Fri Jul 20 13:41:09 2018 -0700
Add Flink instructions to portability contrib page.
---
src/contribute/portability.md | 22 +++++++++++++++++-----
1 file changed, 17 insertions(+), 5 deletions(-)
diff --git a/src/contribute/portability.md b/src/contribute/portability.md
index cd17a92..cb16a66 100644
--- a/src/contribute/portability.md
+++ b/src/contribute/portability.md
@@ -87,7 +87,7 @@ design documents are the following:
* [Runner API](https://s.apache.org/beam-runner-api). Pipeline
representation and discussion on primitive/composite transforms and
optimizations.
-
+
* [Job API](https://s.apache.org/beam-job-api). Job submission and
management protocol.
@@ -134,12 +134,12 @@ portability effort. Work already done is not tracked here either.
windowed wordcount
[[BEAM-2941](https://issues.apache.org/jira/browse/BEAM-2941)] run
portably.
-
+
* **P2 [Feature complete]**: Design and implement portability support
for remaining execution-side features, so that any pipeline from
any SDK can run portably on any runner. These features include side
inputs
- [[BEAM-2863](https://issues.apache.org/jira/browse/BEAM-2863)], User
+ [[BEAM-2863](https://issues.apache.org/jira/browse/BEAM-2863)], User state [[BEAM-2862](https://issues.apache.org/jira/browse/BEAM-2862)], User
timers
[[BEAM-2925](https://issues.apache.org/jira/browse/BEAM-2925)],
Splittable DoFn
@@ -148,7 +148,7 @@ portability effort. Work already done is not tracked here either.
least to the extent that the mobile gaming examples
[[BEAM-2940](https://issues.apache.org/jira/browse/BEAM-2940)] run
portably.
-
+
* **P3 [Performance]**: Measure and tune performance of portable
pipelines using benchmarks such as Nexmark. Features such as
progress reporting
@@ -174,6 +174,18 @@ their respective components.
### Status
-MVP in progress. See the
+MVP complete (for Flink runner). See the
[Portability support table](https://docs.google.com/spreadsheets/d/1KDa_FGn1ShjomGd-UUDOhuh2q73de2tPz6BqHpzqvNI/edit?usp=sharing)
for details.
+
+#### Running Python wordcount on Flink
+
+The Flink runner is currently the only runner to support portable pipeline execution.
+To run a basic Python wordcount (in batch mode):
+
+1. Run once to build the SDK harness container: `./gradlew -p sdks/python/container
+docker`
+2. Start the Flink portable JobService endpoint: `./gradlew :beam-runners-flink_2.11-job-server:runShadow`
+3. Submit the wordcount pipeline to above endpoint: `./gradlew :beam-sdks-python:portableWordCount`
+
+Note: A subset of the functionality is also supported in streaming mode; use `--streaming` in the command line to enable it.