You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by me...@apache.org on 2018/07/22 23:09:47 UTC

[beam-site] 01/03: Add Flink instructions to portability contrib page.

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit bd6b56430b834ef6fc0b90740766d076d5df4306
Author: Thomas Weise <th...@apache.org>
AuthorDate: Fri Jul 20 13:41:09 2018 -0700

    Add Flink instructions to portability contrib page.
---
 src/contribute/portability.md | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/src/contribute/portability.md b/src/contribute/portability.md
index cd17a92..cb16a66 100644
--- a/src/contribute/portability.md
+++ b/src/contribute/portability.md
@@ -87,7 +87,7 @@ design documents are the following:
  * [Runner API](https://s.apache.org/beam-runner-api). Pipeline
    representation and discussion on primitive/composite transforms and
    optimizations.
-   
+
  * [Job API](https://s.apache.org/beam-job-api). Job submission and
    management protocol.
 
@@ -134,12 +134,12 @@ portability effort. Work already done is not tracked here either.
    windowed wordcount
    [[BEAM-2941](https://issues.apache.org/jira/browse/BEAM-2941)] run
    portably.
-    
+
  * **P2 [Feature complete]**: Design and implement portability support
    for remaining execution-side features, so that any pipeline from
    any SDK can run portably on any runner. These features include side
    inputs
-   [[BEAM-2863](https://issues.apache.org/jira/browse/BEAM-2863)], User
+   [[BEAM-2863](https://issues.apache.org/jira/browse/BEAM-2863)], User state [[BEAM-2862](https://issues.apache.org/jira/browse/BEAM-2862)], User
    timers
    [[BEAM-2925](https://issues.apache.org/jira/browse/BEAM-2925)],
    Splittable DoFn
@@ -148,7 +148,7 @@ portability effort. Work already done is not tracked here either.
    least to the extent that the mobile gaming examples
    [[BEAM-2940](https://issues.apache.org/jira/browse/BEAM-2940)] run
    portably.
-    
+
  * **P3 [Performance]**: Measure and tune performance of portable
    pipelines using benchmarks such as Nexmark. Features such as
    progress reporting
@@ -174,6 +174,18 @@ their respective components.
 
 ### Status
 
-MVP in progress. See the
+MVP complete (for Flink runner). See the
 [Portability support table](https://docs.google.com/spreadsheets/d/1KDa_FGn1ShjomGd-UUDOhuh2q73de2tPz6BqHpzqvNI/edit?usp=sharing)
 for details.
+
+#### Running Python wordcount on Flink
+
+The Flink runner is currently the only runner to support portable pipeline execution.
+To run a basic Python wordcount (in batch mode):
+
+1. Run once to build the SDK harness container: `./gradlew -p sdks/python/container
+docker`
+2. Start the Flink portable JobService endpoint: `./gradlew :beam-runners-flink_2.11-job-server:runShadow`
+3. Submit the wordcount pipeline to above endpoint: `./gradlew :beam-sdks-python:portableWordCount`
+
+Note: A subset of the functionality is also supported in streaming mode; use `--streaming` in the command line to enable it.