You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by me...@apache.org on 2018/07/22 23:09:46 UTC
[beam-site] branch mergebot updated (3a39ed6 -> 0c94192)
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.
from 3a39ed6 This closes #507
add 98ecf88 Prepare repository for deployment.
new bd6b564 Add Flink instructions to portability contrib page.
new a790241 Add instructions for local Flink cluster.
new 0c94192 This closes #506
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
content/get-started/quickstart-py/index.html | 37 ++++++++++++++--------------
content/js/language-switch.js | 1 +
src/contribute/portability.md | 29 ++++++++++++++++++----
3 files changed, 44 insertions(+), 23 deletions(-)
[beam-site] 01/03: Add Flink instructions to portability contrib
page.
Posted by me...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit bd6b56430b834ef6fc0b90740766d076d5df4306
Author: Thomas Weise <th...@apache.org>
AuthorDate: Fri Jul 20 13:41:09 2018 -0700
Add Flink instructions to portability contrib page.
---
src/contribute/portability.md | 22 +++++++++++++++++-----
1 file changed, 17 insertions(+), 5 deletions(-)
diff --git a/src/contribute/portability.md b/src/contribute/portability.md
index cd17a92..cb16a66 100644
--- a/src/contribute/portability.md
+++ b/src/contribute/portability.md
@@ -87,7 +87,7 @@ design documents are the following:
* [Runner API](https://s.apache.org/beam-runner-api). Pipeline
representation and discussion on primitive/composite transforms and
optimizations.
-
+
* [Job API](https://s.apache.org/beam-job-api). Job submission and
management protocol.
@@ -134,12 +134,12 @@ portability effort. Work already done is not tracked here either.
windowed wordcount
[[BEAM-2941](https://issues.apache.org/jira/browse/BEAM-2941)] run
portably.
-
+
* **P2 [Feature complete]**: Design and implement portability support
for remaining execution-side features, so that any pipeline from
any SDK can run portably on any runner. These features include side
inputs
- [[BEAM-2863](https://issues.apache.org/jira/browse/BEAM-2863)], User
+ [[BEAM-2863](https://issues.apache.org/jira/browse/BEAM-2863)], User state [[BEAM-2862](https://issues.apache.org/jira/browse/BEAM-2862)], User
timers
[[BEAM-2925](https://issues.apache.org/jira/browse/BEAM-2925)],
Splittable DoFn
@@ -148,7 +148,7 @@ portability effort. Work already done is not tracked here either.
least to the extent that the mobile gaming examples
[[BEAM-2940](https://issues.apache.org/jira/browse/BEAM-2940)] run
portably.
-
+
* **P3 [Performance]**: Measure and tune performance of portable
pipelines using benchmarks such as Nexmark. Features such as
progress reporting
@@ -174,6 +174,18 @@ their respective components.
### Status
-MVP in progress. See the
+MVP complete (for Flink runner). See the
[Portability support table](https://docs.google.com/spreadsheets/d/1KDa_FGn1ShjomGd-UUDOhuh2q73de2tPz6BqHpzqvNI/edit?usp=sharing)
for details.
+
+#### Running Python wordcount on Flink
+
+The Flink runner is currently the only runner to support portable pipeline execution.
+To run a basic Python wordcount (in batch mode):
+
+1. Run once to build the SDK harness container: `./gradlew -p sdks/python/container
+docker`
+2. Start the Flink portable JobService endpoint: `./gradlew :beam-runners-flink_2.11-job-server:runShadow`
+3. Submit the wordcount pipeline to above endpoint: `./gradlew :beam-sdks-python:portableWordCount`
+
+Note: A subset of the functionality is also supported in streaming mode; use `--streaming` in the command line to enable it.
[beam-site] 02/03: Add instructions for local Flink cluster.
Posted by me...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit a790241c0f5b57ca4e6396b187cf51dfa4089dd0
Author: Thomas Weise <th...@apache.org>
AuthorDate: Sun Jul 22 15:43:22 2018 -0700
Add instructions for local Flink cluster.
---
src/contribute/portability.md | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/src/contribute/portability.md b/src/contribute/portability.md
index cb16a66..4e1df83 100644
--- a/src/contribute/portability.md
+++ b/src/contribute/portability.md
@@ -174,18 +174,25 @@ their respective components.
### Status
-MVP complete (for Flink runner). See the
+MVP in progress (near completion for Flink runner). See the
[Portability support table](https://docs.google.com/spreadsheets/d/1KDa_FGn1ShjomGd-UUDOhuh2q73de2tPz6BqHpzqvNI/edit?usp=sharing)
for details.
#### Running Python wordcount on Flink
The Flink runner is currently the only runner to support portable pipeline execution.
-To run a basic Python wordcount (in batch mode):
+To run a basic Python wordcount (in batch mode) with embedded Flink:
1. Run once to build the SDK harness container: `./gradlew -p sdks/python/container
docker`
2. Start the Flink portable JobService endpoint: `./gradlew :beam-runners-flink_2.11-job-server:runShadow`
3. Submit the wordcount pipeline to above endpoint: `./gradlew :beam-sdks-python:portableWordCount`
+To run on a separate [Flink cluster](https://ci.apache.org/projects/flink/flink-docs-release-1.5/quickstart/setup_quickstart.html):
+
+1. Start local Flink cluster
+2. Create shaded JobService jar: `./gradlew :beam-runners-flink_2.11-job-server:installShadowDist`
+3. Start JobService with Flink web service endpoint: `java -jar ./runners/flink/job-server/build/install/beam-runners-flink_2.11-job-server-shadow/lib/beam-runners-flink_2.11-job-server-*.jar--job-host=localhost:8099" "--artifacts-dir=/tmp/flink-artifacts" "--flink-master-url=localhost:8081"`
+4. Submit the pipeline.
+
Note: A subset of the functionality is also supported in streaming mode; use `--streaming` in the command line to enable it.
[beam-site] 03/03: This closes #506
Posted by me...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit 0c941920abd15f7a321ef904a05914afa4616433
Merge: 98ecf88 a790241
Author: Mergebot <me...@apache.org>
AuthorDate: Sun Jul 22 23:09:08 2018 +0000
This closes #506
src/contribute/portability.md | 29 ++++++++++++++++++++++++-----
1 file changed, 24 insertions(+), 5 deletions(-)