You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/06/09 16:02:13 UTC
[GitHub] [beam] robertwb opened a new pull request #11949: Simplify Python on Flink runner instructions.
robertwb opened a new pull request #11949:
URL: https://github.com/apache/beam/pull/11949
Also fix formatting broken during website transition due to BEAM-10214.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #11949: Simplify Python on Flink runner instructions.
Posted by GitBox <gi...@apache.org>.
robertwb commented on pull request #11949:
URL: https://github.com/apache/beam/pull/11949#issuecomment-640891635
Reading it again, I simply separated out the Python vs. Portable instructions completely for readability. PTAL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a change in pull request #11949: Simplify Python on Flink runner instructions.
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #11949:
URL: https://github.com/apache/beam/pull/11949#discussion_r437006323
##########
File path: website/www/site/content/en/documentation/runners/flink.md
##########
@@ -276,36 +276,48 @@ If you have a Flink `JobManager` running on your local machine you can provide `
`flinkMaster`. Otherwise an embedded Flink cluster will be started for the job.
{{< /paragraph >}}
-Starting with Beam 2.18.0, pre-built Docker images are available at Docker Hub.
-
-JobService:
+{{< paragraph class="language-portable" >}}
+Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub:
[Flink 1.8](https://hub.docker.com/r/apache/beam_flink1.8_job_server),
[Flink 1.9](https://hub.docker.com/r/apache/beam_flink1.9_job_server).
[Flink 1.10](https://hub.docker.com/r/apache/beam_flink1.10_job_server).
-
-Beam SDK:
-[Python 2.7](https://hub.docker.com/r/apache/beam_python2.7_sdk),
-[Python 3.5](https://hub.docker.com/r/apache/beam_python3.5_sdk),
-[Python 3.6](https://hub.docker.com/r/apache/beam_python3.6_sdk),
-[Python 3.7](https://hub.docker.com/r/apache/beam_python3.7_sdk).
+{{< /paragraph >}}
To run a pipeline on an embedded Flink cluster:
<!-- Span implictly ended -->
{{< paragraph class="language-py" >}}
-1. Start the JobService endpoint: `docker run --net=host apache/beam_flink1.10_job_server:latest`
+(1) Set the runner to `FlinkRunner`. Optionally set `environment_type` set to `LOOPBACK`. For example:
{{< /paragraph >}}
-{{< paragraph class="language-py" >}}
+{{< highlight py >}}
+import apache_beam as beam
+from apache_beam.options.pipeline_options import PipelineOptions
+
+options = PipelineOptions([
+ "--runner=FlinkRunner",
+ "--environment_type=LOOPBACK"
+])
+with beam.Pipeline(options) as p:
+ ...
+{{< /highlight >}}
+
+{{< paragraph class="language-portable" >}}
+<!-- TODO(BEAM-10214): Use actual lists here and below. -->
+(1) Start the JobService endpoint: `docker run --net=host apache/beam_flink1.10_job_server:latest`
Review comment:
Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #11949: Simplify Python on Flink runner instructions.
Posted by GitBox <gi...@apache.org>.
robertwb commented on pull request #11949:
URL: https://github.com/apache/beam/pull/11949#issuecomment-642170460
Clarified the tabs selection. (Do you think it's worth moving the version compatibility table lower in case people want to see what changes when they toggle the tab? It's kind of out of place here anyway.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #11949: Simplify Python on Flink runner instructions.
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #11949:
URL: https://github.com/apache/beam/pull/11949#issuecomment-642180751
> Do you think it's worth moving the version compatibility table lower in case people want to see what changes when they toggle the tab? It's kind of out of place here anyway.
+1, the version compatibility table is less important than the how-to instructions and should be moved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #11949: Simplify Python on Flink runner instructions.
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #11949:
URL: https://github.com/apache/beam/pull/11949#discussion_r436981715
##########
File path: website/www/site/content/en/documentation/runners/flink.md
##########
@@ -276,36 +276,48 @@ If you have a Flink `JobManager` running on your local machine you can provide `
`flinkMaster`. Otherwise an embedded Flink cluster will be started for the job.
{{< /paragraph >}}
-Starting with Beam 2.18.0, pre-built Docker images are available at Docker Hub.
-
-JobService:
+{{< paragraph class="language-portable" >}}
+Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub:
[Flink 1.8](https://hub.docker.com/r/apache/beam_flink1.8_job_server),
[Flink 1.9](https://hub.docker.com/r/apache/beam_flink1.9_job_server).
[Flink 1.10](https://hub.docker.com/r/apache/beam_flink1.10_job_server).
-
-Beam SDK:
-[Python 2.7](https://hub.docker.com/r/apache/beam_python2.7_sdk),
-[Python 3.5](https://hub.docker.com/r/apache/beam_python3.5_sdk),
-[Python 3.6](https://hub.docker.com/r/apache/beam_python3.6_sdk),
-[Python 3.7](https://hub.docker.com/r/apache/beam_python3.7_sdk).
+{{< /paragraph >}}
To run a pipeline on an embedded Flink cluster:
<!-- Span implictly ended -->
{{< paragraph class="language-py" >}}
-1. Start the JobService endpoint: `docker run --net=host apache/beam_flink1.10_job_server:latest`
+(1) Set the runner to `FlinkRunner`. Optionally set `environment_type` set to `LOOPBACK`. For example:
{{< /paragraph >}}
-{{< paragraph class="language-py" >}}
+{{< highlight py >}}
+import apache_beam as beam
+from apache_beam.options.pipeline_options import PipelineOptions
+
+options = PipelineOptions([
+ "--runner=FlinkRunner",
+ "--environment_type=LOOPBACK"
+])
+with beam.Pipeline(options) as p:
+ ...
+{{< /highlight >}}
+
+{{< paragraph class="language-portable" >}}
+<!-- TODO(BEAM-10214): Use actual lists here and below. -->
+(1) Start the JobService endpoint: `docker run --net=host apache/beam_flink1.10_job_server:latest`
Review comment:
This instruction appears when Python is toggled. I don't think it should.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #11949: Simplify Python on Flink runner instructions.
Posted by GitBox <gi...@apache.org>.
robertwb commented on pull request #11949:
URL: https://github.com/apache/beam/pull/11949#issuecomment-642188187
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb merged pull request #11949: Simplify Python on Flink runner instructions.
Posted by GitBox <gi...@apache.org>.
robertwb merged pull request #11949:
URL: https://github.com/apache/beam/pull/11949
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org