You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/07/29 07:51:02 UTC
[GitHub] [beam] ibzib opened a new pull request #12397: [BEAM-7632] Add Python quickstart instructions for Flink and Spark.
ibzib opened a new pull request #12397:
URL: https://github.com/apache/beam/pull/12397
I removed 'Flink-cluster' instructions because running Beam Python on a distributed Flink cluster is not yet really quick-startable, and I'd rather not mislead folks about that.
R: @aromanenko-dev
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev merged pull request #12397: [BEAM-7632] Add Python quickstart instructions for Flink and Spark.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev merged pull request #12397:
URL: https://github.com/apache/beam/pull/12397
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #12397: [BEAM-7632] Add Python quickstart instructions for Flink and Spark.
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #12397:
URL: https://github.com/apache/beam/pull/12397#discussion_r464552462
##########
File path: website/www/site/content/en/get-started/wordcount-example.md
##########
@@ -387,18 +387,20 @@ python -m apache_beam.examples.wordcount --input YOUR_INPUT_FILE --output counts
{{< /highlight >}}
{{< highlight class="runner-flink-local" >}}
-Currently, running wordcount.py on Flink requires a full download of the Beam source code.
-See https://beam.apache.org/roadmap/portability/#python-on-flink for more information.
+python -m apache_beam.examples.wordcount --input /path/to/inputfile \
+ --output /path/to/write/counts \
+ --runner FlinkRunner
{{< /highlight >}}
{{< highlight class="runner-flink-cluster" >}}
-Currently, running wordcount.py on Flink requires a full download of the Beam source code.
-See https://beam.apache.org/documentation/runners/flink/ for more information.
+# Running Beam Python on a distributed Flink cluster requires additional configuration.
Review comment:
I agree. I will change this once I can figure out what is going wrong with local cluster execution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #12397: [BEAM-7632] Add Python quickstart instructions for Flink and Spark.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #12397:
URL: https://github.com/apache/beam/pull/12397#discussion_r464504070
##########
File path: website/www/site/content/en/get-started/wordcount-example.md
##########
@@ -387,18 +387,20 @@ python -m apache_beam.examples.wordcount --input YOUR_INPUT_FILE --output counts
{{< /highlight >}}
{{< highlight class="runner-flink-local" >}}
-Currently, running wordcount.py on Flink requires a full download of the Beam source code.
-See https://beam.apache.org/roadmap/portability/#python-on-flink for more information.
+python -m apache_beam.examples.wordcount --input /path/to/inputfile \
+ --output /path/to/write/counts \
+ --runner FlinkRunner
{{< /highlight >}}
{{< highlight class="runner-flink-cluster" >}}
-Currently, running wordcount.py on Flink requires a full download of the Beam source code.
-See https://beam.apache.org/documentation/runners/flink/ for more information.
+# Running Beam Python on a distributed Flink cluster requires additional configuration.
Review comment:
Would it be more simple and clear for user to keep only one tab for every engine (e.g. just "Flink") and explain all additional instructions, like `local` and `cluster` mode, inside this tab?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #12397: [BEAM-7632] Add Python quickstart instructions for Flink and Spark.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #12397:
URL: https://github.com/apache/beam/pull/12397#discussion_r464504070
##########
File path: website/www/site/content/en/get-started/wordcount-example.md
##########
@@ -387,18 +387,20 @@ python -m apache_beam.examples.wordcount --input YOUR_INPUT_FILE --output counts
{{< /highlight >}}
{{< highlight class="runner-flink-local" >}}
-Currently, running wordcount.py on Flink requires a full download of the Beam source code.
-See https://beam.apache.org/roadmap/portability/#python-on-flink for more information.
+python -m apache_beam.examples.wordcount --input /path/to/inputfile \
+ --output /path/to/write/counts \
+ --runner FlinkRunner
{{< /highlight >}}
{{< highlight class="runner-flink-cluster" >}}
-Currently, running wordcount.py on Flink requires a full download of the Beam source code.
-See https://beam.apache.org/documentation/runners/flink/ for more information.
+# Running Beam Python on a distributed Flink cluster requires additional configuration.
Review comment:
Would it be more simple and clear for users to keep the only one tab for every engine (e.g. just "Flink") and explain all additional instructions, like `local` and `cluster` mode, inside this tab?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib edited a comment on pull request #12397: [BEAM-7632] Add Python quickstart instructions for Flink and Spark.
Posted by GitBox <gi...@apache.org>.
ibzib edited a comment on pull request #12397:
URL: https://github.com/apache/beam/pull/12397#issuecomment-665351269
> For an existing cluster, isn't it as simple as passing the flink master URL? If so, I think that's worth stating.
For a local cluster, yes (we can just use loopback). Or at least I thought so. I tried this on my local Flink 1.10 cluster:
```sh
python -m apache_beam.examples.wordcount --input /path/to/inputfile \
--output /path/to/write/counts \
--runner FlinkRunner \
--environment_type LOOPBACK \
--flink_master localhost:8081 \
--flink_version 1.10
```
But I got an error:
`ERROR:root:org.apache.flink.runtime.rest.util.RestClientException: [Failed to deserialize JobGraph.]`
This [Stack Overflow post](https://stackoverflow.com/questions/58358308/flink-submitting-job-failed-to-deserialize-jobgraph) suggests a Flink version incompatibility might be the cause, but both the job server jar and the Flink cluster are Flink 1.10, so I'm not sure what's going on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #12397: [BEAM-7632] Add Python quickstart instructions for Flink and Spark.
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #12397:
URL: https://github.com/apache/beam/pull/12397#issuecomment-665351269
> For an existing cluster, isn't it as simple as passing the flink master URL? If so, I think that's worth stating.
For a local cluster, yes (we can just use loopback). Or at least I thought so. I tried this on my local Flink 1.10 cluster:
```sh
python -m apache_beam.examples.wordcount --input /path/to/inputfile \
--output /path/to/write/counts \
--runner FlinkRunner \
--environment_type LOOPBACK \
--flink_master localhost:8081 \
--flink_version 1.10
```
But I got an error:
`ERROR:root:org.apache.flink.runtime.rest.util.RestClientException: [Failed to deserialize JobGraph.]`
This [Stack Overflow post](https://stackoverflow.com/questions/58358308/flink-submitting-job-failed-to-deserialize-jobgraph) a Flink version incompatibility might be the cause, but both the job server jar and the Flink cluster are Flink 1.10, so I'm not sure what's going on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org