You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/07/23 14:41:13 UTC
[GitHub] [beam] aromanenko-dev opened a new pull request #12353: [BEAM-9857] Update Spark runner page to include Docker job server
aromanenko-dev opened a new pull request #12353:
URL: https://github.com/apache/beam/pull/12353
R: @ibzib
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #12353: [BEAM-9857] Update Spark runner page to include Docker job server
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #12353:
URL: https://github.com/apache/beam/pull/12353#discussion_r459982859
##########
File path: website/www/site/content/en/documentation/runners/spark.md
##########
@@ -181,12 +181,20 @@ pip install apache_beam
{{< /highlight >}}
{{< paragraph class="language-py" >}}
-As of now you will need a copy of Apache Beam's source code. You can
-download it on the [Downloads page](/get-started/downloads/). In the future there will be pre-built Docker images
-available.
+Starting from Beam 2.20.0, pre-built Spark Job Service Docker images are available at
+[Docker Hub](https://hub.docker.com/r/apache/beam_spark_job_server).
{{< /paragraph >}}
-{{< paragraph class="language-py" >}}1. Start the JobService endpoint: `./gradlew :runners:spark:job-server:runShadow`{{< /paragraph >}}
+{{< paragraph class="language-py" >}}
+For older Beam versions, you will need a copy of Apache Beam's source code. You can
+download it on the [Downloads page](/get-started/downloads/).
+{{< /paragraph >}}
+
+{{< paragraph class="language-py" >}}
+1. Start the JobService endpoint:
+ * with Docker (default): `docker run --net=host apache/beam_spark_job_server:latest`
Review comment:
Agree, fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #12353: [BEAM-9857] Update Spark runner page to include Docker job server
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #12353:
URL: https://github.com/apache/beam/pull/12353#discussion_r459621903
##########
File path: website/www/site/content/en/documentation/runners/spark.md
##########
@@ -181,12 +181,20 @@ pip install apache_beam
{{< /highlight >}}
{{< paragraph class="language-py" >}}
-As of now you will need a copy of Apache Beam's source code. You can
-download it on the [Downloads page](/get-started/downloads/). In the future there will be pre-built Docker images
-available.
+Starting from Beam 2.20.0, pre-built Spark Job Service Docker images are available at
+[Docker Hub](https://hub.docker.com/r/apache/beam_spark_job_server).
{{< /paragraph >}}
-{{< paragraph class="language-py" >}}1. Start the JobService endpoint: `./gradlew :runners:spark:job-server:runShadow`{{< /paragraph >}}
+{{< paragraph class="language-py" >}}
+For older Beam versions, you will need a copy of Apache Beam's source code. You can
+download it on the [Downloads page](/get-started/downloads/).
+{{< /paragraph >}}
+
+{{< paragraph class="language-py" >}}
+1. Start the JobService endpoint:
+ * with Docker (default): `docker run --net=host apache/beam_spark_job_server:latest`
Review comment:
It's not really a "default." I think "preferred" is more accurate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib merged pull request #12353: [BEAM-9857] Update Spark runner page to include Docker job server
Posted by GitBox <gi...@apache.org>.
ibzib merged pull request #12353:
URL: https://github.com/apache/beam/pull/12353
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org