You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/10/19 05:52:06 UTC

[GitHub] [beam] chamikaramj opened a new pull request, #23721: Adds instructions for running the Multi-language Java quickstart from released Beam

chamikaramj opened a new pull request, #23721:
URL: https://github.com/apache/beam/pull/23721

   Also, adds a noticed due to an ongoing issue for Beam 2.42.0.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj merged pull request #23721: Adds instructions for running the Multi-language Java quickstart from released Beam

Posted by GitBox <gi...@apache.org>.
chamikaramj merged PR #23721:
URL: https://github.com/apache/beam/pull/23721


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #23721: Adds instructions for running the Multi-language Java quickstart from released Beam

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #23721:
URL: https://github.com/apache/beam/pull/23721#issuecomment-1284369862

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #23721: Adds instructions for running the Multi-language Java quickstart from released Beam

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #23721:
URL: https://github.com/apache/beam/pull/23721#issuecomment-1285930518

   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #23721: Adds instructions for running the Multi-language Java quickstart from released Beam

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23721:
URL: https://github.com/apache/beam/pull/23721#issuecomment-1283516757

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #23721: Adds instructions for running the Multi-language Java quickstart from released Beam

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #23721:
URL: https://github.com/apache/beam/pull/23721#issuecomment-1284274868

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pcoet commented on a diff in pull request #23721: Adds instructions for running the Multi-language Java quickstart from released Beam

Posted by GitBox <gi...@apache.org>.
pcoet commented on code in PR #23721:
URL: https://github.com/apache/beam/pull/23721#discussion_r1000905875


##########
website/www/site/content/en/documentation/sdks/java-multi-language-pipelines.md:
##########
@@ -138,26 +138,27 @@ default Beam SDK, you might need to run your own expansion service. In such
 cases, [start the expansion service](#advanced-start-an-expansion-service)
 before running your pipeline.
 
-Here we've provided commands for running the example pipeline using
-Gradle on a [Beam HEAD Git clone](https://github.com/apache/beam).
-If you need a more stable environment, please
-[setup a Java project](/get-started/quickstart-java/) that uses the latest
-released Beam version and include the necessary dependencies.
+### Run with Dataflow runner at HEAD (Beam 2.41.0 and later)
 
-### Run with Dataflow runner
+> **Note:** Due to [issue#23717](https://github.com/apache/beam/issues/23717)
+> Beam 2.42.0 requires manually starting up an expansion service according to instructions

Review Comment:
   Try this:
   
   ```
   > **Note:** Due to [issue#23717](https://github.com/apache/beam/issues/23717),
   > Beam 2.42.0 requires manually starting up an expansion service (see [these instructions]
   > (https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/#advanced-start-an-expansion-service))
   > and using the additional pipeline option `--expansionService=localhost:<PORT>`
   > when executing the pipeline.
   ```



##########
website/www/site/content/en/documentation/sdks/java-multi-language-pipelines.md:
##########
@@ -192,10 +193,15 @@ python -m apache_beam.runners.portability.local_job_service_main -p $JOB_SERVER_
 
 5. Run the pipeline.
 
+> **Note:** Due to [issue#23717](https://github.com/apache/beam/issues/23717)
+> Beam 2.42.0 requires manually starting up an expansion service according to instructions

Review Comment:
   See previous suggestion.



##########
examples/multi-language/README.md:
##########
@@ -166,3 +166,12 @@ of the digit. The second item is the predicted label of the digit.
 ```
 gsutil cat gs://$GCP_BUCKET/multi-language-beam/output*
 ```
+
+### Python Dataframe Wordcount
+
+This is covered by the quick-start guide. Please see

Review Comment:
   Maybe just: "This example is covered in the [Java multi-language pipelines quickstart](https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/). The pipeline source code is available at [PythonDataframeWordCount.java](https://github.com/apache/beam/tree/master/examples/java/src/main/java/org/apache/beam/examples/multilanguage/PythonDataframeWordCount.java)."



##########
website/www/site/content/en/documentation/sdks/java-multi-language-pipelines.md:
##########
@@ -192,10 +193,15 @@ python -m apache_beam.runners.portability.local_job_service_main -p $JOB_SERVER_
 
 5. Run the pipeline.
 
+> **Note:** Due to [issue#23717](https://github.com/apache/beam/issues/23717)

Review Comment:
   See previous suggestion.



##########
website/www/site/content/en/documentation/sdks/java-multi-language-pipelines.md:
##########
@@ -226,19 +232,70 @@ For example, to start the standard expansion service for a Python transform,
 [ExpansionServiceServicer](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/expansion_service.py),
 follow these steps:
 
-1. Activate a Python virtual environment and install Apache Beam, as described
-   in the [Python quick start](/get-started/quickstart-py/).
-2. In the **beam/sdks/python** directory of the Beam source code, run the
-   following command:
+1. Activate a new virtual environment. See [here](https://beam.apache.org/get-started/quickstart-py/#create-and-activate-a-virtual-environment)
+for additional instructions.
+
+```
+mkdir venv_dir
+python -m venv venv_dir
+. venv_dir/bin/activate
+```
+
+2. Install Apache Beam with `gcp` and `dataframe` packages.
+
+```
+pip install apache-beam[gcp,dataframe]
+```
+
+4. Run the following command
 
    ```
-   python apache_beam/runners/portability/expansion_service_main.py -p 18089 --fully_qualified_name_glob "*"
+   python -m apache_beam.runners.portability.expansion_service_main -p <PORT> --fully_qualified_name_glob "*"
    ```
 
 The command runs
 [expansion_service_main.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/expansion_service_main.py), which starts the standard expansion service. When you use
 Gradle to run your Java pipeline, you can specify the expansion service with the
-`expansionService` option. For example: `--expansionService=localhost:18089`.
+`expansionService` option. For example: `--expansionService=localhost:<PORT>`.
+
+### Run with Dataflow runner using a Beam release (Beam 2.43.0 and later)
+
+> **Note:** Due to [issue#23717](https://github.com/apache/beam/issues/23717)

Review Comment:
   See previous suggestion.



##########
website/www/site/content/en/documentation/sdks/java-multi-language-pipelines.md:
##########
@@ -226,19 +232,70 @@ For example, to start the standard expansion service for a Python transform,
 [ExpansionServiceServicer](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/expansion_service.py),
 follow these steps:
 
-1. Activate a Python virtual environment and install Apache Beam, as described
-   in the [Python quick start](/get-started/quickstart-py/).
-2. In the **beam/sdks/python** directory of the Beam source code, run the
-   following command:
+1. Activate a new virtual environment. See [here](https://beam.apache.org/get-started/quickstart-py/#create-and-activate-a-virtual-environment)
+for additional instructions.
+
+```
+mkdir venv_dir
+python -m venv venv_dir
+. venv_dir/bin/activate
+```
+
+2. Install Apache Beam with `gcp` and `dataframe` packages.
+
+```
+pip install apache-beam[gcp,dataframe]
+```
+
+4. Run the following command
 
    ```
-   python apache_beam/runners/portability/expansion_service_main.py -p 18089 --fully_qualified_name_glob "*"
+   python -m apache_beam.runners.portability.expansion_service_main -p <PORT> --fully_qualified_name_glob "*"
    ```
 
 The command runs
 [expansion_service_main.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/expansion_service_main.py), which starts the standard expansion service. When you use
 Gradle to run your Java pipeline, you can specify the expansion service with the
-`expansionService` option. For example: `--expansionService=localhost:18089`.
+`expansionService` option. For example: `--expansionService=localhost:<PORT>`.
+
+### Run with Dataflow runner using a Beam release (Beam 2.43.0 and later)
+
+> **Note:** Due to [issue#23717](https://github.com/apache/beam/issues/23717)
+> Beam 2.42.0 requires manually starting up an expansion service according to instructions
+> [here](https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/#advanced-start-an-expansion-service)
+> and using the additional pipeline option `--expansionService=localhost:<PORT>`
+> when executing the pipeline.
+
+* Checkout the Beam examples Maven archetype for the relevant Beam version.

Review Comment:
   "Checkout" -> "Check out"



##########
website/www/site/content/en/documentation/sdks/java-multi-language-pipelines.md:
##########
@@ -226,19 +232,70 @@ For example, to start the standard expansion service for a Python transform,
 [ExpansionServiceServicer](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/expansion_service.py),
 follow these steps:
 
-1. Activate a Python virtual environment and install Apache Beam, as described
-   in the [Python quick start](/get-started/quickstart-py/).
-2. In the **beam/sdks/python** directory of the Beam source code, run the
-   following command:
+1. Activate a new virtual environment. See [here](https://beam.apache.org/get-started/quickstart-py/#create-and-activate-a-virtual-environment)

Review Comment:
   Maybe use this: "1. Activate a new virtual environment following [these instructions](https://beam.apache.org/get-started/quickstart-py/#create-and-activate-a-virtual-environment)."
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #23721: Adds instructions for running the Multi-language Java quickstart from released Beam

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #23721:
URL: https://github.com/apache/beam/pull/23721#issuecomment-1286241867

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #23721: Adds instructions for running the Multi-language Java quickstart from released Beam

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #23721:
URL: https://github.com/apache/beam/pull/23721#issuecomment-1283467038

   R: @pcoet


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #23721: Adds instructions for running the Multi-language Java quickstart from released Beam

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #23721:
URL: https://github.com/apache/beam/pull/23721#issuecomment-1283468469

   This fixes #23722.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #23721: Adds instructions for running the Multi-language Java quickstart from released Beam

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #23721:
URL: https://github.com/apache/beam/pull/23721#issuecomment-1284369607

   Precommit failure is unrelated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #23721: Adds instructions for running the Multi-language Java quickstart from released Beam

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #23721:
URL: https://github.com/apache/beam/pull/23721#issuecomment-1286242095

   Run Java_Examples_Dataflow_Java11 PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org