You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/03/07 19:56:40 UTC
[GitHub] [beam] pcoet opened a new pull request #17033: BEAM-12770: Documented maven-to-gradle conversion for Dataflow
pcoet opened a new pull request #17033:
URL: https://github.com/apache/beam/pull/17033
* Document Maven-to-Gradle conversion for DataflowRunner.
* Create TODOs for other runner conversion steps.
* Change Gradle guidance to show Kotlin instead of Groovy.
* Tighten up writing and remove unnecessary stuff (too much wordz).
* Make numbering of procedures consistent.
* Remove command prompts (`$`, `PS>`) from commands, to make them easier to copy.
* Normalize line length in markdown.
* Simplify Gradle execute task so that it reads from `--args` instead of `-D exec.args`
* Update sample text creation step to use text of "King Lear", which is the same as the default for DirectRunner.
I think this is a lot more streamlined, but let me know... Thanks!
@lukecwik
@TheNeuralBit
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pcoet commented on pull request #17033: BEAM-12770: Documented maven-to-gradle conversion for Dataflow
Posted by GitBox <gi...@apache.org>.
pcoet commented on pull request #17033:
URL: https://github.com/apache/beam/pull/17033#issuecomment-1067528339
@kileys I made changes, in response to your feedback:
- Fixed the inconsistent filename formatting
- Reworked "Run a pipeline" to emphasize the runner setup steps. Any additional runner-specific instructions should go in the runner sections, instead of in the runnable commands. I think this is better now.
- Responded to your other comments inline.
Thanks!
David
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pcoet commented on a change in pull request #17033: BEAM-12770: Documented maven-to-gradle conversion for Dataflow
Posted by GitBox <gi...@apache.org>.
pcoet commented on a change in pull request #17033:
URL: https://github.com/apache/beam/pull/17033#discussion_r826490392
##########
File path: website/www/site/content/en/get-started/quickstart-java.md
##########
@@ -19,462 +19,376 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache Beam Java SDK Quickstart
+# Apache Beam Java SDK quickstart
-This quickstart shows you how to set up a Java development environment and run an [example pipeline](/get-started/wordcount-example) written with the [Apache Beam Java SDK](/documentation/sdks/java), using a [runner](/documentation#runners) of your choice.
+This quickstart shows you how to set up a Java development environment and run
+an [example pipeline](/get-started/wordcount-example) written with the
+[Apache Beam Java SDK](/documentation/sdks/java), using a
+[runner](/documentation#runners) of your choice.
-If you're interested in contributing to the Apache Beam Java codebase, see the [Contribution Guide](/contribute).
+If you're interested in contributing to the Apache Beam Java codebase, see the
+[Contribution Guide](/contribute).
-{{< toc >}}
+On this page:
-## Set up your Development Environment
-
-1. Download and install the [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html) version 8. Verify that the [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html) environment variable is set and points to your JDK installation.
-
-1. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by following Maven's [installation guide](https://maven.apache.org/install.html) for your specific operating system.
-
-1. Optional: Install [Gradle](https://gradle.org/install/) if you would like to convert your Maven project into Gradle.
-
-## Get the Example Code
-
-Use the following command to generate a Maven project that contains Beam's WordCount examples and builds against the most recent Beam release:
-
-{{< shell unix >}}
-$ mvn archetype:generate \
- -DarchetypeGroupId=org.apache.beam \
- -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
- -DarchetypeVersion={{< param release_latest >}} \
- -DgroupId=org.example \
- -DartifactId=word-count-beam \
- -Dversion="0.1" \
- -Dpackage=org.apache.beam.examples \
- -DinteractiveMode=false
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> mvn archetype:generate `
- -D archetypeGroupId=org.apache.beam `
- -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
- -D archetypeVersion={{< param release_latest >}} `
- -D groupId=org.example `
- -D artifactId=word-count-beam `
- -D version="0.1" `
- -D package=org.apache.beam.examples `
- -D interactiveMode=false
-{{< /shell >}}
-
-This will create a `word-count-beam` directory that contains a `pom.xml` and several example pipelines that count words in text files.
-
-{{< shell unix >}}
-$ cd word-count-beam/
-
-$ ls
-pom.xml src
-
-$ ls src/main/java/org/apache/beam/examples/
-DebuggingWordCount.java WindowedWordCount.java common
-MinimalWordCount.java WordCount.java
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> cd .\word-count-beam
-
-PS> dir
-
-...
-
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM src
--a---- 7/19/2018 11:00 PM 16051 pom.xml
-
-PS> dir .\src\main\java\org\apache\beam\examples
-
-...
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM common
-d----- 7/19/2018 11:00 PM complete
-d----- 7/19/2018 11:00 PM subprocess
--a---- 7/19/2018 11:00 PM 7073 DebuggingWordCount.java
--a---- 7/19/2018 11:00 PM 5945 MinimalWordCount.java
--a---- 7/19/2018 11:00 PM 9490 WindowedWordCount.java
--a---- 7/19/2018 11:00 PM 7662 WordCount.java
-{{< /shell >}}
-
-For a detailed introduction to the Beam concepts used in these examples, see the [WordCount Example Walkthrough](/get-started/wordcount-example). Here, we'll just focus on executing `WordCount.java`.
-
-## Optional: Convert from Maven to Gradle Project
-
-The steps below explain how to convert the build for the Direct Runner from Maven to Gradle. Converting the builds for the other runners is a more involved process and is out of scope for this guide. For additional guidance, see [Migrating Builds From Apache Maven](https://docs.gradle.org/current/userguide/migrating_from_maven.html).
-
-1. Ensure you are in the same directory as the `pom.xml` file generated from the previous step. Automatically convert your project from Maven to Gradle by running:
-{{< highlight >}}
-$ gradle init
-{{< /highlight >}}
-You'll be asked if you want to generate a Gradle build. Enter **yes**. You'll also be prompted to choose a DSL (Groovy or Kotlin). This tutorial uses Groovy, so select that if you don't have a preference.
-1. After you've converted the project to Gradle, open the generated `build.gradle` file, and, in the `repositories` block, replace `mavenLocal()` with `mavenCentral()`:
-{{< highlight >}}
-repositories {
- mavenCentral()
- maven {
- url = uri('https://repository.apache.org/content/repositories/snapshots/')
- }
+{{< toc >}}
- maven {
- url = uri('http://repo.maven.apache.org/maven2')
+## Set up your development environment
+
+1. Download and install the
+ [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html)
+ version 8, 11, or 17. Verify that the
+ [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html)
+ environment variable is set and points to your JDK installation.
+2. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by
+ following the [installation guide](https://maven.apache.org/install.html)
+ for your operating system.
+3. Optional: If you want to convert your Maven project to Gradle, install
+ [Gradle](https://gradle.org/install/).
+
+## Get the example code
+
+1. Generate a Maven example project that builds against the latest Beam release:
+ {{< shell unix >}}
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.beam \
+ -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
+ -DarchetypeVersion={{< param release_latest >}} \
+ -DgroupId=org.example \
+ -DartifactId=word-count-beam \
+ -Dversion="0.1" \
+ -Dpackage=org.apache.beam.examples \
+ -DinteractiveMode=false
+ {{< /shell >}}
+ {{< shell powerShell >}}
+mvn archetype:generate `
+ -D archetypeGroupId=org.apache.beam `
+ -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
+ -D archetypeVersion={{< param release_latest >}} `
+ -D groupId=org.example `
+ -D artifactId=word-count-beam `
+ -D version="0.1" `
+ -D package=org.apache.beam.examples `
+ -D interactiveMode=false
+ {{< /shell >}}
+
+ Maven creates a new project in the **word-count-beam** directory.
+
+2. Change into **word-count-beam**:
+ {{< shell unix >}}
+cd word-count-beam/
+ {{< /shell >}}
+ {{< shell powerShell >}}
+cd .\word-count-beam
+ {{< /shell >}}
+ The directory contains a **pom.xml** and a **src** directory with example
+ pipelines.
+
+3. List the example pipelines:
+ {{< shell unix >}}
+ls src/main/java/org/apache/beam/examples/
+ {{< /shell >}}
+ {{< shell powerShell >}}
+dir .\src\main\java\org\apache\beam\examples
+ {{< /shell >}}
+ You should see the following examples:
+ * **DebuggingWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java))
+ * **MinimalWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java))
+ * **WindowedWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java))
+ * **WordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WordCount.java))
+
+ The example used in this tutorial, **WordCount.java**, defines a
+ Beam pipeline that counts words from an input file (by default, a **.txt**
+ file containing Shakespeare's "King Lear"). To learn more about the examples,
+ see the [WordCount Example Walkthrough](/get-started/wordcount-example).
+
+## Optional: Convert from Maven to Gradle
+
+The steps below explain how to convert the build from Maven to Gradle for the
+following runners:
+* Direct runner
+* Dataflow runner
+
+The conversion process for other runners is similar. For additional guidance,
+see
+[Migrating Builds From Apache Maven](https://docs.gradle.org/current/userguide/migrating_from_maven.html).
+
+1. In the directory with the `pom.xml` file, run the automated Maven-to-Gradle
+ conversion:
+ {{< highlight >}}
+gradle init
+ {{< /highlight >}}
+ You'll be asked if you want to generate a Gradle build. Enter **yes**. You'll
+ also be prompted to choose a DSL (Groovy or Kotlin). For this tutorial, enter
+ **2** for Kotlin.
+2. Open the generated **build.gradle.kts** file and make the following changes:
+ 1. In `repositories`, replace `mavenLocal()` with `mavenCentral()`.
+ 2. In `repositories`, declare a repository for Confluent Kafka dependencies:
+ {{< highlight >}}
+maven {
+ url = uri("https://packages.confluent.io/maven/")
+}
+ {{< /highlight >}}
+ 3. At the end of the build script, add the following conditional dependency:
+ {{< highlight >}}
+if (project.hasProperty("dataflow-runner")) {
+ dependencies {
+ runtimeOnly("org.apache.beam:beam-runners-google-cloud-dataflow-java:{{< param release_latest >}}")
}
}
-{{< /highlight >}}
-1. Add the following task in `build.gradle` to allow you to execute pipelines with Gradle:
-{{< highlight >}}
-task execute (type:JavaExec) {
- mainClass = System.getProperty("mainClass")
- classpath = sourceSets.main.runtimeClasspath
- systemProperties System.getProperties()
- args System.getProperty("exec.args", "").split()
+ {{< /highlight >}}
+ 4. At the end of the build script, add the following task:
+ {{< highlight >}}
+task("execute", JavaExec::class) {
+ classpath = sourceSets["main"].runtimeClasspath
+ mainClass.set(System.getProperty("mainClass"))
}
-{{< /highlight >}}
-1. Rebuild your project by running:
-{{< highlight >}}
-$ gradle build
-{{< /highlight >}}
+ {{< /highlight >}}
+4. Build your project:
+ {{< highlight >}}
+gradle build
+ {{< /highlight >}}
## Get sample text
-> If you're planning to use the DataflowRunner, you can skip this step. The runner will pull text directly from Google Cloud Storage.
+> If you're planning to use the DataflowRunner, you can skip this step. The
+ runner will pull text directly from Google Cloud Storage.
1. In the **word-count-beam** directory, create a file called **sample.txt**.
-1. Add some text to the file. For this example, you can use the text of Shakespeare's [Sonnets](https://storage.cloud.google.com/apache-beam-samples/shakespeare/sonnets.txt).
+2. Add some text to the file. For this example, use the text of Shakespeare's
+ [King Lear](https://storage.cloud.google.com/apache-beam-samples/shakespeare/kinglear.txt).
## Run a pipeline
-A single Beam pipeline can run on multiple Beam [runners](/documentation#runners), including the [FlinkRunner](/documentation/runners/flink), [SparkRunner](/documentation/runners/spark), [NemoRunner](/documentation/runners/nemo), [JetRunner](/documentation/runners/jet), or [DataflowRunner](/documentation/runners/dataflow). The [DirectRunner](/documentation/runners/direct) is a common runner for getting started, as it runs locally on your machine and requires no specific setup. If you're just trying out Beam and you're not sure what to use, use the [DirectRunner](/documentation/runners/direct).
+A single Beam pipeline can run on multiple Beam
+[runners](/documentation#runners), including the
+[FlinkRunner](/documentation/runners/flink),
+[SparkRunner](/documentation/runners/spark),
+[NemoRunner](/documentation/runners/nemo),
+[JetRunner](/documentation/runners/jet), and
+[DataflowRunner](/documentation/runners/dataflow). The
+[DirectRunner](/documentation/runners/direct) is useful for getting started,
+because it runs on your machine and requires no specific setup. If you're just
+trying out Beam and you're not sure what to use, use the
+[DirectRunner](/documentation/runners/direct).
The general process for running a pipeline goes like this:
-1. Ensure you've done any runner-specific setup.
-1. Build your command line:
- 1. Specify a runner with `--runner=<runner>` (defaults to the [DirectRunner](/documentation/runners/direct)).
- 1. Add any runner-specific required options.
- 1. Choose input files and an output location that are accessible to the runner. (For example, you can't access a local file if you are running the pipeline on an external cluster.)
-1. Run the command.
+1. Complete any runner-specific setup.
+2. Build your command line:
+ 1. Specify a runner with `--runner=<runner>` (defaults to the
+ [DirectRunner](/documentation/runners/direct)).
+ 2. Add any runner-specific required options.
+ 3. Choose input files and an output location that are accessible to the
+ runner. (For example, you can't access a local file if you are running
+ the pipeline on an external cluster.)
+3. Run the command.
To run the WordCount pipeline, see the Maven and Gradle examples below.
-### Run WordCount Using Maven
+### Run WordCount using Maven
For Unix shells:
{{< runner direct >}}
-$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
- -Dexec.args="--inputFile=sample.txt --output=counts" -Pdirect-runner
+mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="--inputFile=sample.txt --output=counts" -Pdirect-runner
{{< /runner >}}
-
{{< runner flink >}}
-$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
- -Dexec.args="--runner=FlinkRunner --inputFile=sample.txt --output=counts" -Pflink-runner
+mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="--runner=FlinkRunner --inputFile=sample.txt --output=counts" -Pflink-runner
{{< /runner >}}
-
{{< runner flinkCluster >}}
-$ mvn package exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
- -Dexec.args="--runner=FlinkRunner --flinkMaster=<flink master> --filesToStage=target/word-count-beam-bundled-0.1.jar \
- --inputFile=sample.txt --output=/tmp/counts" -Pflink-runner
-
-You can monitor the running job by visiting the Flink dashboard at http://<flink master>:8081
+mvn package exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="--runner=FlinkRunner --flinkMaster=<flink master> --filesToStage=target/word-count-beam-bundled-0.1.jar \
+ --inputFile=sample.txt --output=/tmp/counts" -Pflink-runner
{{< /runner >}}
-
{{< runner spark >}}
-$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
- -Dexec.args="--runner=SparkRunner --inputFile=sample.txt --output=counts" -Pspark-runner
+mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="--runner=SparkRunner --inputFile=sample.txt --output=counts" -Pspark-runner
{{< /runner >}}
-
{{< runner dataflow >}}
-Make sure you complete the setup steps at /documentation/runners/dataflow/#setup
Review comment:
I added an extra comment emphasizing that you have to follow the setup steps for your runner. Hopefully that will do it...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kileys commented on a change in pull request #17033: BEAM-12770: Documented maven-to-gradle conversion for Dataflow
Posted by GitBox <gi...@apache.org>.
kileys commented on a change in pull request #17033:
URL: https://github.com/apache/beam/pull/17033#discussion_r826463885
##########
File path: website/www/site/content/en/get-started/quickstart-java.md
##########
@@ -19,462 +19,376 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache Beam Java SDK Quickstart
+# Apache Beam Java SDK quickstart
-This quickstart shows you how to set up a Java development environment and run an [example pipeline](/get-started/wordcount-example) written with the [Apache Beam Java SDK](/documentation/sdks/java), using a [runner](/documentation#runners) of your choice.
+This quickstart shows you how to set up a Java development environment and run
+an [example pipeline](/get-started/wordcount-example) written with the
+[Apache Beam Java SDK](/documentation/sdks/java), using a
+[runner](/documentation#runners) of your choice.
-If you're interested in contributing to the Apache Beam Java codebase, see the [Contribution Guide](/contribute).
+If you're interested in contributing to the Apache Beam Java codebase, see the
+[Contribution Guide](/contribute).
-{{< toc >}}
+On this page:
-## Set up your Development Environment
-
-1. Download and install the [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html) version 8. Verify that the [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html) environment variable is set and points to your JDK installation.
-
-1. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by following Maven's [installation guide](https://maven.apache.org/install.html) for your specific operating system.
-
-1. Optional: Install [Gradle](https://gradle.org/install/) if you would like to convert your Maven project into Gradle.
-
-## Get the Example Code
-
-Use the following command to generate a Maven project that contains Beam's WordCount examples and builds against the most recent Beam release:
-
-{{< shell unix >}}
-$ mvn archetype:generate \
- -DarchetypeGroupId=org.apache.beam \
- -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
- -DarchetypeVersion={{< param release_latest >}} \
- -DgroupId=org.example \
- -DartifactId=word-count-beam \
- -Dversion="0.1" \
- -Dpackage=org.apache.beam.examples \
- -DinteractiveMode=false
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> mvn archetype:generate `
- -D archetypeGroupId=org.apache.beam `
- -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
- -D archetypeVersion={{< param release_latest >}} `
- -D groupId=org.example `
- -D artifactId=word-count-beam `
- -D version="0.1" `
- -D package=org.apache.beam.examples `
- -D interactiveMode=false
-{{< /shell >}}
-
-This will create a `word-count-beam` directory that contains a `pom.xml` and several example pipelines that count words in text files.
-
-{{< shell unix >}}
-$ cd word-count-beam/
-
-$ ls
-pom.xml src
-
-$ ls src/main/java/org/apache/beam/examples/
-DebuggingWordCount.java WindowedWordCount.java common
-MinimalWordCount.java WordCount.java
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> cd .\word-count-beam
-
-PS> dir
-
-...
-
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM src
--a---- 7/19/2018 11:00 PM 16051 pom.xml
-
-PS> dir .\src\main\java\org\apache\beam\examples
-
-...
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM common
-d----- 7/19/2018 11:00 PM complete
-d----- 7/19/2018 11:00 PM subprocess
--a---- 7/19/2018 11:00 PM 7073 DebuggingWordCount.java
--a---- 7/19/2018 11:00 PM 5945 MinimalWordCount.java
--a---- 7/19/2018 11:00 PM 9490 WindowedWordCount.java
--a---- 7/19/2018 11:00 PM 7662 WordCount.java
-{{< /shell >}}
-
-For a detailed introduction to the Beam concepts used in these examples, see the [WordCount Example Walkthrough](/get-started/wordcount-example). Here, we'll just focus on executing `WordCount.java`.
-
-## Optional: Convert from Maven to Gradle Project
-
-The steps below explain how to convert the build for the Direct Runner from Maven to Gradle. Converting the builds for the other runners is a more involved process and is out of scope for this guide. For additional guidance, see [Migrating Builds From Apache Maven](https://docs.gradle.org/current/userguide/migrating_from_maven.html).
-
-1. Ensure you are in the same directory as the `pom.xml` file generated from the previous step. Automatically convert your project from Maven to Gradle by running:
-{{< highlight >}}
-$ gradle init
-{{< /highlight >}}
-You'll be asked if you want to generate a Gradle build. Enter **yes**. You'll also be prompted to choose a DSL (Groovy or Kotlin). This tutorial uses Groovy, so select that if you don't have a preference.
-1. After you've converted the project to Gradle, open the generated `build.gradle` file, and, in the `repositories` block, replace `mavenLocal()` with `mavenCentral()`:
-{{< highlight >}}
-repositories {
- mavenCentral()
- maven {
- url = uri('https://repository.apache.org/content/repositories/snapshots/')
- }
+{{< toc >}}
- maven {
- url = uri('http://repo.maven.apache.org/maven2')
+## Set up your development environment
+
+1. Download and install the
+ [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html)
+ version 8, 11, or 17. Verify that the
+ [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html)
+ environment variable is set and points to your JDK installation.
+2. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by
+ following the [installation guide](https://maven.apache.org/install.html)
+ for your operating system.
+3. Optional: If you want to convert your Maven project to Gradle, install
+ [Gradle](https://gradle.org/install/).
+
+## Get the example code
+
+1. Generate a Maven example project that builds against the latest Beam release:
+ {{< shell unix >}}
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.beam \
+ -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
+ -DarchetypeVersion={{< param release_latest >}} \
+ -DgroupId=org.example \
+ -DartifactId=word-count-beam \
+ -Dversion="0.1" \
+ -Dpackage=org.apache.beam.examples \
+ -DinteractiveMode=false
+ {{< /shell >}}
+ {{< shell powerShell >}}
+mvn archetype:generate `
+ -D archetypeGroupId=org.apache.beam `
+ -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
+ -D archetypeVersion={{< param release_latest >}} `
+ -D groupId=org.example `
+ -D artifactId=word-count-beam `
+ -D version="0.1" `
+ -D package=org.apache.beam.examples `
+ -D interactiveMode=false
+ {{< /shell >}}
+
+ Maven creates a new project in the **word-count-beam** directory.
+
+2. Change into **word-count-beam**:
+ {{< shell unix >}}
+cd word-count-beam/
+ {{< /shell >}}
+ {{< shell powerShell >}}
+cd .\word-count-beam
+ {{< /shell >}}
+ The directory contains a **pom.xml** and a **src** directory with example
+ pipelines.
+
+3. List the example pipelines:
+ {{< shell unix >}}
+ls src/main/java/org/apache/beam/examples/
+ {{< /shell >}}
+ {{< shell powerShell >}}
+dir .\src\main\java\org\apache\beam\examples
+ {{< /shell >}}
+ You should see the following examples:
+ * **DebuggingWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java))
+ * **MinimalWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java))
+ * **WindowedWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java))
+ * **WordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WordCount.java))
+
+ The example used in this tutorial, **WordCount.java**, defines a
+ Beam pipeline that counts words from an input file (by default, a **.txt**
+ file containing Shakespeare's "King Lear"). To learn more about the examples,
+ see the [WordCount Example Walkthrough](/get-started/wordcount-example).
+
+## Optional: Convert from Maven to Gradle
+
+The steps below explain how to convert the build from Maven to Gradle for the
+following runners:
+* Direct runner
+* Dataflow runner
+
+The conversion process for other runners is similar. For additional guidance,
+see
+[Migrating Builds From Apache Maven](https://docs.gradle.org/current/userguide/migrating_from_maven.html).
+
+1. In the directory with the `pom.xml` file, run the automated Maven-to-Gradle
+ conversion:
+ {{< highlight >}}
+gradle init
+ {{< /highlight >}}
+ You'll be asked if you want to generate a Gradle build. Enter **yes**. You'll
+ also be prompted to choose a DSL (Groovy or Kotlin). For this tutorial, enter
+ **2** for Kotlin.
+2. Open the generated **build.gradle.kts** file and make the following changes:
+ 1. In `repositories`, replace `mavenLocal()` with `mavenCentral()`.
+ 2. In `repositories`, declare a repository for Confluent Kafka dependencies:
+ {{< highlight >}}
+maven {
+ url = uri("https://packages.confluent.io/maven/")
+}
+ {{< /highlight >}}
+ 3. At the end of the build script, add the following conditional dependency:
+ {{< highlight >}}
+if (project.hasProperty("dataflow-runner")) {
Review comment:
This will be different for the other runners right? Should we add one of the runner select blocks here?
##########
File path: website/www/site/content/en/get-started/quickstart-java.md
##########
@@ -19,462 +19,376 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache Beam Java SDK Quickstart
+# Apache Beam Java SDK quickstart
-This quickstart shows you how to set up a Java development environment and run an [example pipeline](/get-started/wordcount-example) written with the [Apache Beam Java SDK](/documentation/sdks/java), using a [runner](/documentation#runners) of your choice.
+This quickstart shows you how to set up a Java development environment and run
+an [example pipeline](/get-started/wordcount-example) written with the
+[Apache Beam Java SDK](/documentation/sdks/java), using a
+[runner](/documentation#runners) of your choice.
-If you're interested in contributing to the Apache Beam Java codebase, see the [Contribution Guide](/contribute).
+If you're interested in contributing to the Apache Beam Java codebase, see the
+[Contribution Guide](/contribute).
-{{< toc >}}
+On this page:
-## Set up your Development Environment
-
-1. Download and install the [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html) version 8. Verify that the [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html) environment variable is set and points to your JDK installation.
-
-1. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by following Maven's [installation guide](https://maven.apache.org/install.html) for your specific operating system.
-
-1. Optional: Install [Gradle](https://gradle.org/install/) if you would like to convert your Maven project into Gradle.
-
-## Get the Example Code
-
-Use the following command to generate a Maven project that contains Beam's WordCount examples and builds against the most recent Beam release:
-
-{{< shell unix >}}
-$ mvn archetype:generate \
- -DarchetypeGroupId=org.apache.beam \
- -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
- -DarchetypeVersion={{< param release_latest >}} \
- -DgroupId=org.example \
- -DartifactId=word-count-beam \
- -Dversion="0.1" \
- -Dpackage=org.apache.beam.examples \
- -DinteractiveMode=false
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> mvn archetype:generate `
- -D archetypeGroupId=org.apache.beam `
- -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
- -D archetypeVersion={{< param release_latest >}} `
- -D groupId=org.example `
- -D artifactId=word-count-beam `
- -D version="0.1" `
- -D package=org.apache.beam.examples `
- -D interactiveMode=false
-{{< /shell >}}
-
-This will create a `word-count-beam` directory that contains a `pom.xml` and several example pipelines that count words in text files.
-
-{{< shell unix >}}
-$ cd word-count-beam/
-
-$ ls
-pom.xml src
-
-$ ls src/main/java/org/apache/beam/examples/
-DebuggingWordCount.java WindowedWordCount.java common
-MinimalWordCount.java WordCount.java
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> cd .\word-count-beam
-
-PS> dir
-
-...
-
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM src
--a---- 7/19/2018 11:00 PM 16051 pom.xml
-
-PS> dir .\src\main\java\org\apache\beam\examples
-
-...
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM common
-d----- 7/19/2018 11:00 PM complete
-d----- 7/19/2018 11:00 PM subprocess
--a---- 7/19/2018 11:00 PM 7073 DebuggingWordCount.java
--a---- 7/19/2018 11:00 PM 5945 MinimalWordCount.java
--a---- 7/19/2018 11:00 PM 9490 WindowedWordCount.java
--a---- 7/19/2018 11:00 PM 7662 WordCount.java
-{{< /shell >}}
-
-For a detailed introduction to the Beam concepts used in these examples, see the [WordCount Example Walkthrough](/get-started/wordcount-example). Here, we'll just focus on executing `WordCount.java`.
-
-## Optional: Convert from Maven to Gradle Project
-
-The steps below explain how to convert the build for the Direct Runner from Maven to Gradle. Converting the builds for the other runners is a more involved process and is out of scope for this guide. For additional guidance, see [Migrating Builds From Apache Maven](https://docs.gradle.org/current/userguide/migrating_from_maven.html).
-
-1. Ensure you are in the same directory as the `pom.xml` file generated from the previous step. Automatically convert your project from Maven to Gradle by running:
-{{< highlight >}}
-$ gradle init
-{{< /highlight >}}
-You'll be asked if you want to generate a Gradle build. Enter **yes**. You'll also be prompted to choose a DSL (Groovy or Kotlin). This tutorial uses Groovy, so select that if you don't have a preference.
-1. After you've converted the project to Gradle, open the generated `build.gradle` file, and, in the `repositories` block, replace `mavenLocal()` with `mavenCentral()`:
-{{< highlight >}}
-repositories {
- mavenCentral()
- maven {
- url = uri('https://repository.apache.org/content/repositories/snapshots/')
- }
+{{< toc >}}
- maven {
- url = uri('http://repo.maven.apache.org/maven2')
+## Set up your development environment
+
+1. Download and install the
+ [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html)
+ version 8, 11, or 17. Verify that the
+ [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html)
+ environment variable is set and points to your JDK installation.
+2. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by
+ following the [installation guide](https://maven.apache.org/install.html)
+ for your operating system.
+3. Optional: If you want to convert your Maven project to Gradle, install
+ [Gradle](https://gradle.org/install/).
+
+## Get the example code
+
+1. Generate a Maven example project that builds against the latest Beam release:
+ {{< shell unix >}}
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.beam \
+ -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
+ -DarchetypeVersion={{< param release_latest >}} \
+ -DgroupId=org.example \
+ -DartifactId=word-count-beam \
+ -Dversion="0.1" \
+ -Dpackage=org.apache.beam.examples \
+ -DinteractiveMode=false
+ {{< /shell >}}
+ {{< shell powerShell >}}
+mvn archetype:generate `
+ -D archetypeGroupId=org.apache.beam `
+ -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
+ -D archetypeVersion={{< param release_latest >}} `
+ -D groupId=org.example `
+ -D artifactId=word-count-beam `
+ -D version="0.1" `
+ -D package=org.apache.beam.examples `
+ -D interactiveMode=false
+ {{< /shell >}}
+
+ Maven creates a new project in the **word-count-beam** directory.
+
+2. Change into **word-count-beam**:
+ {{< shell unix >}}
+cd word-count-beam/
+ {{< /shell >}}
+ {{< shell powerShell >}}
+cd .\word-count-beam
+ {{< /shell >}}
+ The directory contains a **pom.xml** and a **src** directory with example
Review comment:
When do we use bold vs the red text? Pom.xml in the next section uses `
##########
File path: website/www/site/content/en/get-started/quickstart-java.md
##########
@@ -19,462 +19,376 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache Beam Java SDK Quickstart
+# Apache Beam Java SDK quickstart
-This quickstart shows you how to set up a Java development environment and run an [example pipeline](/get-started/wordcount-example) written with the [Apache Beam Java SDK](/documentation/sdks/java), using a [runner](/documentation#runners) of your choice.
+This quickstart shows you how to set up a Java development environment and run
+an [example pipeline](/get-started/wordcount-example) written with the
+[Apache Beam Java SDK](/documentation/sdks/java), using a
+[runner](/documentation#runners) of your choice.
-If you're interested in contributing to the Apache Beam Java codebase, see the [Contribution Guide](/contribute).
+If you're interested in contributing to the Apache Beam Java codebase, see the
+[Contribution Guide](/contribute).
-{{< toc >}}
+On this page:
-## Set up your Development Environment
-
-1. Download and install the [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html) version 8. Verify that the [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html) environment variable is set and points to your JDK installation.
-
-1. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by following Maven's [installation guide](https://maven.apache.org/install.html) for your specific operating system.
-
-1. Optional: Install [Gradle](https://gradle.org/install/) if you would like to convert your Maven project into Gradle.
-
-## Get the Example Code
-
-Use the following command to generate a Maven project that contains Beam's WordCount examples and builds against the most recent Beam release:
-
-{{< shell unix >}}
-$ mvn archetype:generate \
- -DarchetypeGroupId=org.apache.beam \
- -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
- -DarchetypeVersion={{< param release_latest >}} \
- -DgroupId=org.example \
- -DartifactId=word-count-beam \
- -Dversion="0.1" \
- -Dpackage=org.apache.beam.examples \
- -DinteractiveMode=false
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> mvn archetype:generate `
- -D archetypeGroupId=org.apache.beam `
- -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
- -D archetypeVersion={{< param release_latest >}} `
- -D groupId=org.example `
- -D artifactId=word-count-beam `
- -D version="0.1" `
- -D package=org.apache.beam.examples `
- -D interactiveMode=false
-{{< /shell >}}
-
-This will create a `word-count-beam` directory that contains a `pom.xml` and several example pipelines that count words in text files.
-
-{{< shell unix >}}
-$ cd word-count-beam/
-
-$ ls
-pom.xml src
-
-$ ls src/main/java/org/apache/beam/examples/
-DebuggingWordCount.java WindowedWordCount.java common
-MinimalWordCount.java WordCount.java
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> cd .\word-count-beam
-
-PS> dir
-
-...
-
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM src
--a---- 7/19/2018 11:00 PM 16051 pom.xml
-
-PS> dir .\src\main\java\org\apache\beam\examples
-
-...
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM common
-d----- 7/19/2018 11:00 PM complete
-d----- 7/19/2018 11:00 PM subprocess
--a---- 7/19/2018 11:00 PM 7073 DebuggingWordCount.java
--a---- 7/19/2018 11:00 PM 5945 MinimalWordCount.java
--a---- 7/19/2018 11:00 PM 9490 WindowedWordCount.java
--a---- 7/19/2018 11:00 PM 7662 WordCount.java
-{{< /shell >}}
-
-For a detailed introduction to the Beam concepts used in these examples, see the [WordCount Example Walkthrough](/get-started/wordcount-example). Here, we'll just focus on executing `WordCount.java`.
-
-## Optional: Convert from Maven to Gradle Project
-
-The steps below explain how to convert the build for the Direct Runner from Maven to Gradle. Converting the builds for the other runners is a more involved process and is out of scope for this guide. For additional guidance, see [Migrating Builds From Apache Maven](https://docs.gradle.org/current/userguide/migrating_from_maven.html).
-
-1. Ensure you are in the same directory as the `pom.xml` file generated from the previous step. Automatically convert your project from Maven to Gradle by running:
-{{< highlight >}}
-$ gradle init
-{{< /highlight >}}
-You'll be asked if you want to generate a Gradle build. Enter **yes**. You'll also be prompted to choose a DSL (Groovy or Kotlin). This tutorial uses Groovy, so select that if you don't have a preference.
-1. After you've converted the project to Gradle, open the generated `build.gradle` file, and, in the `repositories` block, replace `mavenLocal()` with `mavenCentral()`:
-{{< highlight >}}
-repositories {
- mavenCentral()
- maven {
- url = uri('https://repository.apache.org/content/repositories/snapshots/')
- }
+{{< toc >}}
- maven {
- url = uri('http://repo.maven.apache.org/maven2')
+## Set up your development environment
+
+1. Download and install the
+ [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html)
+ version 8, 11, or 17. Verify that the
+ [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html)
+ environment variable is set and points to your JDK installation.
+2. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by
+ following the [installation guide](https://maven.apache.org/install.html)
+ for your operating system.
+3. Optional: If you want to convert your Maven project to Gradle, install
+ [Gradle](https://gradle.org/install/).
+
+## Get the example code
+
+1. Generate a Maven example project that builds against the latest Beam release:
+ {{< shell unix >}}
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.beam \
+ -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
+ -DarchetypeVersion={{< param release_latest >}} \
+ -DgroupId=org.example \
+ -DartifactId=word-count-beam \
+ -Dversion="0.1" \
+ -Dpackage=org.apache.beam.examples \
+ -DinteractiveMode=false
+ {{< /shell >}}
+ {{< shell powerShell >}}
+mvn archetype:generate `
+ -D archetypeGroupId=org.apache.beam `
+ -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
+ -D archetypeVersion={{< param release_latest >}} `
+ -D groupId=org.example `
+ -D artifactId=word-count-beam `
+ -D version="0.1" `
+ -D package=org.apache.beam.examples `
+ -D interactiveMode=false
+ {{< /shell >}}
+
+ Maven creates a new project in the **word-count-beam** directory.
+
+2. Change into **word-count-beam**:
+ {{< shell unix >}}
+cd word-count-beam/
+ {{< /shell >}}
+ {{< shell powerShell >}}
+cd .\word-count-beam
+ {{< /shell >}}
+ The directory contains a **pom.xml** and a **src** directory with example
+ pipelines.
+
+3. List the example pipelines:
+ {{< shell unix >}}
+ls src/main/java/org/apache/beam/examples/
+ {{< /shell >}}
+ {{< shell powerShell >}}
+dir .\src\main\java\org\apache\beam\examples
+ {{< /shell >}}
+ You should see the following examples:
+ * **DebuggingWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java))
+ * **MinimalWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java))
+ * **WindowedWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java))
+ * **WordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WordCount.java))
+
+ The example used in this tutorial, **WordCount.java**, defines a
+ Beam pipeline that counts words from an input file (by default, a **.txt**
+ file containing Shakespeare's "King Lear"). To learn more about the examples,
+ see the [WordCount Example Walkthrough](/get-started/wordcount-example).
+
+## Optional: Convert from Maven to Gradle
+
+The steps below explain how to convert the build from Maven to Gradle for the
+following runners:
+* Direct runner
+* Dataflow runner
+
+The conversion process for other runners is similar. For additional guidance,
+see
+[Migrating Builds From Apache Maven](https://docs.gradle.org/current/userguide/migrating_from_maven.html).
+
+1. In the directory with the `pom.xml` file, run the automated Maven-to-Gradle
+ conversion:
+ {{< highlight >}}
+gradle init
+ {{< /highlight >}}
+ You'll be asked if you want to generate a Gradle build. Enter **yes**. You'll
+ also be prompted to choose a DSL (Groovy or Kotlin). For this tutorial, enter
+ **2** for Kotlin.
+2. Open the generated **build.gradle.kts** file and make the following changes:
+ 1. In `repositories`, replace `mavenLocal()` with `mavenCentral()`.
+ 2. In `repositories`, declare a repository for Confluent Kafka dependencies:
+ {{< highlight >}}
+maven {
+ url = uri("https://packages.confluent.io/maven/")
+}
+ {{< /highlight >}}
+ 3. At the end of the build script, add the following conditional dependency:
+ {{< highlight >}}
+if (project.hasProperty("dataflow-runner")) {
+ dependencies {
+ runtimeOnly("org.apache.beam:beam-runners-google-cloud-dataflow-java:{{< param release_latest >}}")
}
}
-{{< /highlight >}}
-1. Add the following task in `build.gradle` to allow you to execute pipelines with Gradle:
-{{< highlight >}}
-task execute (type:JavaExec) {
- mainClass = System.getProperty("mainClass")
- classpath = sourceSets.main.runtimeClasspath
- systemProperties System.getProperties()
- args System.getProperty("exec.args", "").split()
+ {{< /highlight >}}
+ 4. At the end of the build script, add the following task:
+ {{< highlight >}}
+task("execute", JavaExec::class) {
+ classpath = sourceSets["main"].runtimeClasspath
+ mainClass.set(System.getProperty("mainClass"))
}
-{{< /highlight >}}
-1. Rebuild your project by running:
-{{< highlight >}}
-$ gradle build
-{{< /highlight >}}
+ {{< /highlight >}}
+4. Build your project:
+ {{< highlight >}}
+gradle build
+ {{< /highlight >}}
## Get sample text
-> If you're planning to use the DataflowRunner, you can skip this step. The runner will pull text directly from Google Cloud Storage.
+> If you're planning to use the DataflowRunner, you can skip this step. The
+ runner will pull text directly from Google Cloud Storage.
1. In the **word-count-beam** directory, create a file called **sample.txt**.
-1. Add some text to the file. For this example, you can use the text of Shakespeare's [Sonnets](https://storage.cloud.google.com/apache-beam-samples/shakespeare/sonnets.txt).
+2. Add some text to the file. For this example, use the text of Shakespeare's
+ [King Lear](https://storage.cloud.google.com/apache-beam-samples/shakespeare/kinglear.txt).
## Run a pipeline
-A single Beam pipeline can run on multiple Beam [runners](/documentation#runners), including the [FlinkRunner](/documentation/runners/flink), [SparkRunner](/documentation/runners/spark), [NemoRunner](/documentation/runners/nemo), [JetRunner](/documentation/runners/jet), or [DataflowRunner](/documentation/runners/dataflow). The [DirectRunner](/documentation/runners/direct) is a common runner for getting started, as it runs locally on your machine and requires no specific setup. If you're just trying out Beam and you're not sure what to use, use the [DirectRunner](/documentation/runners/direct).
+A single Beam pipeline can run on multiple Beam
+[runners](/documentation#runners), including the
+[FlinkRunner](/documentation/runners/flink),
+[SparkRunner](/documentation/runners/spark),
+[NemoRunner](/documentation/runners/nemo),
+[JetRunner](/documentation/runners/jet), and
+[DataflowRunner](/documentation/runners/dataflow). The
+[DirectRunner](/documentation/runners/direct) is useful for getting started,
+because it runs on your machine and requires no specific setup. If you're just
+trying out Beam and you're not sure what to use, use the
+[DirectRunner](/documentation/runners/direct).
The general process for running a pipeline goes like this:
-1. Ensure you've done any runner-specific setup.
-1. Build your command line:
- 1. Specify a runner with `--runner=<runner>` (defaults to the [DirectRunner](/documentation/runners/direct)).
- 1. Add any runner-specific required options.
- 1. Choose input files and an output location that are accessible to the runner. (For example, you can't access a local file if you are running the pipeline on an external cluster.)
-1. Run the command.
+1. Complete any runner-specific setup.
+2. Build your command line:
+ 1. Specify a runner with `--runner=<runner>` (defaults to the
+ [DirectRunner](/documentation/runners/direct)).
+ 2. Add any runner-specific required options.
+ 3. Choose input files and an output location that are accessible to the
+ runner. (For example, you can't access a local file if you are running
+ the pipeline on an external cluster.)
+3. Run the command.
To run the WordCount pipeline, see the Maven and Gradle examples below.
-### Run WordCount Using Maven
+### Run WordCount using Maven
For Unix shells:
{{< runner direct >}}
-$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
- -Dexec.args="--inputFile=sample.txt --output=counts" -Pdirect-runner
+mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="--inputFile=sample.txt --output=counts" -Pdirect-runner
{{< /runner >}}
-
{{< runner flink >}}
-$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
- -Dexec.args="--runner=FlinkRunner --inputFile=sample.txt --output=counts" -Pflink-runner
+mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="--runner=FlinkRunner --inputFile=sample.txt --output=counts" -Pflink-runner
{{< /runner >}}
-
{{< runner flinkCluster >}}
-$ mvn package exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
- -Dexec.args="--runner=FlinkRunner --flinkMaster=<flink master> --filesToStage=target/word-count-beam-bundled-0.1.jar \
- --inputFile=sample.txt --output=/tmp/counts" -Pflink-runner
-
-You can monitor the running job by visiting the Flink dashboard at http://<flink master>:8081
Review comment:
Can we save this as a comment somewhere? Or is it printed in the console somewhere?
##########
File path: website/www/site/content/en/get-started/quickstart-java.md
##########
@@ -19,462 +19,376 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache Beam Java SDK Quickstart
+# Apache Beam Java SDK quickstart
-This quickstart shows you how to set up a Java development environment and run an [example pipeline](/get-started/wordcount-example) written with the [Apache Beam Java SDK](/documentation/sdks/java), using a [runner](/documentation#runners) of your choice.
+This quickstart shows you how to set up a Java development environment and run
+an [example pipeline](/get-started/wordcount-example) written with the
+[Apache Beam Java SDK](/documentation/sdks/java), using a
+[runner](/documentation#runners) of your choice.
-If you're interested in contributing to the Apache Beam Java codebase, see the [Contribution Guide](/contribute).
+If you're interested in contributing to the Apache Beam Java codebase, see the
+[Contribution Guide](/contribute).
-{{< toc >}}
+On this page:
-## Set up your Development Environment
-
-1. Download and install the [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html) version 8. Verify that the [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html) environment variable is set and points to your JDK installation.
-
-1. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by following Maven's [installation guide](https://maven.apache.org/install.html) for your specific operating system.
-
-1. Optional: Install [Gradle](https://gradle.org/install/) if you would like to convert your Maven project into Gradle.
-
-## Get the Example Code
-
-Use the following command to generate a Maven project that contains Beam's WordCount examples and builds against the most recent Beam release:
-
-{{< shell unix >}}
-$ mvn archetype:generate \
- -DarchetypeGroupId=org.apache.beam \
- -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
- -DarchetypeVersion={{< param release_latest >}} \
- -DgroupId=org.example \
- -DartifactId=word-count-beam \
- -Dversion="0.1" \
- -Dpackage=org.apache.beam.examples \
- -DinteractiveMode=false
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> mvn archetype:generate `
- -D archetypeGroupId=org.apache.beam `
- -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
- -D archetypeVersion={{< param release_latest >}} `
- -D groupId=org.example `
- -D artifactId=word-count-beam `
- -D version="0.1" `
- -D package=org.apache.beam.examples `
- -D interactiveMode=false
-{{< /shell >}}
-
-This will create a `word-count-beam` directory that contains a `pom.xml` and several example pipelines that count words in text files.
-
-{{< shell unix >}}
-$ cd word-count-beam/
-
-$ ls
-pom.xml src
-
-$ ls src/main/java/org/apache/beam/examples/
-DebuggingWordCount.java WindowedWordCount.java common
-MinimalWordCount.java WordCount.java
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> cd .\word-count-beam
-
-PS> dir
-
-...
-
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM src
--a---- 7/19/2018 11:00 PM 16051 pom.xml
-
-PS> dir .\src\main\java\org\apache\beam\examples
-
-...
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM common
-d----- 7/19/2018 11:00 PM complete
-d----- 7/19/2018 11:00 PM subprocess
--a---- 7/19/2018 11:00 PM 7073 DebuggingWordCount.java
--a---- 7/19/2018 11:00 PM 5945 MinimalWordCount.java
--a---- 7/19/2018 11:00 PM 9490 WindowedWordCount.java
--a---- 7/19/2018 11:00 PM 7662 WordCount.java
-{{< /shell >}}
-
-For a detailed introduction to the Beam concepts used in these examples, see the [WordCount Example Walkthrough](/get-started/wordcount-example). Here, we'll just focus on executing `WordCount.java`.
-
-## Optional: Convert from Maven to Gradle Project
-
-The steps below explain how to convert the build for the Direct Runner from Maven to Gradle. Converting the builds for the other runners is a more involved process and is out of scope for this guide. For additional guidance, see [Migrating Builds From Apache Maven](https://docs.gradle.org/current/userguide/migrating_from_maven.html).
-
-1. Ensure you are in the same directory as the `pom.xml` file generated from the previous step. Automatically convert your project from Maven to Gradle by running:
-{{< highlight >}}
-$ gradle init
-{{< /highlight >}}
-You'll be asked if you want to generate a Gradle build. Enter **yes**. You'll also be prompted to choose a DSL (Groovy or Kotlin). This tutorial uses Groovy, so select that if you don't have a preference.
-1. After you've converted the project to Gradle, open the generated `build.gradle` file, and, in the `repositories` block, replace `mavenLocal()` with `mavenCentral()`:
-{{< highlight >}}
-repositories {
- mavenCentral()
- maven {
- url = uri('https://repository.apache.org/content/repositories/snapshots/')
- }
+{{< toc >}}
- maven {
- url = uri('http://repo.maven.apache.org/maven2')
+## Set up your development environment
+
+1. Download and install the
+ [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html)
+ version 8, 11, or 17. Verify that the
+ [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html)
+ environment variable is set and points to your JDK installation.
+2. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by
+ following the [installation guide](https://maven.apache.org/install.html)
+ for your operating system.
+3. Optional: If you want to convert your Maven project to Gradle, install
+ [Gradle](https://gradle.org/install/).
+
+## Get the example code
+
+1. Generate a Maven example project that builds against the latest Beam release:
+ {{< shell unix >}}
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.beam \
+ -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
+ -DarchetypeVersion={{< param release_latest >}} \
+ -DgroupId=org.example \
+ -DartifactId=word-count-beam \
+ -Dversion="0.1" \
+ -Dpackage=org.apache.beam.examples \
+ -DinteractiveMode=false
+ {{< /shell >}}
+ {{< shell powerShell >}}
+mvn archetype:generate `
+ -D archetypeGroupId=org.apache.beam `
+ -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
+ -D archetypeVersion={{< param release_latest >}} `
+ -D groupId=org.example `
+ -D artifactId=word-count-beam `
+ -D version="0.1" `
+ -D package=org.apache.beam.examples `
+ -D interactiveMode=false
+ {{< /shell >}}
+
+ Maven creates a new project in the **word-count-beam** directory.
+
+2. Change into **word-count-beam**:
+ {{< shell unix >}}
+cd word-count-beam/
+ {{< /shell >}}
+ {{< shell powerShell >}}
+cd .\word-count-beam
+ {{< /shell >}}
+ The directory contains a **pom.xml** and a **src** directory with example
+ pipelines.
+
+3. List the example pipelines:
+ {{< shell unix >}}
+ls src/main/java/org/apache/beam/examples/
+ {{< /shell >}}
+ {{< shell powerShell >}}
+dir .\src\main\java\org\apache\beam\examples
+ {{< /shell >}}
+ You should see the following examples:
+ * **DebuggingWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java))
+ * **MinimalWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java))
+ * **WindowedWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java))
+ * **WordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WordCount.java))
+
+ The example used in this tutorial, **WordCount.java**, defines a
+ Beam pipeline that counts words from an input file (by default, a **.txt**
+ file containing Shakespeare's "King Lear"). To learn more about the examples,
+ see the [WordCount Example Walkthrough](/get-started/wordcount-example).
+
+## Optional: Convert from Maven to Gradle
+
+The steps below explain how to convert the build from Maven to Gradle for the
+following runners:
+* Direct runner
+* Dataflow runner
+
+The conversion process for other runners is similar. For additional guidance,
+see
+[Migrating Builds From Apache Maven](https://docs.gradle.org/current/userguide/migrating_from_maven.html).
+
+1. In the directory with the `pom.xml` file, run the automated Maven-to-Gradle
+ conversion:
+ {{< highlight >}}
+gradle init
+ {{< /highlight >}}
+ You'll be asked if you want to generate a Gradle build. Enter **yes**. You'll
+ also be prompted to choose a DSL (Groovy or Kotlin). For this tutorial, enter
+ **2** for Kotlin.
+2. Open the generated **build.gradle.kts** file and make the following changes:
+ 1. In `repositories`, replace `mavenLocal()` with `mavenCentral()`.
+ 2. In `repositories`, declare a repository for Confluent Kafka dependencies:
+ {{< highlight >}}
+maven {
+ url = uri("https://packages.confluent.io/maven/")
+}
+ {{< /highlight >}}
+ 3. At the end of the build script, add the following conditional dependency:
+ {{< highlight >}}
+if (project.hasProperty("dataflow-runner")) {
+ dependencies {
+ runtimeOnly("org.apache.beam:beam-runners-google-cloud-dataflow-java:{{< param release_latest >}}")
}
}
-{{< /highlight >}}
-1. Add the following task in `build.gradle` to allow you to execute pipelines with Gradle:
-{{< highlight >}}
-task execute (type:JavaExec) {
- mainClass = System.getProperty("mainClass")
- classpath = sourceSets.main.runtimeClasspath
- systemProperties System.getProperties()
- args System.getProperty("exec.args", "").split()
+ {{< /highlight >}}
+ 4. At the end of the build script, add the following task:
+ {{< highlight >}}
+task("execute", JavaExec::class) {
+ classpath = sourceSets["main"].runtimeClasspath
+ mainClass.set(System.getProperty("mainClass"))
}
-{{< /highlight >}}
-1. Rebuild your project by running:
-{{< highlight >}}
-$ gradle build
-{{< /highlight >}}
+ {{< /highlight >}}
+4. Build your project:
+ {{< highlight >}}
+gradle build
+ {{< /highlight >}}
## Get sample text
-> If you're planning to use the DataflowRunner, you can skip this step. The runner will pull text directly from Google Cloud Storage.
+> If you're planning to use the DataflowRunner, you can skip this step. The
+ runner will pull text directly from Google Cloud Storage.
1. In the **word-count-beam** directory, create a file called **sample.txt**.
-1. Add some text to the file. For this example, you can use the text of Shakespeare's [Sonnets](https://storage.cloud.google.com/apache-beam-samples/shakespeare/sonnets.txt).
+2. Add some text to the file. For this example, use the text of Shakespeare's
+ [King Lear](https://storage.cloud.google.com/apache-beam-samples/shakespeare/kinglear.txt).
## Run a pipeline
-A single Beam pipeline can run on multiple Beam [runners](/documentation#runners), including the [FlinkRunner](/documentation/runners/flink), [SparkRunner](/documentation/runners/spark), [NemoRunner](/documentation/runners/nemo), [JetRunner](/documentation/runners/jet), or [DataflowRunner](/documentation/runners/dataflow). The [DirectRunner](/documentation/runners/direct) is a common runner for getting started, as it runs locally on your machine and requires no specific setup. If you're just trying out Beam and you're not sure what to use, use the [DirectRunner](/documentation/runners/direct).
+A single Beam pipeline can run on multiple Beam
+[runners](/documentation#runners), including the
+[FlinkRunner](/documentation/runners/flink),
+[SparkRunner](/documentation/runners/spark),
+[NemoRunner](/documentation/runners/nemo),
+[JetRunner](/documentation/runners/jet), and
+[DataflowRunner](/documentation/runners/dataflow). The
+[DirectRunner](/documentation/runners/direct) is useful for getting started,
+because it runs on your machine and requires no specific setup. If you're just
+trying out Beam and you're not sure what to use, use the
+[DirectRunner](/documentation/runners/direct).
The general process for running a pipeline goes like this:
-1. Ensure you've done any runner-specific setup.
-1. Build your command line:
- 1. Specify a runner with `--runner=<runner>` (defaults to the [DirectRunner](/documentation/runners/direct)).
- 1. Add any runner-specific required options.
- 1. Choose input files and an output location that are accessible to the runner. (For example, you can't access a local file if you are running the pipeline on an external cluster.)
-1. Run the command.
+1. Complete any runner-specific setup.
+2. Build your command line:
+ 1. Specify a runner with `--runner=<runner>` (defaults to the
+ [DirectRunner](/documentation/runners/direct)).
+ 2. Add any runner-specific required options.
+ 3. Choose input files and an output location that are accessible to the
+ runner. (For example, you can't access a local file if you are running
+ the pipeline on an external cluster.)
+3. Run the command.
To run the WordCount pipeline, see the Maven and Gradle examples below.
-### Run WordCount Using Maven
+### Run WordCount using Maven
For Unix shells:
{{< runner direct >}}
-$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
- -Dexec.args="--inputFile=sample.txt --output=counts" -Pdirect-runner
+mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="--inputFile=sample.txt --output=counts" -Pdirect-runner
{{< /runner >}}
-
{{< runner flink >}}
-$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
- -Dexec.args="--runner=FlinkRunner --inputFile=sample.txt --output=counts" -Pflink-runner
+mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="--runner=FlinkRunner --inputFile=sample.txt --output=counts" -Pflink-runner
{{< /runner >}}
-
{{< runner flinkCluster >}}
-$ mvn package exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
- -Dexec.args="--runner=FlinkRunner --flinkMaster=<flink master> --filesToStage=target/word-count-beam-bundled-0.1.jar \
- --inputFile=sample.txt --output=/tmp/counts" -Pflink-runner
-
-You can monitor the running job by visiting the Flink dashboard at http://<flink master>:8081
+mvn package exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="--runner=FlinkRunner --flinkMaster=<flink master> --filesToStage=target/word-count-beam-bundled-0.1.jar \
+ --inputFile=sample.txt --output=/tmp/counts" -Pflink-runner
{{< /runner >}}
-
{{< runner spark >}}
-$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
- -Dexec.args="--runner=SparkRunner --inputFile=sample.txt --output=counts" -Pspark-runner
+mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="--runner=SparkRunner --inputFile=sample.txt --output=counts" -Pspark-runner
{{< /runner >}}
-
{{< runner dataflow >}}
-Make sure you complete the setup steps at /documentation/runners/dataflow/#setup
Review comment:
Same with this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pcoet commented on a change in pull request #17033: BEAM-12770: Documented maven-to-gradle conversion for Dataflow
Posted by GitBox <gi...@apache.org>.
pcoet commented on a change in pull request #17033:
URL: https://github.com/apache/beam/pull/17033#discussion_r826488518
##########
File path: website/www/site/content/en/get-started/quickstart-java.md
##########
@@ -19,462 +19,376 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache Beam Java SDK Quickstart
+# Apache Beam Java SDK quickstart
-This quickstart shows you how to set up a Java development environment and run an [example pipeline](/get-started/wordcount-example) written with the [Apache Beam Java SDK](/documentation/sdks/java), using a [runner](/documentation#runners) of your choice.
+This quickstart shows you how to set up a Java development environment and run
+an [example pipeline](/get-started/wordcount-example) written with the
+[Apache Beam Java SDK](/documentation/sdks/java), using a
+[runner](/documentation#runners) of your choice.
-If you're interested in contributing to the Apache Beam Java codebase, see the [Contribution Guide](/contribute).
+If you're interested in contributing to the Apache Beam Java codebase, see the
+[Contribution Guide](/contribute).
-{{< toc >}}
+On this page:
-## Set up your Development Environment
-
-1. Download and install the [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html) version 8. Verify that the [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html) environment variable is set and points to your JDK installation.
-
-1. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by following Maven's [installation guide](https://maven.apache.org/install.html) for your specific operating system.
-
-1. Optional: Install [Gradle](https://gradle.org/install/) if you would like to convert your Maven project into Gradle.
-
-## Get the Example Code
-
-Use the following command to generate a Maven project that contains Beam's WordCount examples and builds against the most recent Beam release:
-
-{{< shell unix >}}
-$ mvn archetype:generate \
- -DarchetypeGroupId=org.apache.beam \
- -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
- -DarchetypeVersion={{< param release_latest >}} \
- -DgroupId=org.example \
- -DartifactId=word-count-beam \
- -Dversion="0.1" \
- -Dpackage=org.apache.beam.examples \
- -DinteractiveMode=false
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> mvn archetype:generate `
- -D archetypeGroupId=org.apache.beam `
- -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
- -D archetypeVersion={{< param release_latest >}} `
- -D groupId=org.example `
- -D artifactId=word-count-beam `
- -D version="0.1" `
- -D package=org.apache.beam.examples `
- -D interactiveMode=false
-{{< /shell >}}
-
-This will create a `word-count-beam` directory that contains a `pom.xml` and several example pipelines that count words in text files.
-
-{{< shell unix >}}
-$ cd word-count-beam/
-
-$ ls
-pom.xml src
-
-$ ls src/main/java/org/apache/beam/examples/
-DebuggingWordCount.java WindowedWordCount.java common
-MinimalWordCount.java WordCount.java
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> cd .\word-count-beam
-
-PS> dir
-
-...
-
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM src
--a---- 7/19/2018 11:00 PM 16051 pom.xml
-
-PS> dir .\src\main\java\org\apache\beam\examples
-
-...
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM common
-d----- 7/19/2018 11:00 PM complete
-d----- 7/19/2018 11:00 PM subprocess
--a---- 7/19/2018 11:00 PM 7073 DebuggingWordCount.java
--a---- 7/19/2018 11:00 PM 5945 MinimalWordCount.java
--a---- 7/19/2018 11:00 PM 9490 WindowedWordCount.java
--a---- 7/19/2018 11:00 PM 7662 WordCount.java
-{{< /shell >}}
-
-For a detailed introduction to the Beam concepts used in these examples, see the [WordCount Example Walkthrough](/get-started/wordcount-example). Here, we'll just focus on executing `WordCount.java`.
-
-## Optional: Convert from Maven to Gradle Project
-
-The steps below explain how to convert the build for the Direct Runner from Maven to Gradle. Converting the builds for the other runners is a more involved process and is out of scope for this guide. For additional guidance, see [Migrating Builds From Apache Maven](https://docs.gradle.org/current/userguide/migrating_from_maven.html).
-
-1. Ensure you are in the same directory as the `pom.xml` file generated from the previous step. Automatically convert your project from Maven to Gradle by running:
-{{< highlight >}}
-$ gradle init
-{{< /highlight >}}
-You'll be asked if you want to generate a Gradle build. Enter **yes**. You'll also be prompted to choose a DSL (Groovy or Kotlin). This tutorial uses Groovy, so select that if you don't have a preference.
-1. After you've converted the project to Gradle, open the generated `build.gradle` file, and, in the `repositories` block, replace `mavenLocal()` with `mavenCentral()`:
-{{< highlight >}}
-repositories {
- mavenCentral()
- maven {
- url = uri('https://repository.apache.org/content/repositories/snapshots/')
- }
+{{< toc >}}
- maven {
- url = uri('http://repo.maven.apache.org/maven2')
+## Set up your development environment
+
+1. Download and install the
+ [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html)
+ version 8, 11, or 17. Verify that the
+ [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html)
+ environment variable is set and points to your JDK installation.
+2. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by
+ following the [installation guide](https://maven.apache.org/install.html)
+ for your operating system.
+3. Optional: If you want to convert your Maven project to Gradle, install
+ [Gradle](https://gradle.org/install/).
+
+## Get the example code
+
+1. Generate a Maven example project that builds against the latest Beam release:
+ {{< shell unix >}}
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.beam \
+ -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
+ -DarchetypeVersion={{< param release_latest >}} \
+ -DgroupId=org.example \
+ -DartifactId=word-count-beam \
+ -Dversion="0.1" \
+ -Dpackage=org.apache.beam.examples \
+ -DinteractiveMode=false
+ {{< /shell >}}
+ {{< shell powerShell >}}
+mvn archetype:generate `
+ -D archetypeGroupId=org.apache.beam `
+ -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
+ -D archetypeVersion={{< param release_latest >}} `
+ -D groupId=org.example `
+ -D artifactId=word-count-beam `
+ -D version="0.1" `
+ -D package=org.apache.beam.examples `
+ -D interactiveMode=false
+ {{< /shell >}}
+
+ Maven creates a new project in the **word-count-beam** directory.
+
+2. Change into **word-count-beam**:
+ {{< shell unix >}}
+cd word-count-beam/
+ {{< /shell >}}
+ {{< shell powerShell >}}
+cd .\word-count-beam
+ {{< /shell >}}
+ The directory contains a **pom.xml** and a **src** directory with example
+ pipelines.
+
+3. List the example pipelines:
+ {{< shell unix >}}
+ls src/main/java/org/apache/beam/examples/
+ {{< /shell >}}
+ {{< shell powerShell >}}
+dir .\src\main\java\org\apache\beam\examples
+ {{< /shell >}}
+ You should see the following examples:
+ * **DebuggingWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java))
+ * **MinimalWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java))
+ * **WindowedWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java))
+ * **WordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WordCount.java))
+
+ The example used in this tutorial, **WordCount.java**, defines a
+ Beam pipeline that counts words from an input file (by default, a **.txt**
+ file containing Shakespeare's "King Lear"). To learn more about the examples,
+ see the [WordCount Example Walkthrough](/get-started/wordcount-example).
+
+## Optional: Convert from Maven to Gradle
+
+The steps below explain how to convert the build from Maven to Gradle for the
+following runners:
+* Direct runner
+* Dataflow runner
+
+The conversion process for other runners is similar. For additional guidance,
+see
+[Migrating Builds From Apache Maven](https://docs.gradle.org/current/userguide/migrating_from_maven.html).
+
+1. In the directory with the `pom.xml` file, run the automated Maven-to-Gradle
+ conversion:
+ {{< highlight >}}
+gradle init
+ {{< /highlight >}}
+ You'll be asked if you want to generate a Gradle build. Enter **yes**. You'll
+ also be prompted to choose a DSL (Groovy or Kotlin). For this tutorial, enter
+ **2** for Kotlin.
+2. Open the generated **build.gradle.kts** file and make the following changes:
+ 1. In `repositories`, replace `mavenLocal()` with `mavenCentral()`.
+ 2. In `repositories`, declare a repository for Confluent Kafka dependencies:
+ {{< highlight >}}
+maven {
+ url = uri("https://packages.confluent.io/maven/")
+}
+ {{< /highlight >}}
+ 3. At the end of the build script, add the following conditional dependency:
+ {{< highlight >}}
+if (project.hasProperty("dataflow-runner")) {
+ dependencies {
+ runtimeOnly("org.apache.beam:beam-runners-google-cloud-dataflow-java:{{< param release_latest >}}")
}
}
-{{< /highlight >}}
-1. Add the following task in `build.gradle` to allow you to execute pipelines with Gradle:
-{{< highlight >}}
-task execute (type:JavaExec) {
- mainClass = System.getProperty("mainClass")
- classpath = sourceSets.main.runtimeClasspath
- systemProperties System.getProperties()
- args System.getProperty("exec.args", "").split()
+ {{< /highlight >}}
+ 4. At the end of the build script, add the following task:
+ {{< highlight >}}
+task("execute", JavaExec::class) {
+ classpath = sourceSets["main"].runtimeClasspath
+ mainClass.set(System.getProperty("mainClass"))
}
-{{< /highlight >}}
-1. Rebuild your project by running:
-{{< highlight >}}
-$ gradle build
-{{< /highlight >}}
+ {{< /highlight >}}
+4. Build your project:
+ {{< highlight >}}
+gradle build
+ {{< /highlight >}}
## Get sample text
-> If you're planning to use the DataflowRunner, you can skip this step. The runner will pull text directly from Google Cloud Storage.
+> If you're planning to use the DataflowRunner, you can skip this step. The
+ runner will pull text directly from Google Cloud Storage.
1. In the **word-count-beam** directory, create a file called **sample.txt**.
-1. Add some text to the file. For this example, you can use the text of Shakespeare's [Sonnets](https://storage.cloud.google.com/apache-beam-samples/shakespeare/sonnets.txt).
+2. Add some text to the file. For this example, use the text of Shakespeare's
+ [King Lear](https://storage.cloud.google.com/apache-beam-samples/shakespeare/kinglear.txt).
## Run a pipeline
-A single Beam pipeline can run on multiple Beam [runners](/documentation#runners), including the [FlinkRunner](/documentation/runners/flink), [SparkRunner](/documentation/runners/spark), [NemoRunner](/documentation/runners/nemo), [JetRunner](/documentation/runners/jet), or [DataflowRunner](/documentation/runners/dataflow). The [DirectRunner](/documentation/runners/direct) is a common runner for getting started, as it runs locally on your machine and requires no specific setup. If you're just trying out Beam and you're not sure what to use, use the [DirectRunner](/documentation/runners/direct).
+A single Beam pipeline can run on multiple Beam
+[runners](/documentation#runners), including the
+[FlinkRunner](/documentation/runners/flink),
+[SparkRunner](/documentation/runners/spark),
+[NemoRunner](/documentation/runners/nemo),
+[JetRunner](/documentation/runners/jet), and
+[DataflowRunner](/documentation/runners/dataflow). The
+[DirectRunner](/documentation/runners/direct) is useful for getting started,
+because it runs on your machine and requires no specific setup. If you're just
+trying out Beam and you're not sure what to use, use the
+[DirectRunner](/documentation/runners/direct).
The general process for running a pipeline goes like this:
-1. Ensure you've done any runner-specific setup.
-1. Build your command line:
- 1. Specify a runner with `--runner=<runner>` (defaults to the [DirectRunner](/documentation/runners/direct)).
- 1. Add any runner-specific required options.
- 1. Choose input files and an output location that are accessible to the runner. (For example, you can't access a local file if you are running the pipeline on an external cluster.)
-1. Run the command.
+1. Complete any runner-specific setup.
+2. Build your command line:
+ 1. Specify a runner with `--runner=<runner>` (defaults to the
+ [DirectRunner](/documentation/runners/direct)).
+ 2. Add any runner-specific required options.
+ 3. Choose input files and an output location that are accessible to the
+ runner. (For example, you can't access a local file if you are running
+ the pipeline on an external cluster.)
+3. Run the command.
To run the WordCount pipeline, see the Maven and Gradle examples below.
-### Run WordCount Using Maven
+### Run WordCount using Maven
For Unix shells:
{{< runner direct >}}
-$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
- -Dexec.args="--inputFile=sample.txt --output=counts" -Pdirect-runner
+mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="--inputFile=sample.txt --output=counts" -Pdirect-runner
{{< /runner >}}
-
{{< runner flink >}}
-$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
- -Dexec.args="--runner=FlinkRunner --inputFile=sample.txt --output=counts" -Pflink-runner
+mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="--runner=FlinkRunner --inputFile=sample.txt --output=counts" -Pflink-runner
{{< /runner >}}
-
{{< runner flinkCluster >}}
-$ mvn package exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
- -Dexec.args="--runner=FlinkRunner --flinkMaster=<flink master> --filesToStage=target/word-count-beam-bundled-0.1.jar \
- --inputFile=sample.txt --output=/tmp/counts" -Pflink-runner
-
-You can monitor the running job by visiting the Flink dashboard at http://<flink master>:8081
Review comment:
I can't think of a way to do it -- I'm open to suggestions. The problem is that the syntax is copyable by a button in the docs. If we add a comment to the runnable command line, and the user copies it, I think it might cause problems. For example, if they try to use the up and down keys to go back to the last command executed. ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kileys merged pull request #17033: BEAM-12770: Documented maven-to-gradle conversion for Dataflow
Posted by GitBox <gi...@apache.org>.
kileys merged pull request #17033:
URL: https://github.com/apache/beam/pull/17033
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pcoet commented on a change in pull request #17033: BEAM-12770: Documented maven-to-gradle conversion for Dataflow
Posted by GitBox <gi...@apache.org>.
pcoet commented on a change in pull request #17033:
URL: https://github.com/apache/beam/pull/17033#discussion_r826478855
##########
File path: website/www/site/content/en/get-started/quickstart-java.md
##########
@@ -19,462 +19,376 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache Beam Java SDK Quickstart
+# Apache Beam Java SDK quickstart
-This quickstart shows you how to set up a Java development environment and run an [example pipeline](/get-started/wordcount-example) written with the [Apache Beam Java SDK](/documentation/sdks/java), using a [runner](/documentation#runners) of your choice.
+This quickstart shows you how to set up a Java development environment and run
+an [example pipeline](/get-started/wordcount-example) written with the
+[Apache Beam Java SDK](/documentation/sdks/java), using a
+[runner](/documentation#runners) of your choice.
-If you're interested in contributing to the Apache Beam Java codebase, see the [Contribution Guide](/contribute).
+If you're interested in contributing to the Apache Beam Java codebase, see the
+[Contribution Guide](/contribute).
-{{< toc >}}
+On this page:
-## Set up your Development Environment
-
-1. Download and install the [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html) version 8. Verify that the [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html) environment variable is set and points to your JDK installation.
-
-1. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by following Maven's [installation guide](https://maven.apache.org/install.html) for your specific operating system.
-
-1. Optional: Install [Gradle](https://gradle.org/install/) if you would like to convert your Maven project into Gradle.
-
-## Get the Example Code
-
-Use the following command to generate a Maven project that contains Beam's WordCount examples and builds against the most recent Beam release:
-
-{{< shell unix >}}
-$ mvn archetype:generate \
- -DarchetypeGroupId=org.apache.beam \
- -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
- -DarchetypeVersion={{< param release_latest >}} \
- -DgroupId=org.example \
- -DartifactId=word-count-beam \
- -Dversion="0.1" \
- -Dpackage=org.apache.beam.examples \
- -DinteractiveMode=false
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> mvn archetype:generate `
- -D archetypeGroupId=org.apache.beam `
- -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
- -D archetypeVersion={{< param release_latest >}} `
- -D groupId=org.example `
- -D artifactId=word-count-beam `
- -D version="0.1" `
- -D package=org.apache.beam.examples `
- -D interactiveMode=false
-{{< /shell >}}
-
-This will create a `word-count-beam` directory that contains a `pom.xml` and several example pipelines that count words in text files.
-
-{{< shell unix >}}
-$ cd word-count-beam/
-
-$ ls
-pom.xml src
-
-$ ls src/main/java/org/apache/beam/examples/
-DebuggingWordCount.java WindowedWordCount.java common
-MinimalWordCount.java WordCount.java
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> cd .\word-count-beam
-
-PS> dir
-
-...
-
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM src
--a---- 7/19/2018 11:00 PM 16051 pom.xml
-
-PS> dir .\src\main\java\org\apache\beam\examples
-
-...
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM common
-d----- 7/19/2018 11:00 PM complete
-d----- 7/19/2018 11:00 PM subprocess
--a---- 7/19/2018 11:00 PM 7073 DebuggingWordCount.java
--a---- 7/19/2018 11:00 PM 5945 MinimalWordCount.java
--a---- 7/19/2018 11:00 PM 9490 WindowedWordCount.java
--a---- 7/19/2018 11:00 PM 7662 WordCount.java
-{{< /shell >}}
-
-For a detailed introduction to the Beam concepts used in these examples, see the [WordCount Example Walkthrough](/get-started/wordcount-example). Here, we'll just focus on executing `WordCount.java`.
-
-## Optional: Convert from Maven to Gradle Project
-
-The steps below explain how to convert the build for the Direct Runner from Maven to Gradle. Converting the builds for the other runners is a more involved process and is out of scope for this guide. For additional guidance, see [Migrating Builds From Apache Maven](https://docs.gradle.org/current/userguide/migrating_from_maven.html).
-
-1. Ensure you are in the same directory as the `pom.xml` file generated from the previous step. Automatically convert your project from Maven to Gradle by running:
-{{< highlight >}}
-$ gradle init
-{{< /highlight >}}
-You'll be asked if you want to generate a Gradle build. Enter **yes**. You'll also be prompted to choose a DSL (Groovy or Kotlin). This tutorial uses Groovy, so select that if you don't have a preference.
-1. After you've converted the project to Gradle, open the generated `build.gradle` file, and, in the `repositories` block, replace `mavenLocal()` with `mavenCentral()`:
-{{< highlight >}}
-repositories {
- mavenCentral()
- maven {
- url = uri('https://repository.apache.org/content/repositories/snapshots/')
- }
+{{< toc >}}
- maven {
- url = uri('http://repo.maven.apache.org/maven2')
+## Set up your development environment
+
+1. Download and install the
+ [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html)
+ version 8, 11, or 17. Verify that the
+ [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html)
+ environment variable is set and points to your JDK installation.
+2. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by
+ following the [installation guide](https://maven.apache.org/install.html)
+ for your operating system.
+3. Optional: If you want to convert your Maven project to Gradle, install
+ [Gradle](https://gradle.org/install/).
+
+## Get the example code
+
+1. Generate a Maven example project that builds against the latest Beam release:
+ {{< shell unix >}}
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.beam \
+ -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
+ -DarchetypeVersion={{< param release_latest >}} \
+ -DgroupId=org.example \
+ -DartifactId=word-count-beam \
+ -Dversion="0.1" \
+ -Dpackage=org.apache.beam.examples \
+ -DinteractiveMode=false
+ {{< /shell >}}
+ {{< shell powerShell >}}
+mvn archetype:generate `
+ -D archetypeGroupId=org.apache.beam `
+ -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
+ -D archetypeVersion={{< param release_latest >}} `
+ -D groupId=org.example `
+ -D artifactId=word-count-beam `
+ -D version="0.1" `
+ -D package=org.apache.beam.examples `
+ -D interactiveMode=false
+ {{< /shell >}}
+
+ Maven creates a new project in the **word-count-beam** directory.
+
+2. Change into **word-count-beam**:
+ {{< shell unix >}}
+cd word-count-beam/
+ {{< /shell >}}
+ {{< shell powerShell >}}
+cd .\word-count-beam
+ {{< /shell >}}
+ The directory contains a **pom.xml** and a **src** directory with example
+ pipelines.
+
+3. List the example pipelines:
+ {{< shell unix >}}
+ls src/main/java/org/apache/beam/examples/
+ {{< /shell >}}
+ {{< shell powerShell >}}
+dir .\src\main\java\org\apache\beam\examples
+ {{< /shell >}}
+ You should see the following examples:
+ * **DebuggingWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java))
+ * **MinimalWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java))
+ * **WindowedWordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java))
+ * **WordCount.java** ([GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WordCount.java))
+
+ The example used in this tutorial, **WordCount.java**, defines a
+ Beam pipeline that counts words from an input file (by default, a **.txt**
+ file containing Shakespeare's "King Lear"). To learn more about the examples,
+ see the [WordCount Example Walkthrough](/get-started/wordcount-example).
+
+## Optional: Convert from Maven to Gradle
+
+The steps below explain how to convert the build from Maven to Gradle for the
+following runners:
+* Direct runner
+* Dataflow runner
+
+The conversion process for other runners is similar. For additional guidance,
+see
+[Migrating Builds From Apache Maven](https://docs.gradle.org/current/userguide/migrating_from_maven.html).
+
+1. In the directory with the `pom.xml` file, run the automated Maven-to-Gradle
+ conversion:
+ {{< highlight >}}
+gradle init
+ {{< /highlight >}}
+ You'll be asked if you want to generate a Gradle build. Enter **yes**. You'll
+ also be prompted to choose a DSL (Groovy or Kotlin). For this tutorial, enter
+ **2** for Kotlin.
+2. Open the generated **build.gradle.kts** file and make the following changes:
+ 1. In `repositories`, replace `mavenLocal()` with `mavenCentral()`.
+ 2. In `repositories`, declare a repository for Confluent Kafka dependencies:
+ {{< highlight >}}
+maven {
+ url = uri("https://packages.confluent.io/maven/")
+}
+ {{< /highlight >}}
+ 3. At the end of the build script, add the following conditional dependency:
+ {{< highlight >}}
+if (project.hasProperty("dataflow-runner")) {
Review comment:
That's a good question. My thought was that we would add the other runner logic to this block, too, so that we have a single conversion process. For the user, it's just a single copy-paste. Otherwise we need a step for each runner, at which point we should not bother with the conditional logic (`if (project.hasProperty(...)`), because we're assuming that the user is only going to try one runner. But I'm inclined to follow the Maven model and put all the "profiles" here. Let me know if you disagree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pcoet commented on a change in pull request #17033: BEAM-12770: Documented maven-to-gradle conversion for Dataflow
Posted by GitBox <gi...@apache.org>.
pcoet commented on a change in pull request #17033:
URL: https://github.com/apache/beam/pull/17033#discussion_r826476030
##########
File path: website/www/site/content/en/get-started/quickstart-java.md
##########
@@ -19,462 +19,376 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache Beam Java SDK Quickstart
+# Apache Beam Java SDK quickstart
-This quickstart shows you how to set up a Java development environment and run an [example pipeline](/get-started/wordcount-example) written with the [Apache Beam Java SDK](/documentation/sdks/java), using a [runner](/documentation#runners) of your choice.
+This quickstart shows you how to set up a Java development environment and run
+an [example pipeline](/get-started/wordcount-example) written with the
+[Apache Beam Java SDK](/documentation/sdks/java), using a
+[runner](/documentation#runners) of your choice.
-If you're interested in contributing to the Apache Beam Java codebase, see the [Contribution Guide](/contribute).
+If you're interested in contributing to the Apache Beam Java codebase, see the
+[Contribution Guide](/contribute).
-{{< toc >}}
+On this page:
-## Set up your Development Environment
-
-1. Download and install the [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html) version 8. Verify that the [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html) environment variable is set and points to your JDK installation.
-
-1. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by following Maven's [installation guide](https://maven.apache.org/install.html) for your specific operating system.
-
-1. Optional: Install [Gradle](https://gradle.org/install/) if you would like to convert your Maven project into Gradle.
-
-## Get the Example Code
-
-Use the following command to generate a Maven project that contains Beam's WordCount examples and builds against the most recent Beam release:
-
-{{< shell unix >}}
-$ mvn archetype:generate \
- -DarchetypeGroupId=org.apache.beam \
- -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
- -DarchetypeVersion={{< param release_latest >}} \
- -DgroupId=org.example \
- -DartifactId=word-count-beam \
- -Dversion="0.1" \
- -Dpackage=org.apache.beam.examples \
- -DinteractiveMode=false
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> mvn archetype:generate `
- -D archetypeGroupId=org.apache.beam `
- -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
- -D archetypeVersion={{< param release_latest >}} `
- -D groupId=org.example `
- -D artifactId=word-count-beam `
- -D version="0.1" `
- -D package=org.apache.beam.examples `
- -D interactiveMode=false
-{{< /shell >}}
-
-This will create a `word-count-beam` directory that contains a `pom.xml` and several example pipelines that count words in text files.
-
-{{< shell unix >}}
-$ cd word-count-beam/
-
-$ ls
-pom.xml src
-
-$ ls src/main/java/org/apache/beam/examples/
-DebuggingWordCount.java WindowedWordCount.java common
-MinimalWordCount.java WordCount.java
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> cd .\word-count-beam
-
-PS> dir
-
-...
-
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM src
--a---- 7/19/2018 11:00 PM 16051 pom.xml
-
-PS> dir .\src\main\java\org\apache\beam\examples
-
-...
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM common
-d----- 7/19/2018 11:00 PM complete
-d----- 7/19/2018 11:00 PM subprocess
--a---- 7/19/2018 11:00 PM 7073 DebuggingWordCount.java
--a---- 7/19/2018 11:00 PM 5945 MinimalWordCount.java
--a---- 7/19/2018 11:00 PM 9490 WindowedWordCount.java
--a---- 7/19/2018 11:00 PM 7662 WordCount.java
-{{< /shell >}}
-
-For a detailed introduction to the Beam concepts used in these examples, see the [WordCount Example Walkthrough](/get-started/wordcount-example). Here, we'll just focus on executing `WordCount.java`.
-
-## Optional: Convert from Maven to Gradle Project
-
-The steps below explain how to convert the build for the Direct Runner from Maven to Gradle. Converting the builds for the other runners is a more involved process and is out of scope for this guide. For additional guidance, see [Migrating Builds From Apache Maven](https://docs.gradle.org/current/userguide/migrating_from_maven.html).
-
-1. Ensure you are in the same directory as the `pom.xml` file generated from the previous step. Automatically convert your project from Maven to Gradle by running:
-{{< highlight >}}
-$ gradle init
-{{< /highlight >}}
-You'll be asked if you want to generate a Gradle build. Enter **yes**. You'll also be prompted to choose a DSL (Groovy or Kotlin). This tutorial uses Groovy, so select that if you don't have a preference.
-1. After you've converted the project to Gradle, open the generated `build.gradle` file, and, in the `repositories` block, replace `mavenLocal()` with `mavenCentral()`:
-{{< highlight >}}
-repositories {
- mavenCentral()
- maven {
- url = uri('https://repository.apache.org/content/repositories/snapshots/')
- }
+{{< toc >}}
- maven {
- url = uri('http://repo.maven.apache.org/maven2')
+## Set up your development environment
+
+1. Download and install the
+ [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html)
+ version 8, 11, or 17. Verify that the
+ [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html)
+ environment variable is set and points to your JDK installation.
+2. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by
+ following the [installation guide](https://maven.apache.org/install.html)
+ for your operating system.
+3. Optional: If you want to convert your Maven project to Gradle, install
+ [Gradle](https://gradle.org/install/).
+
+## Get the example code
+
+1. Generate a Maven example project that builds against the latest Beam release:
+ {{< shell unix >}}
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.beam \
+ -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
+ -DarchetypeVersion={{< param release_latest >}} \
+ -DgroupId=org.example \
+ -DartifactId=word-count-beam \
+ -Dversion="0.1" \
+ -Dpackage=org.apache.beam.examples \
+ -DinteractiveMode=false
+ {{< /shell >}}
+ {{< shell powerShell >}}
+mvn archetype:generate `
+ -D archetypeGroupId=org.apache.beam `
+ -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
+ -D archetypeVersion={{< param release_latest >}} `
+ -D groupId=org.example `
+ -D artifactId=word-count-beam `
+ -D version="0.1" `
+ -D package=org.apache.beam.examples `
+ -D interactiveMode=false
+ {{< /shell >}}
+
+ Maven creates a new project in the **word-count-beam** directory.
+
+2. Change into **word-count-beam**:
+ {{< shell unix >}}
+cd word-count-beam/
+ {{< /shell >}}
+ {{< shell powerShell >}}
+cd .\word-count-beam
+ {{< /shell >}}
+ The directory contains a **pom.xml** and a **src** directory with example
Review comment:
We should use bold for filenames and the red text for code. There's a grey area for filenames in code... I'll fix the pom.xml in the next section. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pcoet commented on a change in pull request #17033: BEAM-12770: Documented maven-to-gradle conversion for Dataflow
Posted by GitBox <gi...@apache.org>.
pcoet commented on a change in pull request #17033:
URL: https://github.com/apache/beam/pull/17033#discussion_r826490951
##########
File path: website/www/site/content/en/get-started/quickstart-java.md
##########
@@ -19,462 +19,376 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache Beam Java SDK Quickstart
+# Apache Beam Java SDK quickstart
-This quickstart shows you how to set up a Java development environment and run an [example pipeline](/get-started/wordcount-example) written with the [Apache Beam Java SDK](/documentation/sdks/java), using a [runner](/documentation#runners) of your choice.
+This quickstart shows you how to set up a Java development environment and run
+an [example pipeline](/get-started/wordcount-example) written with the
+[Apache Beam Java SDK](/documentation/sdks/java), using a
+[runner](/documentation#runners) of your choice.
-If you're interested in contributing to the Apache Beam Java codebase, see the [Contribution Guide](/contribute).
+If you're interested in contributing to the Apache Beam Java codebase, see the
+[Contribution Guide](/contribute).
-{{< toc >}}
+On this page:
-## Set up your Development Environment
-
-1. Download and install the [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html) version 8. Verify that the [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html) environment variable is set and points to your JDK installation.
-
-1. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by following Maven's [installation guide](https://maven.apache.org/install.html) for your specific operating system.
-
-1. Optional: Install [Gradle](https://gradle.org/install/) if you would like to convert your Maven project into Gradle.
-
-## Get the Example Code
-
-Use the following command to generate a Maven project that contains Beam's WordCount examples and builds against the most recent Beam release:
-
-{{< shell unix >}}
-$ mvn archetype:generate \
- -DarchetypeGroupId=org.apache.beam \
- -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
- -DarchetypeVersion={{< param release_latest >}} \
- -DgroupId=org.example \
- -DartifactId=word-count-beam \
- -Dversion="0.1" \
- -Dpackage=org.apache.beam.examples \
- -DinteractiveMode=false
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> mvn archetype:generate `
- -D archetypeGroupId=org.apache.beam `
- -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
- -D archetypeVersion={{< param release_latest >}} `
- -D groupId=org.example `
- -D artifactId=word-count-beam `
- -D version="0.1" `
- -D package=org.apache.beam.examples `
- -D interactiveMode=false
-{{< /shell >}}
-
-This will create a `word-count-beam` directory that contains a `pom.xml` and several example pipelines that count words in text files.
-
-{{< shell unix >}}
-$ cd word-count-beam/
-
-$ ls
-pom.xml src
-
-$ ls src/main/java/org/apache/beam/examples/
-DebuggingWordCount.java WindowedWordCount.java common
-MinimalWordCount.java WordCount.java
-{{< /shell >}}
-
-{{< shell powerShell >}}
-PS> cd .\word-count-beam
-
-PS> dir
-
-...
-
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM src
--a---- 7/19/2018 11:00 PM 16051 pom.xml
-
-PS> dir .\src\main\java\org\apache\beam\examples
-
-...
-Mode LastWriteTime Length Name
----- ------------- ------ ----
-d----- 7/19/2018 11:00 PM common
-d----- 7/19/2018 11:00 PM complete
-d----- 7/19/2018 11:00 PM subprocess
--a---- 7/19/2018 11:00 PM 7073 DebuggingWordCount.java
--a---- 7/19/2018 11:00 PM 5945 MinimalWordCount.java
--a---- 7/19/2018 11:00 PM 9490 WindowedWordCount.java
--a---- 7/19/2018 11:00 PM 7662 WordCount.java
-{{< /shell >}}
-
-For a detailed introduction to the Beam concepts used in these examples, see the [WordCount Example Walkthrough](/get-started/wordcount-example). Here, we'll just focus on executing `WordCount.java`.
-
-## Optional: Convert from Maven to Gradle Project
-
-The steps below explain how to convert the build for the Direct Runner from Maven to Gradle. Converting the builds for the other runners is a more involved process and is out of scope for this guide. For additional guidance, see [Migrating Builds From Apache Maven](https://docs.gradle.org/current/userguide/migrating_from_maven.html).
-
-1. Ensure you are in the same directory as the `pom.xml` file generated from the previous step. Automatically convert your project from Maven to Gradle by running:
-{{< highlight >}}
-$ gradle init
-{{< /highlight >}}
-You'll be asked if you want to generate a Gradle build. Enter **yes**. You'll also be prompted to choose a DSL (Groovy or Kotlin). This tutorial uses Groovy, so select that if you don't have a preference.
-1. After you've converted the project to Gradle, open the generated `build.gradle` file, and, in the `repositories` block, replace `mavenLocal()` with `mavenCentral()`:
-{{< highlight >}}
-repositories {
- mavenCentral()
- maven {
- url = uri('https://repository.apache.org/content/repositories/snapshots/')
- }
+{{< toc >}}
- maven {
- url = uri('http://repo.maven.apache.org/maven2')
+## Set up your development environment
+
+1. Download and install the
+ [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html)
+ version 8, 11, or 17. Verify that the
+ [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html)
+ environment variable is set and points to your JDK installation.
+2. Download and install [Apache Maven](https://maven.apache.org/download.cgi) by
+ following the [installation guide](https://maven.apache.org/install.html)
+ for your operating system.
+3. Optional: If you want to convert your Maven project to Gradle, install
+ [Gradle](https://gradle.org/install/).
+
+## Get the example code
+
+1. Generate a Maven example project that builds against the latest Beam release:
+ {{< shell unix >}}
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.beam \
+ -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
+ -DarchetypeVersion={{< param release_latest >}} \
+ -DgroupId=org.example \
+ -DartifactId=word-count-beam \
+ -Dversion="0.1" \
+ -Dpackage=org.apache.beam.examples \
+ -DinteractiveMode=false
+ {{< /shell >}}
+ {{< shell powerShell >}}
+mvn archetype:generate `
+ -D archetypeGroupId=org.apache.beam `
+ -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
+ -D archetypeVersion={{< param release_latest >}} `
+ -D groupId=org.example `
+ -D artifactId=word-count-beam `
+ -D version="0.1" `
+ -D package=org.apache.beam.examples `
+ -D interactiveMode=false
+ {{< /shell >}}
+
+ Maven creates a new project in the **word-count-beam** directory.
+
+2. Change into **word-count-beam**:
+ {{< shell unix >}}
+cd word-count-beam/
+ {{< /shell >}}
+ {{< shell powerShell >}}
+cd .\word-count-beam
+ {{< /shell >}}
+ The directory contains a **pom.xml** and a **src** directory with example
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #17033: BEAM-12770: Documented maven-to-gradle conversion for Dataflow
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #17033:
URL: https://github.com/apache/beam/pull/17033#issuecomment-1067225296
@kileys - could you please review this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kileys commented on pull request #17033: BEAM-12770: Documented maven-to-gradle conversion for Dataflow
Posted by GitBox <gi...@apache.org>.
kileys commented on pull request #17033:
URL: https://github.com/apache/beam/pull/17033#issuecomment-1068336109
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org