You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/09/10 18:20:10 UTC
[GitHub] [beam] leiyiz opened a new pull request #12813: write to file ability for java suite
leiyiz opened a new pull request #12813:
URL: https://github.com/apache/beam/pull/12813
if the pubsubmode is set to PUBLISH and the runner is DIRECT, it would write the generated events to file
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] leiyiz commented on a change in pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
leiyiz commented on a change in pull request #12813:
URL: https://github.com/apache/beam/pull/12813#discussion_r487267868
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
maybe `generateEventFilePathPrefix` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] leiyiz commented on a change in pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
leiyiz commented on a change in pull request #12813:
URL: https://github.com/apache/beam/pull/12813#discussion_r486813312
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
sorry that I might have came out wrong, the logic later down the line is that if the source is null, following steps of pipeline will not get added.
see [this part](https://github.com/apache/beam/blob/009677318246ff5f0ff8370d8f3bf4ca40322343/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java#L1176)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on a change in pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
y1chi commented on a change in pull request #12813:
URL: https://github.com/apache/beam/pull/12813#discussion_r486661742
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
is this needed?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If {@literal true}, only generate events and write them to local file. */
+ @JsonProperty public boolean generateInputFileOnly = false;
Review comment:
It may help to allow user to specify a file path
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on a change in pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
y1chi commented on a change in pull request #12813:
URL: https://github.com/apache/beam/pull/12813#discussion_r487171952
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
generateInputFilePathPrefix to be explicit?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on a change in pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
y1chi commented on a change in pull request #12813:
URL: https://github.com/apache/beam/pull/12813#discussion_r486666964
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
this maybe the wrong place to put the branch, can't we swap out the steps after the source?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] leiyiz commented on pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
leiyiz commented on pull request #12813:
URL: https://github.com/apache/beam/pull/12813#issuecomment-690756599
@y1chi
@pabloem
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] leiyiz commented on a change in pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
leiyiz commented on a change in pull request #12813:
URL: https://github.com/apache/beam/pull/12813#discussion_r486767972
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If {@literal true}, only generate events and write them to local file. */
+ @JsonProperty public boolean generateInputFileOnly = false;
Review comment:
Done
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
sorry that I might have came out wrong, the logic later down the line is that if the source is null, following steps of pipeline will not get added.
see [this part](https://github.com/apache/beam/blob/009677318246ff5f0ff8370d8f3bf4ca40322343/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java#L1176)
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
maybe `generateEventFilePathPrefix` ?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
Done
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If {@literal true}, only generate events and write them to local file. */
+ @JsonProperty public boolean generateInputFileOnly = false;
Review comment:
Done
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
sorry that I might have came out wrong, the logic later down the line is that if the source is null, following steps of pipeline will not get added.
see [this part](https://github.com/apache/beam/blob/009677318246ff5f0ff8370d8f3bf4ca40322343/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java#L1176)
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
maybe `generateEventFilePathPrefix` ?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
Done
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If {@literal true}, only generate events and write them to local file. */
+ @JsonProperty public boolean generateInputFileOnly = false;
Review comment:
Done
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
sorry that I might have came out wrong, the logic later down the line is that if the source is null, following steps of pipeline will not get added.
see [this part](https://github.com/apache/beam/blob/009677318246ff5f0ff8370d8f3bf4ca40322343/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java#L1176)
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
maybe `generateEventFilePathPrefix` ?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on a change in pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
y1chi commented on a change in pull request #12813:
URL: https://github.com/apache/beam/pull/12813#discussion_r487276993
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
SGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] leiyiz commented on a change in pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
leiyiz commented on a change in pull request #12813:
URL: https://github.com/apache/beam/pull/12813#discussion_r486767972
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If {@literal true}, only generate events and write them to local file. */
+ @JsonProperty public boolean generateInputFileOnly = false;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] leiyiz commented on a change in pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
leiyiz commented on a change in pull request #12813:
URL: https://github.com/apache/beam/pull/12813#discussion_r486767972
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If {@literal true}, only generate events and write them to local file. */
+ @JsonProperty public boolean generateInputFileOnly = false;
Review comment:
Done
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
sorry that I might have came out wrong, the logic later down the line is that if the source is null, following steps of pipeline will not get added.
see [this part](https://github.com/apache/beam/blob/009677318246ff5f0ff8370d8f3bf4ca40322343/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java#L1176)
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
maybe `generateEventFilePathPrefix` ?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
Done
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
sorry that I might have came out wrong, the logic later down the line is that if the source is null, following steps of pipeline will not get added.
see [this part](https://github.com/apache/beam/blob/009677318246ff5f0ff8370d8f3bf4ca40322343/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java#L1176)
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
maybe `generateEventFilePathPrefix` ?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
Done
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If {@literal true}, only generate events and write them to local file. */
+ @JsonProperty public boolean generateInputFileOnly = false;
Review comment:
Done
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
sorry that I might have came out wrong, the logic later down the line is that if the source is null, following steps of pipeline will not get added.
see [this part](https://github.com/apache/beam/blob/009677318246ff5f0ff8370d8f3bf4ca40322343/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java#L1176)
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
maybe `generateEventFilePathPrefix` ?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
Done
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If {@literal true}, only generate events and write them to local file. */
+ @JsonProperty public boolean generateInputFileOnly = false;
Review comment:
Done
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
sorry that I might have came out wrong, the logic later down the line is that if the source is null, following steps of pipeline will not get added.
see [this part](https://github.com/apache/beam/blob/009677318246ff5f0ff8370d8f3bf4ca40322343/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java#L1176)
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
maybe `generateEventFilePathPrefix` ?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
Done
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If {@literal true}, only generate events and write them to local file. */
+ @JsonProperty public boolean generateInputFileOnly = false;
Review comment:
Done
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
sorry that I might have came out wrong, the logic later down the line is that if the source is null, following steps of pipeline will not get added.
see [this part](https://github.com/apache/beam/blob/009677318246ff5f0ff8370d8f3bf4ca40322343/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java#L1176)
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If {@literal true}, only generate events and write them to local file. */
+ @JsonProperty public boolean generateInputFileOnly = false;
Review comment:
Done
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
sorry that I might have came out wrong, the logic later down the line is that if the source is null, following steps of pipeline will not get added.
see [this part](https://github.com/apache/beam/blob/009677318246ff5f0ff8370d8f3bf4ca40322343/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java#L1176)
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
maybe `generateEventFilePathPrefix` ?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
Done
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If {@literal true}, only generate events and write them to local file. */
+ @JsonProperty public boolean generateInputFileOnly = false;
Review comment:
Done
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
sorry that I might have came out wrong, the logic later down the line is that if the source is null, following steps of pipeline will not get added.
see [this part](https://github.com/apache/beam/blob/009677318246ff5f0ff8370d8f3bf4ca40322343/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java#L1176)
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
maybe `generateEventFilePathPrefix` ?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] leiyiz commented on a change in pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
leiyiz commented on a change in pull request #12813:
URL: https://github.com/apache/beam/pull/12813#discussion_r487285561
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on a change in pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
y1chi commented on a change in pull request #12813:
URL: https://github.com/apache/beam/pull/12813#discussion_r487169637
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
I see, sounds good.
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
generateInputFilePathPrefix to be explicit?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
SGTM
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
I see, sounds good.
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
generateInputFilePathPrefix to be explicit?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
SGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #12813:
URL: https://github.com/apache/beam/pull/12813
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on a change in pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
y1chi commented on a change in pull request #12813:
URL: https://github.com/apache/beam/pull/12813#discussion_r487169637
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
I see, sounds good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on a change in pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
y1chi commented on a change in pull request #12813:
URL: https://github.com/apache/beam/pull/12813#discussion_r486666964
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
this maybe the wrong place to put the transform, can't we swap out the steps after the source?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12813:
URL: https://github.com/apache/beam/pull/12813#issuecomment-692116395
this LGTM. Waiting for @y1chi if he has any comments : )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] leiyiz commented on a change in pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
leiyiz commented on a change in pull request #12813:
URL: https://github.com/apache/beam/pull/12813#discussion_r486663750
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
yeah, if the source is not set to null then the actual query body will get executed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on a change in pull request #12813: write to file ability for java suite
Posted by GitBox <gi...@apache.org>.
y1chi commented on a change in pull request #12813:
URL: https://github.com/apache/beam/pull/12813#discussion_r487169637
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
I see, sounds good.
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
generateInputFilePathPrefix to be explicit?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
SGTM
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
I see, sounds good.
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
generateInputFilePathPrefix to be explicit?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
SGTM
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
I see, sounds good.
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
generateInputFilePathPrefix to be explicit?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
SGTM
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
I see, sounds good.
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
generateInputFilePathPrefix to be explicit?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
SGTM
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
I see, sounds good.
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
generateInputFilePathPrefix to be explicit?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
SGTM
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
I see, sounds good.
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
generateInputFilePathPrefix to be explicit?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
SGTM
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
##########
@@ -922,6 +922,11 @@ private void setupPubSubResources(long now) throws IOException {
switch (configuration.sourceType) {
case DIRECT:
source = sourceEventsFromSynthetic(p);
+ if (configuration.generateInputFileOnly) {
+ PCollection<Event> events = source;
+ source = null;
Review comment:
I see, sounds good.
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
generateInputFilePathPrefix to be explicit?
##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
##########
@@ -42,6 +42,9 @@
/** Where events come from. */
@JsonProperty public NexmarkUtils.SourceType sourceType = NexmarkUtils.SourceType.DIRECT;
+ /** If provided, only generate events and write them to local file with this prefix. */
+ @JsonProperty public String generateInputFileOnlyPrefix = null;
Review comment:
SGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org