You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/08/30 04:12:33 UTC
[GitHub] [beam] pulasthi opened a new pull request #12731: [BEAM-10830] Twister2 quickstarts and the runner maven archetypes
pulasthi opened a new pull request #12731:
URL: https://github.com/apache/beam/pull/12731
Adding changes to complete quickstart for Twister2 runner and adding runner maven archetypes
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace
--- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pulasthi commented on pull request #12731: [BEAM-10830] Twister2 quickstarts and the runner maven archetypes
Posted by GitBox <gi...@apache.org>.
pulasthi commented on pull request #12731:
URL: https://github.com/apache/beam/pull/12731#issuecomment-683374018
@iemejia Sorry about the delay in getting this out. I am not sure what triggers I need to call in order to validate these changes, would you be able to call them or point them out. Thanks in advance
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pulasthi commented on a change in pull request #12731: [BEAM-10830] Twister2 quickstarts and the runner maven archetypes
Posted by GitBox <gi...@apache.org>.
pulasthi commented on a change in pull request #12731:
URL: https://github.com/apache/beam/pull/12731#discussion_r494750703
##########
File path: runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2PipelineResult.java
##########
@@ -17,15 +17,21 @@
*/
package org.apache.beam.runners.twister2;
+import edu.iu.dsc.tws.api.scheduler.Twister2JobState;
import java.io.IOException;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.metrics.MetricResults;
import org.joda.time.Duration;
+import org.mortbay.log.Log;
Review comment:
@iemejia No, I will create separate PR that encapsulates all the changes you have mentioned. I will remove this log import
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12731: [BEAM-10830] Twister2 quickstarts and the runner maven archetypes
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12731:
URL: https://github.com/apache/beam/pull/12731#issuecomment-683376404
# [Codecov](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=h1) Report
> Merging [#12731](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/1d25e2ebeb4a0f74278dbd0cfbaa00f36abd73dc?el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12731 +/- ##
=======================================
Coverage 40.22% 40.22%
=======================================
Files 454 454
Lines 53669 53669
=======================================
Hits 21587 21587
Misses 32082 32082
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=footer). Last update [1d25e2e...8339055](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pulasthi commented on pull request #12731: [BEAM-10830] Twister2 quickstarts and the runner maven archetypes
Posted by GitBox <gi...@apache.org>.
pulasthi commented on pull request #12731:
URL: https://github.com/apache/beam/pull/12731#issuecomment-683373894
R: @iemejia
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12731: [BEAM-10830] Twister2 quickstarts and the runner maven archetypes
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12731:
URL: https://github.com/apache/beam/pull/12731#issuecomment-683376404
# [Codecov](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=h1) Report
> Merging [#12731](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/1d25e2ebeb4a0f74278dbd0cfbaa00f36abd73dc?el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12731 +/- ##
=======================================
Coverage 40.22% 40.22%
=======================================
Files 454 454
Lines 53669 53669
=======================================
Hits 21587 21587
Misses 32082 32082
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=footer). Last update [1d25e2e...8339055](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on a change in pull request #12731: [BEAM-10830] Twister2 quickstarts and the runner maven archetypes
Posted by GitBox <gi...@apache.org>.
iemejia commented on a change in pull request #12731:
URL: https://github.com/apache/beam/pull/12731#discussion_r487696110
##########
File path: runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2PipelineResult.java
##########
@@ -17,15 +17,21 @@
*/
package org.apache.beam.runners.twister2;
+import edu.iu.dsc.tws.api.scheduler.Twister2JobState;
import java.io.IOException;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.metrics.MetricResults;
import org.joda.time.Duration;
+import org.mortbay.log.Log;
Review comment:
Is there any particular reason to use a different logger kind here?
We use slf4j in most of the Beam code base for logging so maybe good to follow this on the Twister2 runner. Notice that this can be done in a separate PR.
##########
File path: sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
##########
@@ -281,6 +281,17 @@
</dependency>
</dependencies>
</profile>
+ <profile>
+ <id>twister2-runner</id>
Review comment:
:+1:
##########
File path: runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2Runner.java
##########
@@ -119,19 +129,27 @@ public PipelineResult run(Pipeline pipeline) {
.addComputeResource(options.getWorkerCPUs(), options.getRamMegaBytes(), workers)
.setConfig(jobConfig)
.build();
- Twister2JobState jobState = Twister2Submitter.submitJob(twister2Job, config);
- Twister2PipelineResult result = new Twister2PipelineResult();
- // TODO: Need to fix the check for "RUNNING" once fix for this is done on Twister2 end.
- if (jobState.getJobstate() == DriverJobState.FAILED
- || jobState.getJobstate() == DriverJobState.RUNNING) {
- throw new RuntimeException("Pipeline execution failed", jobState.getCause());
+ Twister2JobState jobState;
+ if (isLocalMode(options)) {
+ jobState = LocalSubmitter.submitJob(twister2Job, config);
} else {
- result.setState(PipelineResult.State.DONE);
+ jobState = Twister2Submitter.submitJob(twister2Job, config);
}
+
+ Twister2PipelineResult result = new Twister2PipelineResult(jobState);
return result;
}
+ /** Check if the Runner is set to use Twister local mode or pointing to a deployment. */
+ private boolean isLocalMode(Twister2PipelineOptions options) {
+ if (options.getTwister2Home() == null || "".equals(options.getTwister2Home())) {
Review comment:
with guava (use vendor if so) this can be simpler, but that's not blocking at all for this PR
`return Strings.isNullOrEmpty(options.getTwister2Home());`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pulasthi commented on a change in pull request #12731: [BEAM-10830] Twister2 quickstarts and the runner maven archetypes
Posted by GitBox <gi...@apache.org>.
pulasthi commented on a change in pull request #12731:
URL: https://github.com/apache/beam/pull/12731#discussion_r494750703
##########
File path: runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2PipelineResult.java
##########
@@ -17,15 +17,21 @@
*/
package org.apache.beam.runners.twister2;
+import edu.iu.dsc.tws.api.scheduler.Twister2JobState;
import java.io.IOException;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.metrics.MetricResults;
import org.joda.time.Duration;
+import org.mortbay.log.Log;
Review comment:
@iemejia No, I will create separate PR that encapsulates all the changes you have mentioned. I will remove this log import
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #12731: [BEAM-10830] Twister2 quickstarts and the runner maven archetypes
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #12731:
URL: https://github.com/apache/beam/pull/12731#issuecomment-692041712
Run Twister2 ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12731: [BEAM-10830] Twister2 quickstarts and the runner maven archetypes
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12731:
URL: https://github.com/apache/beam/pull/12731#issuecomment-683376404
# [Codecov](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=h1) Report
> Merging [#12731](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/1d25e2ebeb4a0f74278dbd0cfbaa00f36abd73dc?el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12731 +/- ##
=======================================
Coverage 40.22% 40.22%
=======================================
Files 454 454
Lines 53669 53669
=======================================
Hits 21587 21587
Misses 32082 32082
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=footer). Last update [1d25e2e...8339055](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #12731: [BEAM-10830] Twister2 quickstarts and the runner maven archetypes
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #12731:
URL: https://github.com/apache/beam/pull/12731#issuecomment-683376404
# [Codecov](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=h1) Report
> Merging [#12731](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/1d25e2ebeb4a0f74278dbd0cfbaa00f36abd73dc?el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12731 +/- ##
=======================================
Coverage 40.22% 40.22%
=======================================
Files 454 454
Lines 53669 53669
=======================================
Hits 21587 21587
Misses 32082 32082
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=footer). Last update [1d25e2e...8339055](https://codecov.io/gh/apache/beam/pull/12731?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia merged pull request #12731: [BEAM-10830] Twister2 quickstarts and the runner maven archetypes
Posted by GitBox <gi...@apache.org>.
iemejia merged pull request #12731:
URL: https://github.com/apache/beam/pull/12731
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org