You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/31 08:33:20 UTC
[GitHub] [beam] jbartok opened a new pull request #11273: Update Jet Runner
to Jet version 4.0
jbartok opened a new pull request #11273: Update Jet Runner to Jet version 4.0
URL: https://github.com/apache/beam/pull/11273
Hazelcast Jet recently released version 4.0 (https://jet-start.sh/blog/2020/03/02/jet-40-is-released) and this pull request updates Beam's Jet Runner to that version. It also extends the Runners web page with information about the 4.0 version.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm edited a comment on issue #11273: Update Jet Runner to
Jet version 4.0
Posted by GitBox <gi...@apache.org>.
mxm edited a comment on issue #11273: Update Jet Runner to Jet version 4.0
URL: https://github.com/apache/beam/pull/11273#issuecomment-606821680
Btw, there doesn't appear to be a Jenkins job (see .test-infra) for running the validates runner or other integration tests. Do you plan to add one?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] jbartok commented on issue #11273: Update Jet Runner to Jet
version 4.0
Posted by GitBox <gi...@apache.org>.
jbartok commented on issue #11273: Update Jet Runner to Jet version 4.0
URL: https://github.com/apache/beam/pull/11273#issuecomment-606480985
R: @mxm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] jbartok commented on a change in pull request #11273: Update
Jet Runner to Jet version 4.0
Posted by GitBox <gi...@apache.org>.
jbartok commented on a change in pull request #11273: Update Jet Runner to Jet version 4.0
URL: https://github.com/apache/beam/pull/11273#discussion_r400846449
##########
File path: website/src/documentation/runners/jet.md
##########
@@ -59,24 +59,58 @@ Issue following command in the Beam examples project to start new Jet cluster an
```
### Running WordCount on a Remote Jet Cluster ##
-Download latest stable Hazelcast Jet code from [Hazelcast Website](https://jet.hazelcast.org/download/) and
-start Jet cluster.
-The simplest way is to start Jet cluster member using the `jet-start` script that comes with Jet distribution.
-The members use the [auto discovery feature](https://docs.hazelcast.org/docs/3.12/manual/html-single/index.html#setting-up-clusters)
+Download latest stable Hazelcast Jet version, that Beam depends on, from
+[Hazelcast Jet Website](https://jet-start.sh/download). At the time of writing the
+[latest Beam release]({{site.baseurl}}/get-started/downloads/#releases) is 2.19.0, which still depends on Hazelcast
+Jet 3.0, but following versions should migrate to Jet 4.0.
Review comment:
Did so, much better indeed. Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] jbartok commented on issue #11273: [BEAM-9654] Update Jet
Runner to Jet version 4.0
Posted by GitBox <gi...@apache.org>.
jbartok commented on issue #11273: [BEAM-9654] Update Jet Runner to Jet version 4.0
URL: https://github.com/apache/beam/pull/11273#issuecomment-607132404
Question: the runner's page (https://beam.apache.org/documentation/runners/jet/) has an "Adapt for" button, to change content between Jet 3.0 and Jet 4.0. It worked perfectly in a locally served website, but looks like it doesn't in the deployed one.... Any idea why?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] jbartok commented on issue #11273: [BEAM-9654] Update Jet
Runner to Jet version 4.0
Posted by GitBox <gi...@apache.org>.
jbartok commented on issue #11273: [BEAM-9654] Update Jet Runner to Jet version 4.0
URL: https://github.com/apache/beam/pull/11273#issuecomment-607134043
Yeah, works in the merged in version too, locally, but what ended up deployed is broken... No idea why.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] jbartok commented on issue #11273: [BEAM-9654] Update Jet
Runner to Jet version 4.0
Posted by GitBox <gi...@apache.org>.
jbartok commented on issue #11273: [BEAM-9654] Update Jet Runner to Jet version 4.0
URL: https://github.com/apache/beam/pull/11273#issuecomment-607170840
Yeah, now that I've flushed my browser cash works indeed... Thanks for checking.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11273: Update Jet
Runner to Jet version 4.0
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11273: Update Jet Runner to Jet version 4.0
URL: https://github.com/apache/beam/pull/11273#discussion_r400811820
##########
File path: website/src/documentation/runners/jet.md
##########
@@ -59,24 +59,58 @@ Issue following command in the Beam examples project to start new Jet cluster an
```
### Running WordCount on a Remote Jet Cluster ##
-Download latest stable Hazelcast Jet code from [Hazelcast Website](https://jet.hazelcast.org/download/) and
-start Jet cluster.
-The simplest way is to start Jet cluster member using the `jet-start` script that comes with Jet distribution.
-The members use the [auto discovery feature](https://docs.hazelcast.org/docs/3.12/manual/html-single/index.html#setting-up-clusters)
+Download latest stable Hazelcast Jet version, that Beam depends on, from
+[Hazelcast Jet Website](https://jet-start.sh/download). At the time of writing the
+[latest Beam release]({{site.baseurl}}/get-started/downloads/#releases) is 2.19.0, which still depends on Hazelcast
+Jet 3.0, but following versions should migrate to Jet 4.0.
Review comment:
Should we include a version table here similar to the Flink Runner page? I would not hardcode anything regarding the Beam release at the time of the writing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11273: Update Jet Runner to Jet
version 4.0
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11273: Update Jet Runner to Jet version 4.0
URL: https://github.com/apache/beam/pull/11273#issuecomment-607096048
I think it would be helpful to compare the jet runner with other runners. If there is already a Gradle target, it is very easy to do. The Jenkins job dsl scripts in `.test-infra/jenkins` automatically create the Jenkins job. You can see how they work if you just look at the ones for Flink or other runners.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm merged pull request #11273: Update Jet Runner to Jet
version 4.0
Posted by GitBox <gi...@apache.org>.
mxm merged pull request #11273: Update Jet Runner to Jet version 4.0
URL: https://github.com/apache/beam/pull/11273
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11273: Update Jet Runner to Jet
version 4.0
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11273: Update Jet Runner to Jet version 4.0
URL: https://github.com/apache/beam/pull/11273#issuecomment-607096987
Btw, the JIRA is missing, I've created it here: https://jira.apache.org/jira/browse/BEAM-9654
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11273: [BEAM-9654] Update Jet Runner
to Jet version 4.0
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11273: [BEAM-9654] Update Jet Runner to Jet version 4.0
URL: https://github.com/apache/beam/pull/11273#issuecomment-607143548
Works for me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11273: Update Jet Runner to Jet
version 4.0
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11273: Update Jet Runner to Jet version 4.0
URL: https://github.com/apache/beam/pull/11273#issuecomment-606821680
Btw, there doesn't appear to be a Jenkins job (see .testinfra) for running the validates runner or other integration tests. Do you plan to add one?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] jbartok commented on issue #11273: Update Jet Runner to Jet
version 4.0
Posted by GitBox <gi...@apache.org>.
jbartok commented on issue #11273: Update Jet Runner to Jet version 4.0
URL: https://github.com/apache/beam/pull/11273#issuecomment-607054819
> Btw, there doesn't appear to be a Jenkins job (see .test-infra) for running the validates runner or other integration tests. Do you plan to add one?
I don't plan to add one, because I'm completely unfamiliar with the whole topic. I do have some Gradle targets in the Jet Runner module to execute @ValidatesRunner & @NeedsRunner tests with it, but that's it. I did check those, just as many pass now as before the upgrade.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services