You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/25 17:39:02 UTC
[GitHub] [beam] robertwb opened a new pull request #11221: Add base SDK
version to environment capabilities for Python and Java.
robertwb opened a new pull request #11221: Add base SDK version to environment capabilities for Python and Java.
URL: https://github.com/apache/beam/pull/11221
Go doesn't seem to meaningfully provide this information yet
(always using latest unless manually specified).
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on issue #11221: Add base SDK version to
environment capabilities for Python and Java.
Posted by GitBox <gi...@apache.org>.
robertwb commented on issue #11221: Add base SDK version to environment capabilities for Python and Java.
URL: https://github.com/apache/beam/pull/11221#issuecomment-604265227
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on a change in pull request #11221: Add
base SDK version to environment capabilities for Python and Java.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #11221: Add base SDK version to environment capabilities for Python and Java.
URL: https://github.com/apache/beam/pull/11221#discussion_r398654663
##########
File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##########
@@ -239,6 +239,7 @@ public static Environment createProcessEnvironment(
capabilities.addAll(ModelCoders.urns());
capabilities.add(BeamUrns.getUrn(StandardProtocols.Enum.MULTI_CORE_BUNDLE_PROCESSING));
capabilities.add(BeamUrns.getUrn(StandardProtocols.Enum.PROGRESS_REPORTING));
+ capabilities.add("beam:version:sdk_base:" + JAVA_SDK_HARNESS_CONTAINER_URL);
Review comment:
What about going with `"beam:sdk:version:" + ReleaseInfo.getReleaseInfo().getSdkVersion()` and `"beam:sdk:language:java"`?
Any detection we do about container image paths should likely be done from the container manifest itself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ananvay commented on issue #11221: Add base SDK version to
environment capabilities for Python and Java.
Posted by GitBox <gi...@apache.org>.
ananvay commented on issue #11221: Add base SDK version to environment capabilities for Python and Java.
URL: https://github.com/apache/beam/pull/11221#issuecomment-604022464
/cc: @lostluck for tracking the addition of this to Go SDK later.
LGTM. Thanks Robert!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on issue #11221: Add base SDK version to
environment capabilities for Python and Java.
Posted by GitBox <gi...@apache.org>.
robertwb commented on issue #11221: Add base SDK version to environment capabilities for Python and Java.
URL: https://github.com/apache/beam/pull/11221#issuecomment-603983689
R: @ananvay
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on issue #11221: Add base SDK version to
environment capabilities for Python and Java.
Posted by GitBox <gi...@apache.org>.
robertwb commented on issue #11221: Add base SDK version to environment capabilities for Python and Java.
URL: https://github.com/apache/beam/pull/11221#issuecomment-604145371
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on a change in pull request #11221:
[BEAM-9614] Add base SDK version to environment capabilities for Python,
Java and initial value for Go
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #11221: [BEAM-9614] Add base SDK version to environment capabilities for Python, Java and initial value for Go
URL: https://github.com/apache/beam/pull/11221#discussion_r398817446
##########
File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##########
@@ -239,6 +239,7 @@ public static Environment createProcessEnvironment(
capabilities.addAll(ModelCoders.urns());
capabilities.add(BeamUrns.getUrn(StandardProtocols.Enum.MULTI_CORE_BUNDLE_PROCESSING));
capabilities.add(BeamUrns.getUrn(StandardProtocols.Enum.PROGRESS_REPORTING));
+ capabilities.add("beam:version:sdk_base:" + JAVA_SDK_HARNESS_CONTAINER_URL);
Review comment:
Yeah, we can't rule out people trying to parse these, but at least then they'll know they're on shaky ground. And as you say, these aren't terribly sticky. (I hope we never have to use them.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on a change in pull request #11221: Add
base SDK version to environment capabilities for Python and Java.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #11221: Add base SDK version to environment capabilities for Python and Java.
URL: https://github.com/apache/beam/pull/11221#discussion_r398774739
##########
File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##########
@@ -239,6 +239,7 @@ public static Environment createProcessEnvironment(
capabilities.addAll(ModelCoders.urns());
capabilities.add(BeamUrns.getUrn(StandardProtocols.Enum.MULTI_CORE_BUNDLE_PROCESSING));
capabilities.add(BeamUrns.getUrn(StandardProtocols.Enum.PROGRESS_REPORTING));
+ capabilities.add("beam:version:sdk_base:" + JAVA_SDK_HARNESS_CONTAINER_URL);
Review comment:
The language version is an interesting point and brings up an issue where you could be using the same base container but the user submitting the pipeline is using a different version.
I just feel as though people will attempt to "parse" the language and version out of the container URL but if it comes to that we can always add more capabilities and remove existing ones.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on a change in pull request #11221: Add
base SDK version to environment capabilities for Python and Java.
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #11221: Add base SDK version to environment capabilities for Python and Java.
URL: https://github.com/apache/beam/pull/11221#discussion_r398735979
##########
File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##########
@@ -239,6 +239,7 @@ public static Environment createProcessEnvironment(
capabilities.addAll(ModelCoders.urns());
capabilities.add(BeamUrns.getUrn(StandardProtocols.Enum.MULTI_CORE_BUNDLE_PROCESSING));
capabilities.add(BeamUrns.getUrn(StandardProtocols.Enum.PROGRESS_REPORTING));
+ capabilities.add("beam:version:sdk_base:" + JAVA_SDK_HARNESS_CONTAINER_URL);
Review comment:
I did this because the languages might be versioned as well (as they are in Python, and may be in Java). This version may not align precisely with the version used during construction, but the default container should always be compatible with it to whatever level of granularity our containers are build. I also don't think we should be encourage any branching on `beam:sdk:version`. However, if you feel strongly we could split these out.
Adding a generic marker for go.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on issue #11221: Add base SDK version to
environment capabilities for Python and Java.
Posted by GitBox <gi...@apache.org>.
robertwb commented on issue #11221: Add base SDK version to environment capabilities for Python and Java.
URL: https://github.com/apache/beam/pull/11221#issuecomment-604145339
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb merged pull request #11221: [BEAM-9614] Add base
SDK version to environment capabilities for Python,
Java and initial value for Go
Posted by GitBox <gi...@apache.org>.
robertwb merged pull request #11221: [BEAM-9614] Add base SDK version to environment capabilities for Python, Java and initial value for Go
URL: https://github.com/apache/beam/pull/11221
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on a change in pull request #11221: Add
base SDK version to environment capabilities for Python and Java.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #11221: Add base SDK version to environment capabilities for Python and Java.
URL: https://github.com/apache/beam/pull/11221#discussion_r398654663
##########
File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##########
@@ -239,6 +239,7 @@ public static Environment createProcessEnvironment(
capabilities.addAll(ModelCoders.urns());
capabilities.add(BeamUrns.getUrn(StandardProtocols.Enum.MULTI_CORE_BUNDLE_PROCESSING));
capabilities.add(BeamUrns.getUrn(StandardProtocols.Enum.PROGRESS_REPORTING));
+ capabilities.add("beam:version:sdk_base:" + JAVA_SDK_HARNESS_CONTAINER_URL);
Review comment:
What about going with `"beam:sdk:version:" + ReleaseInfo.getReleaseInfo().getSdkVersion()` and `"beam:sdk:language:java"`?
Any detection we do about container image paths should likely be done from the container manifest itself.
We could add `beam:sdk:language:go` to the Go SDK.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services