You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2019/04/01 17:01:00 UTC
[jira] [Work logged] (BEAM-6955) Support Dataflow --sdk_location
with modified version number
[ https://issues.apache.org/jira/browse/BEAM-6955?focusedWorklogId=221356&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221356 ]
ASF GitHub Bot logged work on BEAM-6955:
----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Apr/19 17:00
Start Date: 01/Apr/19 17:00
Worklog Time Spent: 10m
Work Description: dlesco commented on pull request #8189: [BEAM-6955] Support Dataflow --sdk_location with modified version number
URL: https://github.com/apache/beam/pull/8189
Determine the version tag to use for the Google Container Registry,
for the service image versions to use on the Dataflow worker nodes.
Users of Dataflow may be using a locally-modified version of
Apache Beam, which they submit to Dataflow with the
--sdk_location option. Those users would most likely modify the
version number of Apache Beam, so they can distinguish it from the
public distribution of Apache Beam.
However, the remote nodes in Dataflow still need to bootsrap the
worker service with a Docker image that a version tag exists for.
The most appropriate way for system integrators to modify the
Apache Beam version number would be to add a Local Version Identifier:
https://www.python.org/dev/peps/pep-0440/#local-version-identifiers
If people only use Local Version Identifiers, then we could use
the "public" attribute of the pkg_resources version object.
If people instead use a post-release version identifier:
https://www.python.org/dev/peps/pep-0440/#post-releases
then only the "base_version" attribute would work both of these
version number changes.
Since Dataflow documentation does not specify how to modify
version numbers, I am choosing to use "base_version" attribute.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) <br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 221356)
Time Spent: 10m
Remaining Estimate: 0h
> Support Dataflow --sdk_location with modified version number
> ------------------------------------------------------------
>
> Key: BEAM-6955
> URL: https://issues.apache.org/jira/browse/BEAM-6955
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Affects Versions: 2.11.0
> Reporter: Daniel Lescohier
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Support Dataflow --sdk_location with modified version number
> Determine the version tag to use for the Google Container Registry, for the service image versions to use on the Dataflow worker nodes. Users of Dataflow may be using a locally-modified version of Apache Beam, which they submit to Dataflow with the --sdk_location option. Those users would most likely modify the version number of Apache Beam, so they can distinguish it from the public distribution of Apache Beam. However, the remote nodes in Dataflow still need to bootsrap the worker service with a Docker image that a version tag exists for.
> The most appropriate way for system integrators to modify the Apache Beam version number would be to add a Local Version Identifier: https://www.python.org/dev/peps/pep-0440/#local-version-identifiers
> If people only use Local Version Identifiers, then we could use the "public" attribute of the pkg_resources version object.
> If people instead use a post-release version identifier: https://www.python.org/dev/peps/pep-0440/#post-releases then only the "base_version" attribute would work both of these version number changes.
> Since Dataflow documentation does not specify how to modify version numbers, I am choosing to use "base_version" attribute.
> Will shortly submit a PR with the change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)