You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Hannah Jiang <ha...@google.com> on 2019/08/27 22:34:55 UTC
Improve container support
Hi team
I am working on improving docker container support for Beam. We would like
to publish prebuilt containers for each release version and daily snapshot.
Current work focuses on release images only and it would be part of the
release process.
The release images will be pushed to GCR which is publicly
accessible(pullable). We will use the following locations.
*Repository*: gcr.io/beam
*Project*: apache-beam-testing
More details, including naming and tagging scheme, can be found at wiki
<https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process>
which
is written by several contributors.
I would like to discuss these two questions.
*1. How many tests do we need to run before pushing images to gcr*?
Publishing artifacts is the last step of the release process, so at this
moment, we already verified all codebase. In addition, many Jenkins tests
use containers, so it is already verified several times. Do we need to run
it again?
*2. How many tests do we need to run to validate pushed images?*
When we push the images, we assume the images would work and pass all the
tests. After pushing, we should confirm the images are pullable and
useable. I suggest we run several tests on dataflow with each pushed image.
What do you think?
This work can be refined later as we explore more during our release
process.
Please comment or edit the wiki page or reply to this email with your
opinions.
Thanks,
Hannah
Re: Improve container support
Posted by Robert Bradshaw <ro...@google.com>.
Good to know. I added an icon and link. Good pointer about trying to
see what we can do to make these official.
On Tue, Nov 19, 2019 at 11:16 AM Kyle Weaver <kc...@google.com> wrote:
>
> We do link from https://beam.apache.org/documentation/runtime/environments/.
>
> On Tue, Nov 19, 2019 at 10:59 AM Robert Bradshaw <ro...@google.com> wrote:
>>
>> We should probably add a link to these from our site as well, for visibility.
>>
>> On Tue, Nov 19, 2019 at 10:56 AM Kyle Weaver <kc...@google.com> wrote:
>> >
>> > +1 Thanks for bringing that up Chad, I had the same problem locating the docker images on Docker hub (searching "apachebeam" is the only way that seems to work).
>> >
>> > Another little thing I noticed is that https://hub.docker.com/u/apachebeam is mostly empty. It'd be good to at least add the Beam logo as a profile picture and a link to the website in the description.
>> >
>> > On Tue, Nov 19, 2019 at 10:21 AM Chad Dombrova <ch...@gmail.com> wrote:
>> >>
>> >> Hi all,
>> >> I think it might be good to update the description of the beam docker images and add some descriptive tags, because searching for "apache beam" in docker hub does not turn up anything: https://hub.docker.com/search?q=apache%20beam&type=image.
>> >>
>> >> I clicked through 10 pages worth and couldn't find it. Maybe I missed something, but it clearly shouldn't be this hard. I did eventually manage to find it through the docs.
>> >>
>> >> Also, googling "apache beam python docker" also does not yield anything useful. In fact, it turns up an unofficial apache beam docker hub image.
>> >>
>> >> One thing I noticed is that images designated as "Official Images" get listed first, so it would be good to get that done as well.
>> >>
>> >> thanks!
>> >> chad
>> >>
>> >>
>> >>
>> >>
>> >> On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang <ha...@google.com> wrote:
>> >>>
>> >>> Hi team
>> >>>
>> >>> I haven't received any objections, so will proceed with settings mentioned in a previous email.
>> >>>
>> >>> A reminder to PMC members, please let me know your docker hub id if you want to be an admin.
>> >>>
>> >>> Thanks,
>> >>> Hannah
>> >>>
>> >>> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka <go...@google.com> wrote:
>> >>>>
>> >>>> Please ignore the previous email. I was looking at the older document in the mail thread.
>> >>>>
>> >>>> On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka <go...@google.com> wrote:
>> >>>>>
>> >>>>> I think sdk in the name is obsolete as they are all under sdks name space.
>> >>>>>
>> >>>>> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang <ha...@google.com> wrote:
>> >>>>>>
>> >>>>>> Hi Team
>> >>>>>>
>> >>>>>> Thanks for all the comments about beam containers.
>> >>>>>> After considering various opinions and investigating gcr and docker hub, we decided to push images to docker hub.
>> >>>>>>
>> >>>>>> Each image will have two tags, {version}_rc and {version}. {version} tag will be added after the release candidate image is verified.
>> >>>>>> Meanwhile, we will have latest tag for each repository, which always points to the most recent verified release image, so users can pull it by default.
>> >>>>>>
>> >>>>>> Docker hub doesn't support leveled repository, which means we should follow repository:tag format.
>> >>>>>> it's too general if we use {language_version} as repository for SDK images. (version is added when we support multiple versions.)
>> >>>>>> So I would like to include sdk to repository. Images generated at local will also have the same name.
>> >>>>>> Here are some examples:
>> >>>>>>
>> >>>>>> python2.7_sdk:2.15.0
>> >>>>>> java_sdk:2.15.0_rc
>> >>>>>> go_sdk:latest
>> >>>>>>
>> >>>>>> I will proceed with this format if there is no strong opposition by tomorrow noon(PST).
>> >>>>>>
>> >>>>>> To PMC members:
>> >>>>>> Permission control will follow the pypi model. All interested PMC members will be added as admins and release managers will be granted push permission.
>> >>>>>> Please let me know your docker id if you want to be added as an admin.
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Hannah
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise <th...@apache.org> wrote:
>> >>>>>>>
>> >>>>>>> This will greatly simplify trying out portable runners: https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>> >>>>>>>
>> >>>>>>> Can't wait for following to disappear from the instructions page: ./gradlew :sdks:python:container:docker
>> >>>>>>>
>> >>>>>>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise <th...@apache.org> wrote:
>> >>>>>>>>
>> >>>>>>>> Awesome, thank you!
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang <ha...@google.com> wrote:
>> >>>>>>>>>
>> >>>>>>>>> Hi Thomas
>> >>>>>>>>>
>> >>>>>>>>> I created snapshot images from head as of around 2PM today.
>> >>>>>>>>> You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.
>> >>>>>>>>>
>> >>>>>>>>> Thanks,
>> >>>>>>>>> Hannah
>> >>>>>>>>>
>> >>>>>>>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <th...@apache.org> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> Hi Hannah,
>> >>>>>>>>>>
>> >>>>>>>>>> Thank you, I know how to build the containers locally, but not how to publish them!
>> >>>>>>>>>>
>> >>>>>>>>>> The cwiki says "Publishing images to gcr.io/beam requires permissions in apache-beam-testing project."
>> >>>>>>>>>>
>> >>>>>>>>>> Can I get access to the testing project (at least temporarily) and what would I need to setup to run the publish target that is shown on cwiki?
>> >>>>>>>>>>
>> >>>>>>>>>> Thanks,
>> >>>>>>>>>> Thomas
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <ha...@google.com> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>> Hi Thomas
>> >>>>>>>>>>>
>> >>>>>>>>>>> I haven't uploaded any snapshot images yet. Here is how you can create one from head.
>> >>>>>>>>>>> > cd [...]/beam/
>> >>>>>>>>>>> # For Python
>> >>>>>>>>>>> > ./gradlew :sdks:python:container:py{version}:docker where version is {2,35,36,37}
>> >>>>>>>>>>> # For Java
>> >>>>>>>>>>> > ./gradlew -p sdks/java/container docker
>> >>>>>>>>>>> # For Go
>> >>>>>>>>>>> > ./gradlew -p sdks/go/container docker
>> >>>>>>>>>>>
>> >>>>>>>>>>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot from head.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Please let me know if you have any questions.
>> >>>>>>>>>>> Hannah
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <th...@apache.org> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I actually found something in [1], but it is 2.15 unfortunately.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> [1] https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <th...@apache.org> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Thanks for working on this. Do you happen to have publicly accessible snapshots published for your testing currently (even when the final location isn't sorted out)?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> I would like to use a 2.16 based Python SDK image for working on my downstream project, but could not find anything in gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>>> Thomas
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <va...@google.com> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com> wrote:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Hi team
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I am working on improving docker container support for Beam. We would like to publish prebuilt containers for each release version and daily snapshot. Current work focuses on release images only and it would be part of the release process.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> The release images will be pushed to GCR which is publicly accessible(pullable). We will use the following locations.
>> >>>>>>>>>>>>>>> Repository: gcr.io/beam
>> >>>>>>>>>>>>>>> Project: apache-beam-testing
>> >>>>>>>>>>>>>>> More details, including naming and tagging scheme, can be found at wiki which is written by several contributors.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I would like to discuss these two questions.
>> >>>>>>>>>>>>>>> 1. How many tests do we need to run before pushing images to gcr?
>> >>>>>>>>>>>>>>> Publishing artifacts is the last step of the release process, so at this moment, we already verified all codebase. In addition, many Jenkins tests use containers, so it is already verified several times. Do we need to run it again?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> In a docker repository, one container image can have multiple tags. One possibility is that on the last step of the release process, after sufficient testing, we place a production tag on an image that was already pushed with a dev tag.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> For example a dev tag may look like:
>> >>>>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look like:
>> >>>>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same image at the end.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> We should also plan what the process of updating the container image will look like, if we need to release the image with additional changes, and how we will test these changes before the final push (or placing production tag).
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> 2. How many tests do we need to run to validate pushed images?
>> >>>>>>>>>>>>>>> When we push the images, we assume the images would work and pass all the tests. After pushing, we should confirm the images are pullable and useable. I suggest we run several tests on dataflow with each pushed image. What do you think?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I think it makes sense to do - Beam runners that use SDK container images should have some continuously running tests, which periodically check that all supported images are pullable and still compatible with the runner.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> This work can be refined later as we explore more during our release process.
>> >>>>>>>>>>>>>>> Please comment or edit the wiki page or reply to this email with your opinions.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>>>>> Hannah
Re: Improve container support
Posted by Kyle Weaver <kc...@google.com>.
We do link from https://beam.apache.org/documentation/runtime/environments/.
On Tue, Nov 19, 2019 at 10:59 AM Robert Bradshaw <ro...@google.com>
wrote:
> We should probably add a link to these from our site as well, for
> visibility.
>
> On Tue, Nov 19, 2019 at 10:56 AM Kyle Weaver <kc...@google.com> wrote:
> >
> > +1 Thanks for bringing that up Chad, I had the same problem locating the
> docker images on Docker hub (searching "apachebeam" is the only way that
> seems to work).
> >
> > Another little thing I noticed is that
> https://hub.docker.com/u/apachebeam is mostly empty. It'd be good to at
> least add the Beam logo as a profile picture and a link to the website in
> the description.
> >
> > On Tue, Nov 19, 2019 at 10:21 AM Chad Dombrova <ch...@gmail.com>
> wrote:
> >>
> >> Hi all,
> >> I think it might be good to update the description of the beam docker
> images and add some descriptive tags, because searching for "apache beam"
> in docker hub does not turn up anything:
> https://hub.docker.com/search?q=apache%20beam&type=image.
> >>
> >> I clicked through 10 pages worth and couldn't find it. Maybe I missed
> something, but it clearly shouldn't be this hard. I did eventually manage
> to find it through the docs.
> >>
> >> Also, googling "apache beam python docker" also does not yield anything
> useful. In fact, it turns up an unofficial apache beam docker hub image.
> >>
> >> One thing I noticed is that images designated as "Official Images" get
> listed first, so it would be good to get that done as well.
> >>
> >> thanks!
> >> chad
> >>
> >>
> >>
> >>
> >> On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang <ha...@google.com>
> wrote:
> >>>
> >>> Hi team
> >>>
> >>> I haven't received any objections, so will proceed with settings
> mentioned in a previous email.
> >>>
> >>> A reminder to PMC members, please let me know your docker hub id if
> you want to be an admin.
> >>>
> >>> Thanks,
> >>> Hannah
> >>>
> >>> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka <go...@google.com> wrote:
> >>>>
> >>>> Please ignore the previous email. I was looking at the older document
> in the mail thread.
> >>>>
> >>>> On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka <go...@google.com>
> wrote:
> >>>>>
> >>>>> I think sdk in the name is obsolete as they are all under sdks name
> space.
> >>>>>
> >>>>> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang <ha...@google.com>
> wrote:
> >>>>>>
> >>>>>> Hi Team
> >>>>>>
> >>>>>> Thanks for all the comments about beam containers.
> >>>>>> After considering various opinions and investigating gcr and docker
> hub, we decided to push images to docker hub.
> >>>>>>
> >>>>>> Each image will have two tags, {version}_rc and {version}.
> {version} tag will be added after the release candidate image is verified.
> >>>>>> Meanwhile, we will have latest tag for each repository, which
> always points to the most recent verified release image, so users can pull
> it by default.
> >>>>>>
> >>>>>> Docker hub doesn't support leveled repository, which means we
> should follow repository:tag format.
> >>>>>> it's too general if we use {language_version} as repository for SDK
> images. (version is added when we support multiple versions.)
> >>>>>> So I would like to include sdk to repository. Images generated at
> local will also have the same name.
> >>>>>> Here are some examples:
> >>>>>>
> >>>>>> python2.7_sdk:2.15.0
> >>>>>> java_sdk:2.15.0_rc
> >>>>>> go_sdk:latest
> >>>>>>
> >>>>>> I will proceed with this format if there is no strong opposition by
> tomorrow noon(PST).
> >>>>>>
> >>>>>> To PMC members:
> >>>>>> Permission control will follow the pypi model. All interested PMC
> members will be added as admins and release managers will be granted push
> permission.
> >>>>>> Please let me know your docker id if you want to be added as an
> admin.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Hannah
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise <th...@apache.org> wrote:
> >>>>>>>
> >>>>>>> This will greatly simplify trying out portable runners:
> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
> >>>>>>>
> >>>>>>> Can't wait for following to disappear from the instructions page:
> ./gradlew :sdks:python:container:docker
> >>>>>>>
> >>>>>>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise <th...@apache.org>
> wrote:
> >>>>>>>>
> >>>>>>>> Awesome, thank you!
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang <
> hannahjiang@google.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi Thomas
> >>>>>>>>>
> >>>>>>>>> I created snapshot images from head as of around 2PM today.
> >>>>>>>>> You can pull images from
> gcr.io/apache-beam-testing/beam/sdks/snapshot.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Hannah
> >>>>>>>>>
> >>>>>>>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <th...@apache.org>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi Hannah,
> >>>>>>>>>>
> >>>>>>>>>> Thank you, I know how to build the containers locally, but not
> how to publish them!
> >>>>>>>>>>
> >>>>>>>>>> The cwiki says "Publishing images to gcr.io/beam requires
> permissions in apache-beam-testing project."
> >>>>>>>>>>
> >>>>>>>>>> Can I get access to the testing project (at least temporarily)
> and what would I need to setup to run the publish target that is shown on
> cwiki?
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Thomas
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <
> hannahjiang@google.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Thomas
> >>>>>>>>>>>
> >>>>>>>>>>> I haven't uploaded any snapshot images yet. Here is how you
> can create one from head.
> >>>>>>>>>>> > cd [...]/beam/
> >>>>>>>>>>> # For Python
> >>>>>>>>>>> > ./gradlew :sdks:python:container:py{version}:docker where
> version is {2,35,36,37}
> >>>>>>>>>>> # For Java
> >>>>>>>>>>> > ./gradlew -p sdks/java/container docker
> >>>>>>>>>>> # For Go
> >>>>>>>>>>> > ./gradlew -p sdks/go/container docker
> >>>>>>>>>>>
> >>>>>>>>>>> The 2.15 one is just for testing, not a real 2.15.0, nor a
> snapshot from head.
> >>>>>>>>>>>
> >>>>>>>>>>> Please let me know if you have any questions.
> >>>>>>>>>>> Hannah
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <th...@apache.org>
> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> I actually found something in [1], but it is 2.15
> unfortunately.
> >>>>>>>>>>>>
> >>>>>>>>>>>> [1]
> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <th...@apache.org>
> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for working on this. Do you happen to have publicly
> accessible snapshots published for your testing currently (even when the
> final location isn't sorted out)?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I would like to use a 2.16 based Python SDK image for
> working on my downstream project, but could not find anything in
> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> Thomas
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
> valentyn@google.com> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
> hannahjiang@google.com> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi team
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I am working on improving docker container support for
> Beam. We would like to publish prebuilt containers for each release version
> and daily snapshot. Current work focuses on release images only and it
> would be part of the release process.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The release images will be pushed to GCR which is publicly
> accessible(pullable). We will use the following locations.
> >>>>>>>>>>>>>>> Repository: gcr.io/beam
> >>>>>>>>>>>>>>> Project: apache-beam-testing
> >>>>>>>>>>>>>>> More details, including naming and tagging scheme, can be
> found at wiki which is written by several contributors.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I would like to discuss these two questions.
> >>>>>>>>>>>>>>> 1. How many tests do we need to run before pushing images
> to gcr?
> >>>>>>>>>>>>>>> Publishing artifacts is the last step of the release
> process, so at this moment, we already verified all codebase. In addition,
> many Jenkins tests use containers, so it is already verified several times.
> Do we need to run it again?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> In a docker repository, one container image can have
> multiple tags. One possibility is that on the last step of the release
> process, after sufficient testing, we place a production tag on an image
> that was already pushed with a dev tag.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> For example a dev tag may look like:
> >>>>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag
> may look like:
> >>>>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to
> the same image at the end.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We should also plan what the process of updating the
> container image will look like, if we need to release the image with
> additional changes, and how we will test these changes before the final
> push (or placing production tag).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2. How many tests do we need to run to validate pushed
> images?
> >>>>>>>>>>>>>>> When we push the images, we assume the images would work
> and pass all the tests. After pushing, we should confirm the images are
> pullable and useable. I suggest we run several tests on dataflow with each
> pushed image. What do you think?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I think it makes sense to do - Beam runners that use SDK
> container images should have some continuously running tests, which
> periodically check that all supported images are pullable and still
> compatible with the runner.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> This work can be refined later as we explore more during
> our release process.
> >>>>>>>>>>>>>>> Please comment or edit the wiki page or reply to this
> email with your opinions.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Hannah
>
Re: Improve container support
Posted by Robert Bradshaw <ro...@google.com>.
We should probably add a link to these from our site as well, for visibility.
On Tue, Nov 19, 2019 at 10:56 AM Kyle Weaver <kc...@google.com> wrote:
>
> +1 Thanks for bringing that up Chad, I had the same problem locating the docker images on Docker hub (searching "apachebeam" is the only way that seems to work).
>
> Another little thing I noticed is that https://hub.docker.com/u/apachebeam is mostly empty. It'd be good to at least add the Beam logo as a profile picture and a link to the website in the description.
>
> On Tue, Nov 19, 2019 at 10:21 AM Chad Dombrova <ch...@gmail.com> wrote:
>>
>> Hi all,
>> I think it might be good to update the description of the beam docker images and add some descriptive tags, because searching for "apache beam" in docker hub does not turn up anything: https://hub.docker.com/search?q=apache%20beam&type=image.
>>
>> I clicked through 10 pages worth and couldn't find it. Maybe I missed something, but it clearly shouldn't be this hard. I did eventually manage to find it through the docs.
>>
>> Also, googling "apache beam python docker" also does not yield anything useful. In fact, it turns up an unofficial apache beam docker hub image.
>>
>> One thing I noticed is that images designated as "Official Images" get listed first, so it would be good to get that done as well.
>>
>> thanks!
>> chad
>>
>>
>>
>>
>> On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang <ha...@google.com> wrote:
>>>
>>> Hi team
>>>
>>> I haven't received any objections, so will proceed with settings mentioned in a previous email.
>>>
>>> A reminder to PMC members, please let me know your docker hub id if you want to be an admin.
>>>
>>> Thanks,
>>> Hannah
>>>
>>> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka <go...@google.com> wrote:
>>>>
>>>> Please ignore the previous email. I was looking at the older document in the mail thread.
>>>>
>>>> On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka <go...@google.com> wrote:
>>>>>
>>>>> I think sdk in the name is obsolete as they are all under sdks name space.
>>>>>
>>>>> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang <ha...@google.com> wrote:
>>>>>>
>>>>>> Hi Team
>>>>>>
>>>>>> Thanks for all the comments about beam containers.
>>>>>> After considering various opinions and investigating gcr and docker hub, we decided to push images to docker hub.
>>>>>>
>>>>>> Each image will have two tags, {version}_rc and {version}. {version} tag will be added after the release candidate image is verified.
>>>>>> Meanwhile, we will have latest tag for each repository, which always points to the most recent verified release image, so users can pull it by default.
>>>>>>
>>>>>> Docker hub doesn't support leveled repository, which means we should follow repository:tag format.
>>>>>> it's too general if we use {language_version} as repository for SDK images. (version is added when we support multiple versions.)
>>>>>> So I would like to include sdk to repository. Images generated at local will also have the same name.
>>>>>> Here are some examples:
>>>>>>
>>>>>> python2.7_sdk:2.15.0
>>>>>> java_sdk:2.15.0_rc
>>>>>> go_sdk:latest
>>>>>>
>>>>>> I will proceed with this format if there is no strong opposition by tomorrow noon(PST).
>>>>>>
>>>>>> To PMC members:
>>>>>> Permission control will follow the pypi model. All interested PMC members will be added as admins and release managers will be granted push permission.
>>>>>> Please let me know your docker id if you want to be added as an admin.
>>>>>>
>>>>>> Thanks,
>>>>>> Hannah
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise <th...@apache.org> wrote:
>>>>>>>
>>>>>>> This will greatly simplify trying out portable runners: https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>>>>>>>
>>>>>>> Can't wait for following to disappear from the instructions page: ./gradlew :sdks:python:container:docker
>>>>>>>
>>>>>>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise <th...@apache.org> wrote:
>>>>>>>>
>>>>>>>> Awesome, thank you!
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang <ha...@google.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi Thomas
>>>>>>>>>
>>>>>>>>> I created snapshot images from head as of around 2PM today.
>>>>>>>>> You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Hannah
>>>>>>>>>
>>>>>>>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <th...@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Hannah,
>>>>>>>>>>
>>>>>>>>>> Thank you, I know how to build the containers locally, but not how to publish them!
>>>>>>>>>>
>>>>>>>>>> The cwiki says "Publishing images to gcr.io/beam requires permissions in apache-beam-testing project."
>>>>>>>>>>
>>>>>>>>>> Can I get access to the testing project (at least temporarily) and what would I need to setup to run the publish target that is shown on cwiki?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Thomas
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <ha...@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Thomas
>>>>>>>>>>>
>>>>>>>>>>> I haven't uploaded any snapshot images yet. Here is how you can create one from head.
>>>>>>>>>>> > cd [...]/beam/
>>>>>>>>>>> # For Python
>>>>>>>>>>> > ./gradlew :sdks:python:container:py{version}:docker where version is {2,35,36,37}
>>>>>>>>>>> # For Java
>>>>>>>>>>> > ./gradlew -p sdks/java/container docker
>>>>>>>>>>> # For Go
>>>>>>>>>>> > ./gradlew -p sdks/go/container docker
>>>>>>>>>>>
>>>>>>>>>>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot from head.
>>>>>>>>>>>
>>>>>>>>>>> Please let me know if you have any questions.
>>>>>>>>>>> Hannah
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <th...@apache.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I actually found something in [1], but it is 2.15 unfortunately.
>>>>>>>>>>>>
>>>>>>>>>>>> [1] https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <th...@apache.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for working on this. Do you happen to have publicly accessible snapshots published for your testing currently (even when the final location isn't sorted out)?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would like to use a 2.16 based Python SDK image for working on my downstream project, but could not find anything in gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <va...@google.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi team
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am working on improving docker container support for Beam. We would like to publish prebuilt containers for each release version and daily snapshot. Current work focuses on release images only and it would be part of the release process.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The release images will be pushed to GCR which is publicly accessible(pullable). We will use the following locations.
>>>>>>>>>>>>>>> Repository: gcr.io/beam
>>>>>>>>>>>>>>> Project: apache-beam-testing
>>>>>>>>>>>>>>> More details, including naming and tagging scheme, can be found at wiki which is written by several contributors.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I would like to discuss these two questions.
>>>>>>>>>>>>>>> 1. How many tests do we need to run before pushing images to gcr?
>>>>>>>>>>>>>>> Publishing artifacts is the last step of the release process, so at this moment, we already verified all codebase. In addition, many Jenkins tests use containers, so it is already verified several times. Do we need to run it again?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In a docker repository, one container image can have multiple tags. One possibility is that on the last step of the release process, after sufficient testing, we place a production tag on an image that was already pushed with a dev tag.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For example a dev tag may look like:
>>>>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look like:
>>>>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same image at the end.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We should also plan what the process of updating the container image will look like, if we need to release the image with additional changes, and how we will test these changes before the final push (or placing production tag).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2. How many tests do we need to run to validate pushed images?
>>>>>>>>>>>>>>> When we push the images, we assume the images would work and pass all the tests. After pushing, we should confirm the images are pullable and useable. I suggest we run several tests on dataflow with each pushed image. What do you think?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think it makes sense to do - Beam runners that use SDK container images should have some continuously running tests, which periodically check that all supported images are pullable and still compatible with the runner.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This work can be refined later as we explore more during our release process.
>>>>>>>>>>>>>>> Please comment or edit the wiki page or reply to this email with your opinions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Hannah
Re: Improve container support
Posted by Kyle Weaver <kc...@google.com>.
+1 Thanks for bringing that up Chad, I had the same problem locating the
docker images on Docker hub (searching "apachebeam" is the only way that
seems to work).
Another little thing I noticed is that https://hub.docker.com/u/apachebeam is
mostly empty. It'd be good to at least add the Beam logo as a profile
picture and a link to the website in the description.
On Tue, Nov 19, 2019 at 10:21 AM Chad Dombrova <ch...@gmail.com> wrote:
> Hi all,
> I think it might be good to update the description of the beam docker
> images and add some descriptive tags, because searching for "apache beam"
> in docker hub does not turn up anything:
> https://hub.docker.com/search?q=apache%20beam&type=image.
>
> I clicked through 10 pages worth and couldn't find it. Maybe I missed
> something, but it clearly shouldn't be this hard. I did eventually manage
> to find it through the docs.
>
> Also, googling "apache beam python docker" also does not yield anything
> useful. In fact, it turns up an *unofficial* apache beam docker hub
> image.
>
> One thing I noticed is that images designated as "Official Images" get
> listed first, so it would be good to get that done as well.
>
> thanks!
> chad
>
>
>
>
> On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang <ha...@google.com>
> wrote:
>
>> Hi team
>>
>> I haven't received any objections, so will proceed with settings
>> mentioned in a previous email.
>>
>> A reminder to PMC members, please let me know your docker hub id if you
>> want to be an admin.
>>
>> Thanks,
>> Hannah
>>
>> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka <go...@google.com> wrote:
>>
>>> Please ignore the previous email. I was looking at the older document in
>>> the mail thread.
>>>
>>> On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka <go...@google.com> wrote:
>>>
>>>> I think sdk in the name is obsolete as they are all under sdks name
>>>> space.
>>>>
>>>> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang <ha...@google.com>
>>>> wrote:
>>>>
>>>>> Hi Team
>>>>>
>>>>> Thanks for all the comments about beam containers.
>>>>> After considering various opinions and investigating gcr and docker
>>>>> hub, we decided to push images to docker hub.
>>>>>
>>>>> Each image will have two tags, {version}_rc and {version}. {version}
>>>>> tag will be added after the release candidate image is verified.
>>>>> Meanwhile, we will have* latest* tag for each repository, which
>>>>> always points to the most recent verified release image, so users can pull
>>>>> it by default.
>>>>>
>>>>> Docker hub doesn't support leveled repository, which means we should
>>>>> follow *repository:tag* format.
>>>>> it's too general if we use {language_version} as repository for SDK
>>>>> images. (version is added when we support multiple versions.)
>>>>> So I would like to include *sdk* to repository. Images generated at
>>>>> local will also have the same name.
>>>>> Here are some examples:
>>>>>
>>>>> - python2.7_sdk:2.15.0
>>>>> - java_sdk:2.15.0_rc
>>>>> - go_sdk:latest
>>>>>
>>>>> I will proceed with this format if there is no strong opposition by
>>>>> tomorrow noon(PST).
>>>>>
>>>>> *To PMC members*:
>>>>> Permission control will follow the pypi model. All interested PMC
>>>>> members will be added as admins and release managers will be granted push
>>>>> permission.
>>>>> Please let me know your *docker id* if you want to be added as an
>>>>> admin.
>>>>>
>>>>> Thanks,
>>>>> Hannah
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>>> This will greatly simplify trying out portable runners:
>>>>>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>>>>>>
>>>>>> Can't wait for following to disappear from the instructions page: ./gradlew
>>>>>> :sdks:python:container:docker
>>>>>>
>>>>>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise <th...@apache.org> wrote:
>>>>>>
>>>>>>> Awesome, thank you!
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang <ha...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Thomas
>>>>>>>>
>>>>>>>> I created snapshot images from head as of around 2PM today.
>>>>>>>> You can pull images from
>>>>>>>> gcr.io/apache-beam-testing/beam/sdks/snapshot.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Hannah
>>>>>>>>
>>>>>>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <th...@apache.org> wrote:
>>>>>>>>
>>>>>>>>> Hi Hannah,
>>>>>>>>>
>>>>>>>>> Thank you, I know how to build the containers locally, but not how
>>>>>>>>> to publish them!
>>>>>>>>>
>>>>>>>>> The cwiki says "Publishing images to gcr.io/beam requires
>>>>>>>>> permissions in apache-beam-testing project."
>>>>>>>>>
>>>>>>>>> Can I get access to the testing project (at least temporarily) and
>>>>>>>>> what would I need to setup to run the publish target that is shown on cwiki?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Thomas
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <
>>>>>>>>> hannahjiang@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Thomas
>>>>>>>>>>
>>>>>>>>>> I haven't uploaded any snapshot images yet. Here is how you can
>>>>>>>>>> create one from head.
>>>>>>>>>> > cd [...]/beam/
>>>>>>>>>> # For Python
>>>>>>>>>> > ./gradlew :sdks:python:container:py{version}:docker *where
>>>>>>>>>> version is {2,35,36,37}*
>>>>>>>>>> # For Java
>>>>>>>>>> > ./gradlew -p sdks/java/container docker
>>>>>>>>>> # For Go
>>>>>>>>>> > ./gradlew -p sdks/go/container docker
>>>>>>>>>>
>>>>>>>>>> The 2.15 one is just for testing, not a real 2.15.0, nor a
>>>>>>>>>> snapshot from head.
>>>>>>>>>>
>>>>>>>>>> Please let me know if you have any questions.
>>>>>>>>>> Hannah
>>>>>>>>>>
>>>>>>>>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <th...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I actually found something in [1], but it is 2.15 unfortunately.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <th...@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for working on this. Do you happen to have publicly
>>>>>>>>>>>> accessible snapshots published for your testing currently (even when the
>>>>>>>>>>>> final location isn't sorted out)?
>>>>>>>>>>>>
>>>>>>>>>>>> I would like to use a 2.16 based Python SDK image for working
>>>>>>>>>>>> on my downstream project, but could not find anything in
>>>>>>>>>>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Thomas
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>>>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>>>>>>>>>>>>> hannahjiang@google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi team
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am working on improving docker container support for Beam.
>>>>>>>>>>>>>> We would like to publish prebuilt containers for each release version and
>>>>>>>>>>>>>> daily snapshot. Current work focuses on release images only and it would be
>>>>>>>>>>>>>> part of the release process.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The release images will be pushed to GCR which is publicly
>>>>>>>>>>>>>> accessible(pullable). We will use the following locations.
>>>>>>>>>>>>>> *Repository*: gcr.io/beam
>>>>>>>>>>>>>> *Project*: apache-beam-testing
>>>>>>>>>>>>>> More details, including naming and tagging scheme, can be
>>>>>>>>>>>>>> found at wiki
>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> which
>>>>>>>>>>>>>> is written by several contributors.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I would like to discuss these two questions.
>>>>>>>>>>>>>> *1. How many tests do we need to run before pushing images to
>>>>>>>>>>>>>> gcr*?
>>>>>>>>>>>>>> Publishing artifacts is the last step of the release process,
>>>>>>>>>>>>>> so at this moment, we already verified all codebase. In addition, many
>>>>>>>>>>>>>> Jenkins tests use containers, so it is already verified several times. Do
>>>>>>>>>>>>>> we need to run it again?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> In a docker repository, one container image can have multiple
>>>>>>>>>>>>> tags. One possibility is that on the last step of the release process,
>>>>>>>>>>>>> after sufficient testing, we place a production tag on an image that was
>>>>>>>>>>>>> already pushed with a dev tag.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For example a dev tag may look like:
>>>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag
>>>>>>>>>>>>> may look like:
>>>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the
>>>>>>>>>>>>> same image at the end.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We should also plan what the process of updating the container
>>>>>>>>>>>>> image will look like, if we need to release the image with
>>>>>>>>>>>>> additional changes, and how we will test these changes before the final
>>>>>>>>>>>>> push (or placing production tag).
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *2. How many tests do we need to run to validate pushed
>>>>>>>>>>>>>> images?*
>>>>>>>>>>>>>> When we push the images, we assume the images would work and
>>>>>>>>>>>>>> pass all the tests. After pushing, we should confirm the images are
>>>>>>>>>>>>>> pullable and useable. I suggest we run several tests on dataflow with each
>>>>>>>>>>>>>> pushed image. What do you think?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think it makes sense to do - Beam runners that use SDK
>>>>>>>>>>>>> container images should have some continuously running tests, which
>>>>>>>>>>>>> periodically check that all supported images are pullable and still
>>>>>>>>>>>>> compatible with the runner.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This work can be refined later as we explore more during our
>>>>>>>>>>>>>> release process.
>>>>>>>>>>>>>> Please comment or edit the wiki page or reply to this email
>>>>>>>>>>>>>> with your opinions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Hannah
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
Re: Improve container support
Posted by Chad Dombrova <ch...@gmail.com>.
Hi all,
I think it might be good to update the description of the beam docker
images and add some descriptive tags, because searching for "apache beam"
in docker hub does not turn up anything:
https://hub.docker.com/search?q=apache%20beam&type=image.
I clicked through 10 pages worth and couldn't find it. Maybe I missed
something, but it clearly shouldn't be this hard. I did eventually manage
to find it through the docs.
Also, googling "apache beam python docker" also does not yield anything
useful. In fact, it turns up an *unofficial* apache beam docker hub image.
One thing I noticed is that images designated as "Official Images" get
listed first, so it would be good to get that done as well.
thanks!
chad
On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang <ha...@google.com> wrote:
> Hi team
>
> I haven't received any objections, so will proceed with settings mentioned
> in a previous email.
>
> A reminder to PMC members, please let me know your docker hub id if you
> want to be an admin.
>
> Thanks,
> Hannah
>
> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka <go...@google.com> wrote:
>
>> Please ignore the previous email. I was looking at the older document in
>> the mail thread.
>>
>> On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka <go...@google.com> wrote:
>>
>>> I think sdk in the name is obsolete as they are all under sdks name
>>> space.
>>>
>>> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang <ha...@google.com>
>>> wrote:
>>>
>>>> Hi Team
>>>>
>>>> Thanks for all the comments about beam containers.
>>>> After considering various opinions and investigating gcr and docker
>>>> hub, we decided to push images to docker hub.
>>>>
>>>> Each image will have two tags, {version}_rc and {version}. {version}
>>>> tag will be added after the release candidate image is verified.
>>>> Meanwhile, we will have* latest* tag for each repository, which always
>>>> points to the most recent verified release image, so users can pull it by
>>>> default.
>>>>
>>>> Docker hub doesn't support leveled repository, which means we should
>>>> follow *repository:tag* format.
>>>> it's too general if we use {language_version} as repository for SDK
>>>> images. (version is added when we support multiple versions.)
>>>> So I would like to include *sdk* to repository. Images generated at
>>>> local will also have the same name.
>>>> Here are some examples:
>>>>
>>>> - python2.7_sdk:2.15.0
>>>> - java_sdk:2.15.0_rc
>>>> - go_sdk:latest
>>>>
>>>> I will proceed with this format if there is no strong opposition by
>>>> tomorrow noon(PST).
>>>>
>>>> *To PMC members*:
>>>> Permission control will follow the pypi model. All interested PMC
>>>> members will be added as admins and release managers will be granted push
>>>> permission.
>>>> Please let me know your *docker id* if you want to be added as an
>>>> admin.
>>>>
>>>> Thanks,
>>>> Hannah
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise <th...@apache.org> wrote:
>>>>
>>>>> This will greatly simplify trying out portable runners:
>>>>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>>>>>
>>>>> Can't wait for following to disappear from the instructions page: ./gradlew
>>>>> :sdks:python:container:docker
>>>>>
>>>>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>>> Awesome, thank you!
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang <ha...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Thomas
>>>>>>>
>>>>>>> I created snapshot images from head as of around 2PM today.
>>>>>>> You can pull images from
>>>>>>> gcr.io/apache-beam-testing/beam/sdks/snapshot.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Hannah
>>>>>>>
>>>>>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <th...@apache.org> wrote:
>>>>>>>
>>>>>>>> Hi Hannah,
>>>>>>>>
>>>>>>>> Thank you, I know how to build the containers locally, but not how
>>>>>>>> to publish them!
>>>>>>>>
>>>>>>>> The cwiki says "Publishing images to gcr.io/beam requires
>>>>>>>> permissions in apache-beam-testing project."
>>>>>>>>
>>>>>>>> Can I get access to the testing project (at least temporarily) and
>>>>>>>> what would I need to setup to run the publish target that is shown on cwiki?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Thomas
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <
>>>>>>>> hannahjiang@google.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Thomas
>>>>>>>>>
>>>>>>>>> I haven't uploaded any snapshot images yet. Here is how you can
>>>>>>>>> create one from head.
>>>>>>>>> > cd [...]/beam/
>>>>>>>>> # For Python
>>>>>>>>> > ./gradlew :sdks:python:container:py{version}:docker *where
>>>>>>>>> version is {2,35,36,37}*
>>>>>>>>> # For Java
>>>>>>>>> > ./gradlew -p sdks/java/container docker
>>>>>>>>> # For Go
>>>>>>>>> > ./gradlew -p sdks/go/container docker
>>>>>>>>>
>>>>>>>>> The 2.15 one is just for testing, not a real 2.15.0, nor a
>>>>>>>>> snapshot from head.
>>>>>>>>>
>>>>>>>>> Please let me know if you have any questions.
>>>>>>>>> Hannah
>>>>>>>>>
>>>>>>>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <th...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I actually found something in [1], but it is 2.15 unfortunately.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>>>>>>>>>
>>>>>>>>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <th...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks for working on this. Do you happen to have publicly
>>>>>>>>>>> accessible snapshots published for your testing currently (even when the
>>>>>>>>>>> final location isn't sorted out)?
>>>>>>>>>>>
>>>>>>>>>>> I would like to use a 2.16 based Python SDK image for working on
>>>>>>>>>>> my downstream project, but could not find anything in
>>>>>>>>>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Thomas
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>>>>>>>>>>>> hannahjiang@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi team
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am working on improving docker container support for Beam.
>>>>>>>>>>>>> We would like to publish prebuilt containers for each release version and
>>>>>>>>>>>>> daily snapshot. Current work focuses on release images only and it would be
>>>>>>>>>>>>> part of the release process.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The release images will be pushed to GCR which is publicly
>>>>>>>>>>>>> accessible(pullable). We will use the following locations.
>>>>>>>>>>>>> *Repository*: gcr.io/beam
>>>>>>>>>>>>> *Project*: apache-beam-testing
>>>>>>>>>>>>> More details, including naming and tagging scheme, can be
>>>>>>>>>>>>> found at wiki
>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> which
>>>>>>>>>>>>> is written by several contributors.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would like to discuss these two questions.
>>>>>>>>>>>>> *1. How many tests do we need to run before pushing images to
>>>>>>>>>>>>> gcr*?
>>>>>>>>>>>>> Publishing artifacts is the last step of the release process,
>>>>>>>>>>>>> so at this moment, we already verified all codebase. In addition, many
>>>>>>>>>>>>> Jenkins tests use containers, so it is already verified several times. Do
>>>>>>>>>>>>> we need to run it again?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> In a docker repository, one container image can have multiple
>>>>>>>>>>>> tags. One possibility is that on the last step of the release process,
>>>>>>>>>>>> after sufficient testing, we place a production tag on an image that was
>>>>>>>>>>>> already pushed with a dev tag.
>>>>>>>>>>>>
>>>>>>>>>>>> For example a dev tag may look like:
>>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may
>>>>>>>>>>>> look like:
>>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the
>>>>>>>>>>>> same image at the end.
>>>>>>>>>>>>
>>>>>>>>>>>> We should also plan what the process of updating the container
>>>>>>>>>>>> image will look like, if we need to release the image with
>>>>>>>>>>>> additional changes, and how we will test these changes before the final
>>>>>>>>>>>> push (or placing production tag).
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> *2. How many tests do we need to run to validate pushed
>>>>>>>>>>>>> images?*
>>>>>>>>>>>>> When we push the images, we assume the images would work and
>>>>>>>>>>>>> pass all the tests. After pushing, we should confirm the images are
>>>>>>>>>>>>> pullable and useable. I suggest we run several tests on dataflow with each
>>>>>>>>>>>>> pushed image. What do you think?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I think it makes sense to do - Beam runners that use SDK
>>>>>>>>>>>> container images should have some continuously running tests, which
>>>>>>>>>>>> periodically check that all supported images are pullable and still
>>>>>>>>>>>> compatible with the runner.
>>>>>>>>>>>>
>>>>>>>>>>>> This work can be refined later as we explore more during our
>>>>>>>>>>>>> release process.
>>>>>>>>>>>>> Please comment or edit the wiki page or reply to this email
>>>>>>>>>>>>> with your opinions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Hannah
>>>>>>>>>>>>>
>>>>>>>>>>>>
Re: Improve container support
Posted by Hannah Jiang <ha...@google.com>.
Hi team
I haven't received any objections, so will proceed with settings mentioned
in a previous email.
A reminder to PMC members, please let me know your docker hub id if you
want to be an admin.
Thanks,
Hannah
On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka <go...@google.com> wrote:
> Please ignore the previous email. I was looking at the older document in
> the mail thread.
>
> On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka <go...@google.com> wrote:
>
>> I think sdk in the name is obsolete as they are all under sdks name space.
>>
>> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang <ha...@google.com>
>> wrote:
>>
>>> Hi Team
>>>
>>> Thanks for all the comments about beam containers.
>>> After considering various opinions and investigating gcr and docker hub,
>>> we decided to push images to docker hub.
>>>
>>> Each image will have two tags, {version}_rc and {version}. {version} tag
>>> will be added after the release candidate image is verified.
>>> Meanwhile, we will have* latest* tag for each repository, which always
>>> points to the most recent verified release image, so users can pull it by
>>> default.
>>>
>>> Docker hub doesn't support leveled repository, which means we should
>>> follow *repository:tag* format.
>>> it's too general if we use {language_version} as repository for SDK
>>> images. (version is added when we support multiple versions.)
>>> So I would like to include *sdk* to repository. Images generated at
>>> local will also have the same name.
>>> Here are some examples:
>>>
>>> - python2.7_sdk:2.15.0
>>> - java_sdk:2.15.0_rc
>>> - go_sdk:latest
>>>
>>> I will proceed with this format if there is no strong opposition by
>>> tomorrow noon(PST).
>>>
>>> *To PMC members*:
>>> Permission control will follow the pypi model. All interested PMC
>>> members will be added as admins and release managers will be granted push
>>> permission.
>>> Please let me know your *docker id* if you want to be added as an admin.
>>>
>>> Thanks,
>>> Hannah
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise <th...@apache.org> wrote:
>>>
>>>> This will greatly simplify trying out portable runners:
>>>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>>>>
>>>> Can't wait for following to disappear from the instructions page: ./gradlew
>>>> :sdks:python:container:docker
>>>>
>>>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise <th...@apache.org> wrote:
>>>>
>>>>> Awesome, thank you!
>>>>>
>>>>>
>>>>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang <ha...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Thomas
>>>>>>
>>>>>> I created snapshot images from head as of around 2PM today.
>>>>>> You can pull images from
>>>>>> gcr.io/apache-beam-testing/beam/sdks/snapshot.
>>>>>>
>>>>>> Thanks,
>>>>>> Hannah
>>>>>>
>>>>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <th...@apache.org> wrote:
>>>>>>
>>>>>>> Hi Hannah,
>>>>>>>
>>>>>>> Thank you, I know how to build the containers locally, but not how
>>>>>>> to publish them!
>>>>>>>
>>>>>>> The cwiki says "Publishing images to gcr.io/beam requires
>>>>>>> permissions in apache-beam-testing project."
>>>>>>>
>>>>>>> Can I get access to the testing project (at least temporarily) and
>>>>>>> what would I need to setup to run the publish target that is shown on cwiki?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Thomas
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <ha...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Thomas
>>>>>>>>
>>>>>>>> I haven't uploaded any snapshot images yet. Here is how you can
>>>>>>>> create one from head.
>>>>>>>> > cd [...]/beam/
>>>>>>>> # For Python
>>>>>>>> > ./gradlew :sdks:python:container:py{version}:docker *where
>>>>>>>> version is {2,35,36,37}*
>>>>>>>> # For Java
>>>>>>>> > ./gradlew -p sdks/java/container docker
>>>>>>>> # For Go
>>>>>>>> > ./gradlew -p sdks/go/container docker
>>>>>>>>
>>>>>>>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot
>>>>>>>> from head.
>>>>>>>>
>>>>>>>> Please let me know if you have any questions.
>>>>>>>> Hannah
>>>>>>>>
>>>>>>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <th...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I actually found something in [1], but it is 2.15 unfortunately.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>>>>>>>>
>>>>>>>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <th...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for working on this. Do you happen to have publicly
>>>>>>>>>> accessible snapshots published for your testing currently (even when the
>>>>>>>>>> final location isn't sorted out)?
>>>>>>>>>>
>>>>>>>>>> I would like to use a 2.16 based Python SDK image for working on
>>>>>>>>>> my downstream project, but could not find anything in
>>>>>>>>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Thomas
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>>>>>>>>>>> hannahjiang@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi team
>>>>>>>>>>>>
>>>>>>>>>>>> I am working on improving docker container support for Beam. We
>>>>>>>>>>>> would like to publish prebuilt containers for each release version and
>>>>>>>>>>>> daily snapshot. Current work focuses on release images only and it would be
>>>>>>>>>>>> part of the release process.
>>>>>>>>>>>>
>>>>>>>>>>>> The release images will be pushed to GCR which is publicly
>>>>>>>>>>>> accessible(pullable). We will use the following locations.
>>>>>>>>>>>> *Repository*: gcr.io/beam
>>>>>>>>>>>> *Project*: apache-beam-testing
>>>>>>>>>>>> More details, including naming and tagging scheme, can be found
>>>>>>>>>>>> at wiki
>>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> which
>>>>>>>>>>>> is written by several contributors.
>>>>>>>>>>>>
>>>>>>>>>>>> I would like to discuss these two questions.
>>>>>>>>>>>> *1. How many tests do we need to run before pushing images to
>>>>>>>>>>>> gcr*?
>>>>>>>>>>>> Publishing artifacts is the last step of the release process,
>>>>>>>>>>>> so at this moment, we already verified all codebase. In addition, many
>>>>>>>>>>>> Jenkins tests use containers, so it is already verified several times. Do
>>>>>>>>>>>> we need to run it again?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> In a docker repository, one container image can have multiple
>>>>>>>>>>> tags. One possibility is that on the last step of the release process,
>>>>>>>>>>> after sufficient testing, we place a production tag on an image that was
>>>>>>>>>>> already pushed with a dev tag.
>>>>>>>>>>>
>>>>>>>>>>> For example a dev tag may look like:
>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may
>>>>>>>>>>> look like:
>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the
>>>>>>>>>>> same image at the end.
>>>>>>>>>>>
>>>>>>>>>>> We should also plan what the process of updating the container
>>>>>>>>>>> image will look like, if we need to release the image with
>>>>>>>>>>> additional changes, and how we will test these changes before the final
>>>>>>>>>>> push (or placing production tag).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *2. How many tests do we need to run to validate pushed images?*
>>>>>>>>>>>> When we push the images, we assume the images would work and
>>>>>>>>>>>> pass all the tests. After pushing, we should confirm the images are
>>>>>>>>>>>> pullable and useable. I suggest we run several tests on dataflow with each
>>>>>>>>>>>> pushed image. What do you think?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I think it makes sense to do - Beam runners that use SDK
>>>>>>>>>>> container images should have some continuously running tests, which
>>>>>>>>>>> periodically check that all supported images are pullable and still
>>>>>>>>>>> compatible with the runner.
>>>>>>>>>>>
>>>>>>>>>>> This work can be refined later as we explore more during our
>>>>>>>>>>>> release process.
>>>>>>>>>>>> Please comment or edit the wiki page or reply to this email
>>>>>>>>>>>> with your opinions.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Hannah
>>>>>>>>>>>>
>>>>>>>>>>>
Re: Improve container support
Posted by Ankur Goenka <go...@google.com>.
Please ignore the previous email. I was looking at the older document in
the mail thread.
On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka <go...@google.com> wrote:
> I think sdk in the name is obsolete as they are all under sdks name space.
>
> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang <ha...@google.com>
> wrote:
>
>> Hi Team
>>
>> Thanks for all the comments about beam containers.
>> After considering various opinions and investigating gcr and docker hub,
>> we decided to push images to docker hub.
>>
>> Each image will have two tags, {version}_rc and {version}. {version} tag
>> will be added after the release candidate image is verified.
>> Meanwhile, we will have* latest* tag for each repository, which always
>> points to the most recent verified release image, so users can pull it by
>> default.
>>
>> Docker hub doesn't support leveled repository, which means we should
>> follow *repository:tag* format.
>> it's too general if we use {language_version} as repository for SDK
>> images. (version is added when we support multiple versions.)
>> So I would like to include *sdk* to repository. Images generated at
>> local will also have the same name.
>> Here are some examples:
>>
>> - python2.7_sdk:2.15.0
>> - java_sdk:2.15.0_rc
>> - go_sdk:latest
>>
>> I will proceed with this format if there is no strong opposition by
>> tomorrow noon(PST).
>>
>> *To PMC members*:
>> Permission control will follow the pypi model. All interested PMC members
>> will be added as admins and release managers will be granted push
>> permission.
>> Please let me know your *docker id* if you want to be added as an admin.
>>
>> Thanks,
>> Hannah
>>
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise <th...@apache.org> wrote:
>>
>>> This will greatly simplify trying out portable runners:
>>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>>>
>>> Can't wait for following to disappear from the instructions page: ./gradlew
>>> :sdks:python:container:docker
>>>
>>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise <th...@apache.org> wrote:
>>>
>>>> Awesome, thank you!
>>>>
>>>>
>>>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang <ha...@google.com>
>>>> wrote:
>>>>
>>>>> Hi Thomas
>>>>>
>>>>> I created snapshot images from head as of around 2PM today.
>>>>> You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot
>>>>> .
>>>>>
>>>>> Thanks,
>>>>> Hannah
>>>>>
>>>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>>> Hi Hannah,
>>>>>>
>>>>>> Thank you, I know how to build the containers locally, but not how to
>>>>>> publish them!
>>>>>>
>>>>>> The cwiki says "Publishing images to gcr.io/beam requires
>>>>>> permissions in apache-beam-testing project."
>>>>>>
>>>>>> Can I get access to the testing project (at least temporarily) and
>>>>>> what would I need to setup to run the publish target that is shown on cwiki?
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <ha...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Thomas
>>>>>>>
>>>>>>> I haven't uploaded any snapshot images yet. Here is how you can
>>>>>>> create one from head.
>>>>>>> > cd [...]/beam/
>>>>>>> # For Python
>>>>>>> > ./gradlew :sdks:python:container:py{version}:docker *where
>>>>>>> version is {2,35,36,37}*
>>>>>>> # For Java
>>>>>>> > ./gradlew -p sdks/java/container docker
>>>>>>> # For Go
>>>>>>> > ./gradlew -p sdks/go/container docker
>>>>>>>
>>>>>>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot
>>>>>>> from head.
>>>>>>>
>>>>>>> Please let me know if you have any questions.
>>>>>>> Hannah
>>>>>>>
>>>>>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <th...@apache.org> wrote:
>>>>>>>
>>>>>>>> I actually found something in [1], but it is 2.15 unfortunately.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>>>>>>>
>>>>>>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <th...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for working on this. Do you happen to have publicly
>>>>>>>>> accessible snapshots published for your testing currently (even when the
>>>>>>>>> final location isn't sorted out)?
>>>>>>>>>
>>>>>>>>> I would like to use a 2.16 based Python SDK image for working on
>>>>>>>>> my downstream project, but could not find anything in
>>>>>>>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Thomas
>>>>>>>>>
>>>>>>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>>>>>>>>>> hannahjiang@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi team
>>>>>>>>>>>
>>>>>>>>>>> I am working on improving docker container support for Beam. We
>>>>>>>>>>> would like to publish prebuilt containers for each release version and
>>>>>>>>>>> daily snapshot. Current work focuses on release images only and it would be
>>>>>>>>>>> part of the release process.
>>>>>>>>>>>
>>>>>>>>>>> The release images will be pushed to GCR which is publicly
>>>>>>>>>>> accessible(pullable). We will use the following locations.
>>>>>>>>>>> *Repository*: gcr.io/beam
>>>>>>>>>>> *Project*: apache-beam-testing
>>>>>>>>>>> More details, including naming and tagging scheme, can be found
>>>>>>>>>>> at wiki
>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> which
>>>>>>>>>>> is written by several contributors.
>>>>>>>>>>>
>>>>>>>>>>> I would like to discuss these two questions.
>>>>>>>>>>> *1. How many tests do we need to run before pushing images to
>>>>>>>>>>> gcr*?
>>>>>>>>>>> Publishing artifacts is the last step of the release process, so
>>>>>>>>>>> at this moment, we already verified all codebase. In addition, many Jenkins
>>>>>>>>>>> tests use containers, so it is already verified several times. Do we need
>>>>>>>>>>> to run it again?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> In a docker repository, one container image can have multiple
>>>>>>>>>> tags. One possibility is that on the last step of the release process,
>>>>>>>>>> after sufficient testing, we place a production tag on an image that was
>>>>>>>>>> already pushed with a dev tag.
>>>>>>>>>>
>>>>>>>>>> For example a dev tag may look like:
>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may
>>>>>>>>>> look like:
>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the
>>>>>>>>>> same image at the end.
>>>>>>>>>>
>>>>>>>>>> We should also plan what the process of updating the container
>>>>>>>>>> image will look like, if we need to release the image with
>>>>>>>>>> additional changes, and how we will test these changes before the final
>>>>>>>>>> push (or placing production tag).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *2. How many tests do we need to run to validate pushed images?*
>>>>>>>>>>> When we push the images, we assume the images would work and
>>>>>>>>>>> pass all the tests. After pushing, we should confirm the images are
>>>>>>>>>>> pullable and useable. I suggest we run several tests on dataflow with each
>>>>>>>>>>> pushed image. What do you think?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think it makes sense to do - Beam runners that use SDK
>>>>>>>>>> container images should have some continuously running tests, which
>>>>>>>>>> periodically check that all supported images are pullable and still
>>>>>>>>>> compatible with the runner.
>>>>>>>>>>
>>>>>>>>>> This work can be refined later as we explore more during our
>>>>>>>>>>> release process.
>>>>>>>>>>> Please comment or edit the wiki page or reply to this email with
>>>>>>>>>>> your opinions.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Hannah
>>>>>>>>>>>
>>>>>>>>>>
Re: Improve container support
Posted by Ankur Goenka <go...@google.com>.
I think sdk in the name is obsolete as they are all under sdks name space.
On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang <ha...@google.com> wrote:
> Hi Team
>
> Thanks for all the comments about beam containers.
> After considering various opinions and investigating gcr and docker hub,
> we decided to push images to docker hub.
>
> Each image will have two tags, {version}_rc and {version}. {version} tag
> will be added after the release candidate image is verified.
> Meanwhile, we will have* latest* tag for each repository, which always
> points to the most recent verified release image, so users can pull it by
> default.
>
> Docker hub doesn't support leveled repository, which means we should
> follow *repository:tag* format.
> it's too general if we use {language_version} as repository for SDK
> images. (version is added when we support multiple versions.)
> So I would like to include *sdk* to repository. Images generated at local
> will also have the same name.
> Here are some examples:
>
> - python2.7_sdk:2.15.0
> - java_sdk:2.15.0_rc
> - go_sdk:latest
>
> I will proceed with this format if there is no strong opposition by
> tomorrow noon(PST).
>
> *To PMC members*:
> Permission control will follow the pypi model. All interested PMC members
> will be added as admins and release managers will be granted push
> permission.
> Please let me know your *docker id* if you want to be added as an admin.
>
> Thanks,
> Hannah
>
>
>
>
>
>
>
>
> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise <th...@apache.org> wrote:
>
>> This will greatly simplify trying out portable runners:
>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>>
>> Can't wait for following to disappear from the instructions page: ./gradlew
>> :sdks:python:container:docker
>>
>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise <th...@apache.org> wrote:
>>
>>> Awesome, thank you!
>>>
>>>
>>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang <ha...@google.com>
>>> wrote:
>>>
>>>> Hi Thomas
>>>>
>>>> I created snapshot images from head as of around 2PM today.
>>>> You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.
>>>>
>>>> Thanks,
>>>> Hannah
>>>>
>>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <th...@apache.org> wrote:
>>>>
>>>>> Hi Hannah,
>>>>>
>>>>> Thank you, I know how to build the containers locally, but not how to
>>>>> publish them!
>>>>>
>>>>> The cwiki says "Publishing images to gcr.io/beam requires permissions
>>>>> in apache-beam-testing project."
>>>>>
>>>>> Can I get access to the testing project (at least temporarily) and
>>>>> what would I need to setup to run the publish target that is shown on cwiki?
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>>
>>>>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <ha...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Thomas
>>>>>>
>>>>>> I haven't uploaded any snapshot images yet. Here is how you can
>>>>>> create one from head.
>>>>>> > cd [...]/beam/
>>>>>> # For Python
>>>>>> > ./gradlew :sdks:python:container:py{version}:docker *where version
>>>>>> is {2,35,36,37}*
>>>>>> # For Java
>>>>>> > ./gradlew -p sdks/java/container docker
>>>>>> # For Go
>>>>>> > ./gradlew -p sdks/go/container docker
>>>>>>
>>>>>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot
>>>>>> from head.
>>>>>>
>>>>>> Please let me know if you have any questions.
>>>>>> Hannah
>>>>>>
>>>>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <th...@apache.org> wrote:
>>>>>>
>>>>>>> I actually found something in [1], but it is 2.15 unfortunately.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>>>>>>
>>>>>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <th...@apache.org> wrote:
>>>>>>>
>>>>>>>> Thanks for working on this. Do you happen to have publicly
>>>>>>>> accessible snapshots published for your testing currently (even when the
>>>>>>>> final location isn't sorted out)?
>>>>>>>>
>>>>>>>> I would like to use a 2.16 based Python SDK image for working on my
>>>>>>>> downstream project, but could not find anything in
>>>>>>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Thomas
>>>>>>>>
>>>>>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>
>>>>>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>>>>>>>>> hannahjiang@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi team
>>>>>>>>>>
>>>>>>>>>> I am working on improving docker container support for Beam. We
>>>>>>>>>> would like to publish prebuilt containers for each release version and
>>>>>>>>>> daily snapshot. Current work focuses on release images only and it would be
>>>>>>>>>> part of the release process.
>>>>>>>>>>
>>>>>>>>>> The release images will be pushed to GCR which is publicly
>>>>>>>>>> accessible(pullable). We will use the following locations.
>>>>>>>>>> *Repository*: gcr.io/beam
>>>>>>>>>> *Project*: apache-beam-testing
>>>>>>>>>> More details, including naming and tagging scheme, can be found
>>>>>>>>>> at wiki
>>>>>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> which
>>>>>>>>>> is written by several contributors.
>>>>>>>>>>
>>>>>>>>>> I would like to discuss these two questions.
>>>>>>>>>> *1. How many tests do we need to run before pushing images to gcr*
>>>>>>>>>> ?
>>>>>>>>>> Publishing artifacts is the last step of the release process, so
>>>>>>>>>> at this moment, we already verified all codebase. In addition, many Jenkins
>>>>>>>>>> tests use containers, so it is already verified several times. Do we need
>>>>>>>>>> to run it again?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In a docker repository, one container image can have multiple
>>>>>>>>> tags. One possibility is that on the last step of the release process,
>>>>>>>>> after sufficient testing, we place a production tag on an image that was
>>>>>>>>> already pushed with a dev tag.
>>>>>>>>>
>>>>>>>>> For example a dev tag may look like:
>>>>>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may
>>>>>>>>> look like:
>>>>>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the
>>>>>>>>> same image at the end.
>>>>>>>>>
>>>>>>>>> We should also plan what the process of updating the container
>>>>>>>>> image will look like, if we need to release the image with
>>>>>>>>> additional changes, and how we will test these changes before the final
>>>>>>>>> push (or placing production tag).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *2. How many tests do we need to run to validate pushed images?*
>>>>>>>>>> When we push the images, we assume the images would work and pass
>>>>>>>>>> all the tests. After pushing, we should confirm the images are pullable and
>>>>>>>>>> useable. I suggest we run several tests on dataflow with each pushed image.
>>>>>>>>>> What do you think?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I think it makes sense to do - Beam runners that use SDK
>>>>>>>>> container images should have some continuously running tests, which
>>>>>>>>> periodically check that all supported images are pullable and still
>>>>>>>>> compatible with the runner.
>>>>>>>>>
>>>>>>>>> This work can be refined later as we explore more during our
>>>>>>>>>> release process.
>>>>>>>>>> Please comment or edit the wiki page or reply to this email with
>>>>>>>>>> your opinions.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Hannah
>>>>>>>>>>
>>>>>>>>>
Re: Improve container support
Posted by Hannah Jiang <ha...@google.com>.
Hi Team
Thanks for all the comments about beam containers.
After considering various opinions and investigating gcr and docker hub, we
decided to push images to docker hub.
Each image will have two tags, {version}_rc and {version}. {version} tag
will be added after the release candidate image is verified.
Meanwhile, we will have* latest* tag for each repository, which always
points to the most recent verified release image, so users can pull it by
default.
Docker hub doesn't support leveled repository, which means we should follow
*repository:tag* format.
it's too general if we use {language_version} as repository for SDK images.
(version is added when we support multiple versions.)
So I would like to include *sdk* to repository. Images generated at local
will also have the same name.
Here are some examples:
- python2.7_sdk:2.15.0
- java_sdk:2.15.0_rc
- go_sdk:latest
I will proceed with this format if there is no strong opposition by
tomorrow noon(PST).
*To PMC members*:
Permission control will follow the pypi model. All interested PMC members
will be added as admins and release managers will be granted push
permission.
Please let me know your *docker id* if you want to be added as an admin.
Thanks,
Hannah
On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise <th...@apache.org> wrote:
> This will greatly simplify trying out portable runners:
> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>
> Can't wait for following to disappear from the instructions page: ./gradlew
> :sdks:python:container:docker
>
> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise <th...@apache.org> wrote:
>
>> Awesome, thank you!
>>
>>
>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang <ha...@google.com>
>> wrote:
>>
>>> Hi Thomas
>>>
>>> I created snapshot images from head as of around 2PM today.
>>> You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.
>>>
>>> Thanks,
>>> Hannah
>>>
>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <th...@apache.org> wrote:
>>>
>>>> Hi Hannah,
>>>>
>>>> Thank you, I know how to build the containers locally, but not how to
>>>> publish them!
>>>>
>>>> The cwiki says "Publishing images to gcr.io/beam requires permissions
>>>> in apache-beam-testing project."
>>>>
>>>> Can I get access to the testing project (at least temporarily) and what
>>>> would I need to setup to run the publish target that is shown on cwiki?
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>>
>>>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <ha...@google.com>
>>>> wrote:
>>>>
>>>>> Hi Thomas
>>>>>
>>>>> I haven't uploaded any snapshot images yet. Here is how you can create
>>>>> one from head.
>>>>> > cd [...]/beam/
>>>>> # For Python
>>>>> > ./gradlew :sdks:python:container:py{version}:docker *where version
>>>>> is {2,35,36,37}*
>>>>> # For Java
>>>>> > ./gradlew -p sdks/java/container docker
>>>>> # For Go
>>>>> > ./gradlew -p sdks/go/container docker
>>>>>
>>>>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot
>>>>> from head.
>>>>>
>>>>> Please let me know if you have any questions.
>>>>> Hannah
>>>>>
>>>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>>> I actually found something in [1], but it is 2.15 unfortunately.
>>>>>>
>>>>>> [1]
>>>>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>>>>>
>>>>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <th...@apache.org> wrote:
>>>>>>
>>>>>>> Thanks for working on this. Do you happen to have publicly
>>>>>>> accessible snapshots published for your testing currently (even when the
>>>>>>> final location isn't sorted out)?
>>>>>>>
>>>>>>> I would like to use a 2.16 based Python SDK image for working on my
>>>>>>> downstream project, but could not find anything in
>>>>>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Thomas
>>>>>>>
>>>>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>>
>>>>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>>>>>>>> hannahjiang@google.com> wrote:
>>>>>>>>
>>>>>>>>> Hi team
>>>>>>>>>
>>>>>>>>> I am working on improving docker container support for Beam. We
>>>>>>>>> would like to publish prebuilt containers for each release version and
>>>>>>>>> daily snapshot. Current work focuses on release images only and it would be
>>>>>>>>> part of the release process.
>>>>>>>>>
>>>>>>>>> The release images will be pushed to GCR which is publicly
>>>>>>>>> accessible(pullable). We will use the following locations.
>>>>>>>>> *Repository*: gcr.io/beam
>>>>>>>>> *Project*: apache-beam-testing
>>>>>>>>> More details, including naming and tagging scheme, can be found at
>>>>>>>>> wiki
>>>>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> which
>>>>>>>>> is written by several contributors.
>>>>>>>>>
>>>>>>>>> I would like to discuss these two questions.
>>>>>>>>> *1. How many tests do we need to run before pushing images to gcr*
>>>>>>>>> ?
>>>>>>>>> Publishing artifacts is the last step of the release process, so
>>>>>>>>> at this moment, we already verified all codebase. In addition, many Jenkins
>>>>>>>>> tests use containers, so it is already verified several times. Do we need
>>>>>>>>> to run it again?
>>>>>>>>>
>>>>>>>>
>>>>>>>> In a docker repository, one container image can have multiple tags.
>>>>>>>> One possibility is that on the last step of the release process, after
>>>>>>>> sufficient testing, we place a production tag on an image that was already
>>>>>>>> pushed with a dev tag.
>>>>>>>>
>>>>>>>> For example a dev tag may look like:
>>>>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may
>>>>>>>> look like:
>>>>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same
>>>>>>>> image at the end.
>>>>>>>>
>>>>>>>> We should also plan what the process of updating the container
>>>>>>>> image will look like, if we need to release the image with
>>>>>>>> additional changes, and how we will test these changes before the final
>>>>>>>> push (or placing production tag).
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> *2. How many tests do we need to run to validate pushed images?*
>>>>>>>>> When we push the images, we assume the images would work and pass
>>>>>>>>> all the tests. After pushing, we should confirm the images are pullable and
>>>>>>>>> useable. I suggest we run several tests on dataflow with each pushed image.
>>>>>>>>> What do you think?
>>>>>>>>>
>>>>>>>>
>>>>>>>> I think it makes sense to do - Beam runners that use SDK container
>>>>>>>> images should have some continuously running tests, which periodically
>>>>>>>> check that all supported images are pullable and still compatible with the
>>>>>>>> runner.
>>>>>>>>
>>>>>>>> This work can be refined later as we explore more during our
>>>>>>>>> release process.
>>>>>>>>> Please comment or edit the wiki page or reply to this email with
>>>>>>>>> your opinions.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Hannah
>>>>>>>>>
>>>>>>>>
Re: Improve container support
Posted by Thomas Weise <th...@apache.org>.
This will greatly simplify trying out portable runners:
https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
Can't wait for following to disappear from the instructions page: ./gradlew
:sdks:python:container:docker
On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise <th...@apache.org> wrote:
> Awesome, thank you!
>
>
> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang <ha...@google.com>
> wrote:
>
>> Hi Thomas
>>
>> I created snapshot images from head as of around 2PM today.
>> You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.
>>
>> Thanks,
>> Hannah
>>
>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <th...@apache.org> wrote:
>>
>>> Hi Hannah,
>>>
>>> Thank you, I know how to build the containers locally, but not how to
>>> publish them!
>>>
>>> The cwiki says "Publishing images to gcr.io/beam requires permissions
>>> in apache-beam-testing project."
>>>
>>> Can I get access to the testing project (at least temporarily) and what
>>> would I need to setup to run the publish target that is shown on cwiki?
>>>
>>> Thanks,
>>> Thomas
>>>
>>>
>>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <ha...@google.com>
>>> wrote:
>>>
>>>> Hi Thomas
>>>>
>>>> I haven't uploaded any snapshot images yet. Here is how you can create
>>>> one from head.
>>>> > cd [...]/beam/
>>>> # For Python
>>>> > ./gradlew :sdks:python:container:py{version}:docker *where version
>>>> is {2,35,36,37}*
>>>> # For Java
>>>> > ./gradlew -p sdks/java/container docker
>>>> # For Go
>>>> > ./gradlew -p sdks/go/container docker
>>>>
>>>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot
>>>> from head.
>>>>
>>>> Please let me know if you have any questions.
>>>> Hannah
>>>>
>>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <th...@apache.org> wrote:
>>>>
>>>>> I actually found something in [1], but it is 2.15 unfortunately.
>>>>>
>>>>> [1]
>>>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>>>>
>>>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>>> Thanks for working on this. Do you happen to have publicly accessible
>>>>>> snapshots published for your testing currently (even when the final
>>>>>> location isn't sorted out)?
>>>>>>
>>>>>> I would like to use a 2.16 based Python SDK image for working on my
>>>>>> downstream project, but could not find anything in
>>>>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>>
>>>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi team
>>>>>>>>
>>>>>>>> I am working on improving docker container support for Beam. We
>>>>>>>> would like to publish prebuilt containers for each release version and
>>>>>>>> daily snapshot. Current work focuses on release images only and it would be
>>>>>>>> part of the release process.
>>>>>>>>
>>>>>>>> The release images will be pushed to GCR which is publicly
>>>>>>>> accessible(pullable). We will use the following locations.
>>>>>>>> *Repository*: gcr.io/beam
>>>>>>>> *Project*: apache-beam-testing
>>>>>>>> More details, including naming and tagging scheme, can be found at
>>>>>>>> wiki
>>>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> which
>>>>>>>> is written by several contributors.
>>>>>>>>
>>>>>>>> I would like to discuss these two questions.
>>>>>>>> *1. How many tests do we need to run before pushing images to gcr*?
>>>>>>>> Publishing artifacts is the last step of the release process, so at
>>>>>>>> this moment, we already verified all codebase. In addition, many Jenkins
>>>>>>>> tests use containers, so it is already verified several times. Do we need
>>>>>>>> to run it again?
>>>>>>>>
>>>>>>>
>>>>>>> In a docker repository, one container image can have multiple tags.
>>>>>>> One possibility is that on the last step of the release process, after
>>>>>>> sufficient testing, we place a production tag on an image that was already
>>>>>>> pushed with a dev tag.
>>>>>>>
>>>>>>> For example a dev tag may look like:
>>>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look
>>>>>>> like:
>>>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same
>>>>>>> image at the end.
>>>>>>>
>>>>>>> We should also plan what the process of updating the container image
>>>>>>> will look like, if we need to release the image with additional changes,
>>>>>>> and how we will test these changes before the final push (or placing
>>>>>>> production tag).
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> *2. How many tests do we need to run to validate pushed images?*
>>>>>>>> When we push the images, we assume the images would work and pass
>>>>>>>> all the tests. After pushing, we should confirm the images are pullable and
>>>>>>>> useable. I suggest we run several tests on dataflow with each pushed image.
>>>>>>>> What do you think?
>>>>>>>>
>>>>>>>
>>>>>>> I think it makes sense to do - Beam runners that use SDK container
>>>>>>> images should have some continuously running tests, which periodically
>>>>>>> check that all supported images are pullable and still compatible with the
>>>>>>> runner.
>>>>>>>
>>>>>>> This work can be refined later as we explore more during our release
>>>>>>>> process.
>>>>>>>> Please comment or edit the wiki page or reply to this email with
>>>>>>>> your opinions.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Hannah
>>>>>>>>
>>>>>>>
Re: Improve container support
Posted by Thomas Weise <th...@apache.org>.
Awesome, thank you!
On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang <ha...@google.com> wrote:
> Hi Thomas
>
> I created snapshot images from head as of around 2PM today.
> You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.
>
> Thanks,
> Hannah
>
> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <th...@apache.org> wrote:
>
>> Hi Hannah,
>>
>> Thank you, I know how to build the containers locally, but not how to
>> publish them!
>>
>> The cwiki says "Publishing images to gcr.io/beam requires permissions in
>> apache-beam-testing project."
>>
>> Can I get access to the testing project (at least temporarily) and what
>> would I need to setup to run the publish target that is shown on cwiki?
>>
>> Thanks,
>> Thomas
>>
>>
>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <ha...@google.com>
>> wrote:
>>
>>> Hi Thomas
>>>
>>> I haven't uploaded any snapshot images yet. Here is how you can create
>>> one from head.
>>> > cd [...]/beam/
>>> # For Python
>>> > ./gradlew :sdks:python:container:py{version}:docker *where version is
>>> {2,35,36,37}*
>>> # For Java
>>> > ./gradlew -p sdks/java/container docker
>>> # For Go
>>> > ./gradlew -p sdks/go/container docker
>>>
>>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot from
>>> head.
>>>
>>> Please let me know if you have any questions.
>>> Hannah
>>>
>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <th...@apache.org> wrote:
>>>
>>>> I actually found something in [1], but it is 2.15 unfortunately.
>>>>
>>>> [1]
>>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>>>
>>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <th...@apache.org> wrote:
>>>>
>>>>> Thanks for working on this. Do you happen to have publicly accessible
>>>>> snapshots published for your testing currently (even when the final
>>>>> location isn't sorted out)?
>>>>>
>>>>> I would like to use a 2.16 based Python SDK image for working on my
>>>>> downstream project, but could not find anything in
>>>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi team
>>>>>>>
>>>>>>> I am working on improving docker container support for Beam. We
>>>>>>> would like to publish prebuilt containers for each release version and
>>>>>>> daily snapshot. Current work focuses on release images only and it would be
>>>>>>> part of the release process.
>>>>>>>
>>>>>>> The release images will be pushed to GCR which is publicly
>>>>>>> accessible(pullable). We will use the following locations.
>>>>>>> *Repository*: gcr.io/beam
>>>>>>> *Project*: apache-beam-testing
>>>>>>> More details, including naming and tagging scheme, can be found at
>>>>>>> wiki
>>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> which
>>>>>>> is written by several contributors.
>>>>>>>
>>>>>>> I would like to discuss these two questions.
>>>>>>> *1. How many tests do we need to run before pushing images to gcr*?
>>>>>>> Publishing artifacts is the last step of the release process, so at
>>>>>>> this moment, we already verified all codebase. In addition, many Jenkins
>>>>>>> tests use containers, so it is already verified several times. Do we need
>>>>>>> to run it again?
>>>>>>>
>>>>>>
>>>>>> In a docker repository, one container image can have multiple tags.
>>>>>> One possibility is that on the last step of the release process, after
>>>>>> sufficient testing, we place a production tag on an image that was already
>>>>>> pushed with a dev tag.
>>>>>>
>>>>>> For example a dev tag may look like:
>>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look
>>>>>> like:
>>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same
>>>>>> image at the end.
>>>>>>
>>>>>> We should also plan what the process of updating the container image
>>>>>> will look like, if we need to release the image with additional changes,
>>>>>> and how we will test these changes before the final push (or placing
>>>>>> production tag).
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> *2. How many tests do we need to run to validate pushed images?*
>>>>>>> When we push the images, we assume the images would work and pass
>>>>>>> all the tests. After pushing, we should confirm the images are pullable and
>>>>>>> useable. I suggest we run several tests on dataflow with each pushed image.
>>>>>>> What do you think?
>>>>>>>
>>>>>>
>>>>>> I think it makes sense to do - Beam runners that use SDK container
>>>>>> images should have some continuously running tests, which periodically
>>>>>> check that all supported images are pullable and still compatible with the
>>>>>> runner.
>>>>>>
>>>>>> This work can be refined later as we explore more during our release
>>>>>>> process.
>>>>>>> Please comment or edit the wiki page or reply to this email with
>>>>>>> your opinions.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Hannah
>>>>>>>
>>>>>>
Re: Improve container support
Posted by Hannah Jiang <ha...@google.com>.
Hi Thomas
I created snapshot images from head as of around 2PM today.
You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.
Thanks,
Hannah
On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <th...@apache.org> wrote:
> Hi Hannah,
>
> Thank you, I know how to build the containers locally, but not how to
> publish them!
>
> The cwiki says "Publishing images to gcr.io/beam requires permissions in
> apache-beam-testing project."
>
> Can I get access to the testing project (at least temporarily) and what
> would I need to setup to run the publish target that is shown on cwiki?
>
> Thanks,
> Thomas
>
>
> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <ha...@google.com>
> wrote:
>
>> Hi Thomas
>>
>> I haven't uploaded any snapshot images yet. Here is how you can create
>> one from head.
>> > cd [...]/beam/
>> # For Python
>> > ./gradlew :sdks:python:container:py{version}:docker *where version is
>> {2,35,36,37}*
>> # For Java
>> > ./gradlew -p sdks/java/container docker
>> # For Go
>> > ./gradlew -p sdks/go/container docker
>>
>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot from
>> head.
>>
>> Please let me know if you have any questions.
>> Hannah
>>
>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <th...@apache.org> wrote:
>>
>>> I actually found something in [1], but it is 2.15 unfortunately.
>>>
>>> [1]
>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>>
>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <th...@apache.org> wrote:
>>>
>>>> Thanks for working on this. Do you happen to have publicly accessible
>>>> snapshots published for your testing currently (even when the final
>>>> location isn't sorted out)?
>>>>
>>>> I would like to use a 2.16 based Python SDK image for working on my
>>>> downstream project, but could not find anything in
>>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>>
>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hi team
>>>>>>
>>>>>> I am working on improving docker container support for Beam. We would
>>>>>> like to publish prebuilt containers for each release version and daily
>>>>>> snapshot. Current work focuses on release images only and it would be part
>>>>>> of the release process.
>>>>>>
>>>>>> The release images will be pushed to GCR which is publicly
>>>>>> accessible(pullable). We will use the following locations.
>>>>>> *Repository*: gcr.io/beam
>>>>>> *Project*: apache-beam-testing
>>>>>> More details, including naming and tagging scheme, can be found at
>>>>>> wiki
>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> which
>>>>>> is written by several contributors.
>>>>>>
>>>>>> I would like to discuss these two questions.
>>>>>> *1. How many tests do we need to run before pushing images to gcr*?
>>>>>> Publishing artifacts is the last step of the release process, so at
>>>>>> this moment, we already verified all codebase. In addition, many Jenkins
>>>>>> tests use containers, so it is already verified several times. Do we need
>>>>>> to run it again?
>>>>>>
>>>>>
>>>>> In a docker repository, one container image can have multiple tags.
>>>>> One possibility is that on the last step of the release process, after
>>>>> sufficient testing, we place a production tag on an image that was already
>>>>> pushed with a dev tag.
>>>>>
>>>>> For example a dev tag may look like:
>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look
>>>>> like:
>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same
>>>>> image at the end.
>>>>>
>>>>> We should also plan what the process of updating the container image
>>>>> will look like, if we need to release the image with additional changes,
>>>>> and how we will test these changes before the final push (or placing
>>>>> production tag).
>>>>>
>>>>>
>>>>>>
>>>>>> *2. How many tests do we need to run to validate pushed images?*
>>>>>> When we push the images, we assume the images would work and pass all
>>>>>> the tests. After pushing, we should confirm the images are pullable and
>>>>>> useable. I suggest we run several tests on dataflow with each pushed image.
>>>>>> What do you think?
>>>>>>
>>>>>
>>>>> I think it makes sense to do - Beam runners that use SDK container
>>>>> images should have some continuously running tests, which periodically
>>>>> check that all supported images are pullable and still compatible with the
>>>>> runner.
>>>>>
>>>>> This work can be refined later as we explore more during our release
>>>>>> process.
>>>>>> Please comment or edit the wiki page or reply to this email with your
>>>>>> opinions.
>>>>>>
>>>>>> Thanks,
>>>>>> Hannah
>>>>>>
>>>>>
Re: Improve container support
Posted by Thomas Weise <th...@apache.org>.
Hi Hannah,
Thank you, I know how to build the containers locally, but not how to
publish them!
The cwiki says "Publishing images to gcr.io/beam requires permissions in
apache-beam-testing project."
Can I get access to the testing project (at least temporarily) and what
would I need to setup to run the publish target that is shown on cwiki?
Thanks,
Thomas
On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <ha...@google.com> wrote:
> Hi Thomas
>
> I haven't uploaded any snapshot images yet. Here is how you can create one
> from head.
> > cd [...]/beam/
> # For Python
> > ./gradlew :sdks:python:container:py{version}:docker *where version is
> {2,35,36,37}*
> # For Java
> > ./gradlew -p sdks/java/container docker
> # For Go
> > ./gradlew -p sdks/go/container docker
>
> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot from
> head.
>
> Please let me know if you have any questions.
> Hannah
>
> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <th...@apache.org> wrote:
>
>> I actually found something in [1], but it is 2.15 unfortunately.
>>
>> [1]
>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>
>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <th...@apache.org> wrote:
>>
>>> Thanks for working on this. Do you happen to have publicly accessible
>>> snapshots published for your testing currently (even when the final
>>> location isn't sorted out)?
>>>
>>> I would like to use a 2.16 based Python SDK image for working on my
>>> downstream project, but could not find anything in
>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>
>>> Thanks,
>>> Thomas
>>>
>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>>> valentyn@google.com> wrote:
>>>
>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com>
>>>> wrote:
>>>>
>>>>> Hi team
>>>>>
>>>>> I am working on improving docker container support for Beam. We would
>>>>> like to publish prebuilt containers for each release version and daily
>>>>> snapshot. Current work focuses on release images only and it would be part
>>>>> of the release process.
>>>>>
>>>>> The release images will be pushed to GCR which is publicly
>>>>> accessible(pullable). We will use the following locations.
>>>>> *Repository*: gcr.io/beam
>>>>> *Project*: apache-beam-testing
>>>>> More details, including naming and tagging scheme, can be found at
>>>>> wiki
>>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> which
>>>>> is written by several contributors.
>>>>>
>>>>> I would like to discuss these two questions.
>>>>> *1. How many tests do we need to run before pushing images to gcr*?
>>>>> Publishing artifacts is the last step of the release process, so at
>>>>> this moment, we already verified all codebase. In addition, many Jenkins
>>>>> tests use containers, so it is already verified several times. Do we need
>>>>> to run it again?
>>>>>
>>>>
>>>> In a docker repository, one container image can have multiple tags. One
>>>> possibility is that on the last step of the release process, after
>>>> sufficient testing, we place a production tag on an image that was already
>>>> pushed with a dev tag.
>>>>
>>>> For example a dev tag may look like:
>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look
>>>> like:
>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same
>>>> image at the end.
>>>>
>>>> We should also plan what the process of updating the container image
>>>> will look like, if we need to release the image with additional changes,
>>>> and how we will test these changes before the final push (or placing
>>>> production tag).
>>>>
>>>>
>>>>>
>>>>> *2. How many tests do we need to run to validate pushed images?*
>>>>> When we push the images, we assume the images would work and pass all
>>>>> the tests. After pushing, we should confirm the images are pullable and
>>>>> useable. I suggest we run several tests on dataflow with each pushed image.
>>>>> What do you think?
>>>>>
>>>>
>>>> I think it makes sense to do - Beam runners that use SDK container
>>>> images should have some continuously running tests, which periodically
>>>> check that all supported images are pullable and still compatible with the
>>>> runner.
>>>>
>>>> This work can be refined later as we explore more during our release
>>>>> process.
>>>>> Please comment or edit the wiki page or reply to this email with your
>>>>> opinions.
>>>>>
>>>>> Thanks,
>>>>> Hannah
>>>>>
>>>>
Re: Improve container support
Posted by Hannah Jiang <ha...@google.com>.
Hi Thomas
I haven't uploaded any snapshot images yet. Here is how you can create one
from head.
> cd [...]/beam/
# For Python
> ./gradlew :sdks:python:container:py{version}:docker *where version is
{2,35,36,37}*
# For Java
> ./gradlew -p sdks/java/container docker
# For Go
> ./gradlew -p sdks/go/container docker
The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot from
head.
Please let me know if you have any questions.
Hannah
On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <th...@apache.org> wrote:
> I actually found something in [1], but it is 2.15 unfortunately.
>
> [1]
> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>
> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <th...@apache.org> wrote:
>
>> Thanks for working on this. Do you happen to have publicly accessible
>> snapshots published for your testing currently (even when the final
>> location isn't sorted out)?
>>
>> I would like to use a 2.16 based Python SDK image for working on my
>> downstream project, but could not find anything in
>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>
>> Thanks,
>> Thomas
>>
>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com>
>>> wrote:
>>>
>>>> Hi team
>>>>
>>>> I am working on improving docker container support for Beam. We would
>>>> like to publish prebuilt containers for each release version and daily
>>>> snapshot. Current work focuses on release images only and it would be part
>>>> of the release process.
>>>>
>>>> The release images will be pushed to GCR which is publicly
>>>> accessible(pullable). We will use the following locations.
>>>> *Repository*: gcr.io/beam
>>>> *Project*: apache-beam-testing
>>>> More details, including naming and tagging scheme, can be found at wiki
>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> which
>>>> is written by several contributors.
>>>>
>>>> I would like to discuss these two questions.
>>>> *1. How many tests do we need to run before pushing images to gcr*?
>>>> Publishing artifacts is the last step of the release process, so at
>>>> this moment, we already verified all codebase. In addition, many Jenkins
>>>> tests use containers, so it is already verified several times. Do we need
>>>> to run it again?
>>>>
>>>
>>> In a docker repository, one container image can have multiple tags. One
>>> possibility is that on the last step of the release process, after
>>> sufficient testing, we place a production tag on an image that was already
>>> pushed with a dev tag.
>>>
>>> For example a dev tag may look like:
>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look
>>> like:
>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same
>>> image at the end.
>>>
>>> We should also plan what the process of updating the container image
>>> will look like, if we need to release the image with additional changes,
>>> and how we will test these changes before the final push (or placing
>>> production tag).
>>>
>>>
>>>>
>>>> *2. How many tests do we need to run to validate pushed images?*
>>>> When we push the images, we assume the images would work and pass all
>>>> the tests. After pushing, we should confirm the images are pullable and
>>>> useable. I suggest we run several tests on dataflow with each pushed image.
>>>> What do you think?
>>>>
>>>
>>> I think it makes sense to do - Beam runners that use SDK container
>>> images should have some continuously running tests, which periodically
>>> check that all supported images are pullable and still compatible with the
>>> runner.
>>>
>>> This work can be refined later as we explore more during our release
>>>> process.
>>>> Please comment or edit the wiki page or reply to this email with your
>>>> opinions.
>>>>
>>>> Thanks,
>>>> Hannah
>>>>
>>>
Re: Improve container support
Posted by Thomas Weise <th...@apache.org>.
I actually found something in [1], but it is 2.15 unfortunately.
[1]
https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <th...@apache.org> wrote:
> Thanks for working on this. Do you happen to have publicly accessible
> snapshots published for your testing currently (even when the final
> location isn't sorted out)?
>
> I would like to use a 2.16 based Python SDK image for working on my
> downstream project, but could not find anything in
> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>
> Thanks,
> Thomas
>
> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com>
>> wrote:
>>
>>> Hi team
>>>
>>> I am working on improving docker container support for Beam. We would
>>> like to publish prebuilt containers for each release version and daily
>>> snapshot. Current work focuses on release images only and it would be part
>>> of the release process.
>>>
>>> The release images will be pushed to GCR which is publicly
>>> accessible(pullable). We will use the following locations.
>>> *Repository*: gcr.io/beam
>>> *Project*: apache-beam-testing
>>> More details, including naming and tagging scheme, can be found at wiki
>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> which
>>> is written by several contributors.
>>>
>>> I would like to discuss these two questions.
>>> *1. How many tests do we need to run before pushing images to gcr*?
>>> Publishing artifacts is the last step of the release process, so at this
>>> moment, we already verified all codebase. In addition, many Jenkins tests
>>> use containers, so it is already verified several times. Do we need to run
>>> it again?
>>>
>>
>> In a docker repository, one container image can have multiple tags. One
>> possibility is that on the last step of the release process, after
>> sufficient testing, we place a production tag on an image that was already
>> pushed with a dev tag.
>>
>> For example a dev tag may look like:
>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look
>> like:
>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same image
>> at the end.
>>
>> We should also plan what the process of updating the container image will
>> look like, if we need to release the image with additional changes, and how
>> we will test these changes before the final push (or placing production
>> tag).
>>
>>
>>>
>>> *2. How many tests do we need to run to validate pushed images?*
>>> When we push the images, we assume the images would work and pass all
>>> the tests. After pushing, we should confirm the images are pullable and
>>> useable. I suggest we run several tests on dataflow with each pushed image.
>>> What do you think?
>>>
>>
>> I think it makes sense to do - Beam runners that use SDK container
>> images should have some continuously running tests, which periodically
>> check that all supported images are pullable and still compatible with the
>> runner.
>>
>> This work can be refined later as we explore more during our release
>>> process.
>>> Please comment or edit the wiki page or reply to this email with your
>>> opinions.
>>>
>>> Thanks,
>>> Hannah
>>>
>>
Re: Improve container support
Posted by Thomas Weise <th...@apache.org>.
Thanks for working on this. Do you happen to have publicly accessible
snapshots published for your testing currently (even when the final
location isn't sorted out)?
I would like to use a 2.16 based Python SDK image for working on my
downstream project, but could not find anything in
gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
Thanks,
Thomas
On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <va...@google.com>
wrote:
> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com>
> wrote:
>
>> Hi team
>>
>> I am working on improving docker container support for Beam. We would
>> like to publish prebuilt containers for each release version and daily
>> snapshot. Current work focuses on release images only and it would be part
>> of the release process.
>>
>> The release images will be pushed to GCR which is publicly
>> accessible(pullable). We will use the following locations.
>> *Repository*: gcr.io/beam
>> *Project*: apache-beam-testing
>> More details, including naming and tagging scheme, can be found at wiki
>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> which
>> is written by several contributors.
>>
>> I would like to discuss these two questions.
>> *1. How many tests do we need to run before pushing images to gcr*?
>> Publishing artifacts is the last step of the release process, so at this
>> moment, we already verified all codebase. In addition, many Jenkins tests
>> use containers, so it is already verified several times. Do we need to run
>> it again?
>>
>
> In a docker repository, one container image can have multiple tags. One
> possibility is that on the last step of the release process, after
> sufficient testing, we place a production tag on an image that was already
> pushed with a dev tag.
>
> For example a dev tag may look like:
> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look like:
> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same image
> at the end.
>
> We should also plan what the process of updating the container image will
> look like, if we need to release the image with additional changes, and how
> we will test these changes before the final push (or placing production
> tag).
>
>
>>
>> *2. How many tests do we need to run to validate pushed images?*
>> When we push the images, we assume the images would work and pass all the
>> tests. After pushing, we should confirm the images are pullable and
>> useable. I suggest we run several tests on dataflow with each pushed image.
>> What do you think?
>>
>
> I think it makes sense to do - Beam runners that use SDK container images
> should have some continuously running tests, which periodically check that
> all supported images are pullable and still compatible with the runner.
>
> This work can be refined later as we explore more during our release
>> process.
>> Please comment or edit the wiki page or reply to this email with your
>> opinions.
>>
>> Thanks,
>> Hannah
>>
>
Re: Improve container support
Posted by Valentyn Tymofieiev <va...@google.com>.
On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com> wrote:
> Hi team
>
> I am working on improving docker container support for Beam. We would like
> to publish prebuilt containers for each release version and daily snapshot.
> Current work focuses on release images only and it would be part of the
> release process.
>
> The release images will be pushed to GCR which is publicly
> accessible(pullable). We will use the following locations.
> *Repository*: gcr.io/beam
> *Project*: apache-beam-testing
> More details, including naming and tagging scheme, can be found at wiki
> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> which
> is written by several contributors.
>
> I would like to discuss these two questions.
> *1. How many tests do we need to run before pushing images to gcr*?
> Publishing artifacts is the last step of the release process, so at this
> moment, we already verified all codebase. In addition, many Jenkins tests
> use containers, so it is already verified several times. Do we need to run
> it again?
>
In a docker repository, one container image can have multiple tags. One
possibility is that on the last step of the release process, after
sufficient testing, we place a production tag on an image that was already
pushed with a dev tag.
For example a dev tag may look like:
gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look like:
gcr.io/apache-beam/python37:2.16.0 and both will refer to the same image at
the end.
We should also plan what the process of updating the container image will
look like, if we need to release the image with additional changes, and how
we will test these changes before the final push (or placing production
tag).
>
> *2. How many tests do we need to run to validate pushed images?*
> When we push the images, we assume the images would work and pass all the
> tests. After pushing, we should confirm the images are pullable and
> useable. I suggest we run several tests on dataflow with each pushed image.
> What do you think?
>
I think it makes sense to do - Beam runners that use SDK container images
should have some continuously running tests, which periodically check that
all supported images are pullable and still compatible with the runner.
This work can be refined later as we explore more during our release
> process.
> Please comment or edit the wiki page or reply to this email with your
> opinions.
>
> Thanks,
> Hannah
>
Re: Improve container support
Posted by Ahmet Altay <al...@google.com>.
+1 to Pablo's comment. I believe any place that offers this service would
be fine and we would need to figure out how to set it up correctly. Our
pypi model seems to work fine so far (PMC members are owners, release
managers are added as maintainers as needed.). To me, the important part is
that we make progress so that we can add the containers to the release
process. We postponed this decision in the past 2 releases because the
conversation started late along with the release branch cut emails.
If docker hub is the defacto place let's use that. bintray, or gcr.io (with
a new GCP project) also sounds like good options. I am impartial to the
choice of the service. Does anyone have a strong preference here?
Ahmet
On Wed, Aug 28, 2019 at 8:25 AM Pablo Estrada <pa...@google.com> wrote:
> A question about the repository we're using. It seems that Docker Hub is
> the de-facto repo for docker images? Is GCR pretty much the same in terms
> of access, authentication, etc?
> We don't have to figure out the repository immediately, and it's fine to
> iterate - but I wanted to make sure we think about that : )
>
> On Wed, Aug 28, 2019 at 6:08 AM Lukasz Cwik <lc...@google.com> wrote:
>
>> Google locks down docs created wtih @google.com addresses. Hannah please
>> recreate the doc using a non @google.com address and share it with the
>> community. You'll want to replace Google short link with an Apache short
>> link (s.apache.org).
>>
>> On Wed, Aug 28, 2019 at 5:40 AM Gleb Kanterov <gl...@spotify.com> wrote:
>>
>>> Google Doc doesn't seem to be shared with dev@. Can anybody
>>> double-check?
>>>
>>> On Wed, Aug 28, 2019 at 7:36 AM Hannah Jiang <ha...@google.com>
>>> wrote:
>>>
>>>> add dev@
>>>>
>>>> On Tue, Aug 27, 2019 at 9:29 PM Hannah Jiang <ha...@google.com>
>>>> wrote:
>>>>
>>>>> Thanks for commenting and discussions.
>>>>> I created a Google Docs
>>>>> <https://docs.google.com/document/d/1LOraxUHdmuykgOPsjRC00iDqvSapoCuCjZcZm-Vh8o0/edit?usp=sharing> for
>>>>> easy commenting and reviewing. From this moment, all changes will be
>>>>> updated to the Google Docs and I will sync to wiki after finalize all plans.
>>>>>
>>>>> Thanks,
>>>>> Hannah
>>>>>
>>>>> On Tue, Aug 27, 2019 at 9:24 PM Ahmet Altay <al...@google.com> wrote:
>>>>>
>>>>>> Hi datapls-engprod,
>>>>>>
>>>>>> I have a question. Do you know what would it take to create a new gcp
>>>>>> project similar to apache-beam-testing for purposes of distributing gcr
>>>>>> packages? We can use the same billing account.
>>>>>>
>>>>>> Hannah, Robert, depending on the complexity of creating another gcp
>>>>>> project we can go with that, or simply create a new bintray account. Either
>>>>>> way would give us a clean new project to publish artifacts.
>>>>>>
>>>>>> Ahmet
>>>>>>
>>>>>> ---------- Forwarded message ---------
>>>>>> From: Robert Bradshaw <ro...@google.com>
>>>>>> Date: Tue, Aug 27, 2019 at 6:48 PM
>>>>>> Subject: Re: Improve container support
>>>>>> To: dev <de...@beam.apache.org>
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 27, 2019 at 6:20 PM Ahmet Altay <al...@google.com> wrote:
>>>>>> >
>>>>>> > On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw <
>>>>>> robertwb@google.com> wrote:
>>>>>> >>
>>>>>> >> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>>>>>> hannahjiang@google.com> wrote:
>>>>>> >> >
>>>>>> >> > Hi team
>>>>>> >> >
>>>>>> >> > I am working on improving docker container support for Beam. We
>>>>>> would like to publish prebuilt containers for each release version and
>>>>>> daily snapshot. Current work focuses on release images only and it would be
>>>>>> part of the release process.
>>>>>> >>
>>>>>> >> This would be great!
>>>>>> >>
>>>>>> >> > The release images will be pushed to GCR which is publicly
>>>>>> accessible(pullable). We will use the following locations.
>>>>>> >> > Repository: gcr.io/beam
>>>>>> >> > Project: apache-beam-testing
>>>>>> >>
>>>>>> >> Given that these are release artifacts, we should use a project
>>>>>> with
>>>>>> >> more restricted access than "anyone who opens a PR on github."
>>>>>> >
>>>>>> >
>>>>>> > We have two options:
>>>>>> > - gcr.io works based on the permissions of the gcs bucket that is
>>>>>> backing it. GCS supports bucket only permissions. These permissions needs
>>>>>> to be explicitly granted and the service accounts used by jenkins jobs does
>>>>>> not have these explicit permissions today.
>>>>>> > - we can create a new project in gcr, bintray or anything else that
>>>>>> offers the same service.
>>>>>>
>>>>>> I think the cleanest is to simply have a new project whose membership
>>>>>> consists of (interested) PMC members. If we have to populate this
>>>>>> manually I think that'd still be OK as the churn is quite low.
>>>>>>
>>>>>
>>>
>>> --
>>> Cheers,
>>> Gleb
>>>
>>
Re: Improve container support
Posted by Pablo Estrada <pa...@google.com>.
A question about the repository we're using. It seems that Docker Hub is
the de-facto repo for docker images? Is GCR pretty much the same in terms
of access, authentication, etc?
We don't have to figure out the repository immediately, and it's fine to
iterate - but I wanted to make sure we think about that : )
On Wed, Aug 28, 2019 at 6:08 AM Lukasz Cwik <lc...@google.com> wrote:
> Google locks down docs created wtih @google.com addresses. Hannah please
> recreate the doc using a non @google.com address and share it with the
> community. You'll want to replace Google short link with an Apache short
> link (s.apache.org).
>
> On Wed, Aug 28, 2019 at 5:40 AM Gleb Kanterov <gl...@spotify.com> wrote:
>
>> Google Doc doesn't seem to be shared with dev@. Can anybody double-check?
>>
>> On Wed, Aug 28, 2019 at 7:36 AM Hannah Jiang <ha...@google.com>
>> wrote:
>>
>>> add dev@
>>>
>>> On Tue, Aug 27, 2019 at 9:29 PM Hannah Jiang <ha...@google.com>
>>> wrote:
>>>
>>>> Thanks for commenting and discussions.
>>>> I created a Google Docs
>>>> <https://docs.google.com/document/d/1LOraxUHdmuykgOPsjRC00iDqvSapoCuCjZcZm-Vh8o0/edit?usp=sharing> for
>>>> easy commenting and reviewing. From this moment, all changes will be
>>>> updated to the Google Docs and I will sync to wiki after finalize all plans.
>>>>
>>>> Thanks,
>>>> Hannah
>>>>
>>>> On Tue, Aug 27, 2019 at 9:24 PM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>>> Hi datapls-engprod,
>>>>>
>>>>> I have a question. Do you know what would it take to create a new gcp
>>>>> project similar to apache-beam-testing for purposes of distributing gcr
>>>>> packages? We can use the same billing account.
>>>>>
>>>>> Hannah, Robert, depending on the complexity of creating another gcp
>>>>> project we can go with that, or simply create a new bintray account. Either
>>>>> way would give us a clean new project to publish artifacts.
>>>>>
>>>>> Ahmet
>>>>>
>>>>> ---------- Forwarded message ---------
>>>>> From: Robert Bradshaw <ro...@google.com>
>>>>> Date: Tue, Aug 27, 2019 at 6:48 PM
>>>>> Subject: Re: Improve container support
>>>>> To: dev <de...@beam.apache.org>
>>>>>
>>>>>
>>>>> On Tue, Aug 27, 2019 at 6:20 PM Ahmet Altay <al...@google.com> wrote:
>>>>> >
>>>>> > On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw <ro...@google.com>
>>>>> wrote:
>>>>> >>
>>>>> >> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>>>>> hannahjiang@google.com> wrote:
>>>>> >> >
>>>>> >> > Hi team
>>>>> >> >
>>>>> >> > I am working on improving docker container support for Beam. We
>>>>> would like to publish prebuilt containers for each release version and
>>>>> daily snapshot. Current work focuses on release images only and it would be
>>>>> part of the release process.
>>>>> >>
>>>>> >> This would be great!
>>>>> >>
>>>>> >> > The release images will be pushed to GCR which is publicly
>>>>> accessible(pullable). We will use the following locations.
>>>>> >> > Repository: gcr.io/beam
>>>>> >> > Project: apache-beam-testing
>>>>> >>
>>>>> >> Given that these are release artifacts, we should use a project with
>>>>> >> more restricted access than "anyone who opens a PR on github."
>>>>> >
>>>>> >
>>>>> > We have two options:
>>>>> > - gcr.io works based on the permissions of the gcs bucket that is
>>>>> backing it. GCS supports bucket only permissions. These permissions needs
>>>>> to be explicitly granted and the service accounts used by jenkins jobs does
>>>>> not have these explicit permissions today.
>>>>> > - we can create a new project in gcr, bintray or anything else that
>>>>> offers the same service.
>>>>>
>>>>> I think the cleanest is to simply have a new project whose membership
>>>>> consists of (interested) PMC members. If we have to populate this
>>>>> manually I think that'd still be OK as the churn is quite low.
>>>>>
>>>>
>>
>> --
>> Cheers,
>> Gleb
>>
>
Re: Improve container support
Posted by Hannah Jiang <ha...@google.com>.
Hi team
Thanks for discussions and inputs.
It is a kind reminder that I will wait until next Tue (Sep 3) to close the
first round of discussion. I will revise plan according to discussion and
may revisit again if there are something more to discuss.
FYI: We would like to include docker images from v2.16, whose cutoff date
is Sep 11.
Thanks,
Hannah
On Wed, Aug 28, 2019 at 4:39 PM Sam Bourne <sa...@gmail.com> wrote:
> If docker hub is the defacto place let's use that. bintray, or gcr.io (with
>> a new GCP project) also sounds like good options. I am impartial to the
>> choice of the service. Does anyone have a strong preference here?
>
>
> Not a strong preference, but it would simplify stuff for us if you choose
> dockerhub to limit the number of places we need access to behind our
> firewall. Also, the "Official Docker Images" of Flink are there
> https://hub.docker.com/_/flink.
>
> On Thu, Aug 29, 2019 at 6:45 AM Hannah Jiang <ha...@google.com>
> wrote:
>
>> Thanks for letting me know about acces issue and sharing solution.
>> Here I created a new one
>> <https://docs.google.com/document/d/1IKE_aEkrAzkzUE4pD_r_zVuL5amHGetJ1efnbTfmunM/edit#heading=h.5irk4csrpu0y>
>> with gmail.com account.
>> Please let me know if you still see the problems.
>>
>> On Wed, Aug 28, 2019 at 6:08 AM Lukasz Cwik <lc...@google.com> wrote:
>>
>>> Google locks down docs created wtih @google.com addresses. Hannah
>>> please recreate the doc using a non @google.com address and share it
>>> with the community. You'll want to replace Google short link with an Apache
>>> short link (s.apache.org).
>>>
>>> On Wed, Aug 28, 2019 at 5:40 AM Gleb Kanterov <gl...@spotify.com> wrote:
>>>
>>>> Google Doc doesn't seem to be shared with dev@. Can anybody
>>>> double-check?
>>>>
>>>> On Wed, Aug 28, 2019 at 7:36 AM Hannah Jiang <ha...@google.com>
>>>> wrote:
>>>>
>>>>> add dev@
>>>>>
>>>>> On Tue, Aug 27, 2019 at 9:29 PM Hannah Jiang <ha...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks for commenting and discussions.
>>>>>> I created a Google Docs
>>>>>> <https://docs.google.com/document/d/1LOraxUHdmuykgOPsjRC00iDqvSapoCuCjZcZm-Vh8o0/edit?usp=sharing> for
>>>>>> easy commenting and reviewing. From this moment, all changes will be
>>>>>> updated to the Google Docs and I will sync to wiki after finalize all plans.
>>>>>>
>>>>>> Thanks,
>>>>>> Hannah
>>>>>>
>>>>>> On Tue, Aug 27, 2019 at 9:24 PM Ahmet Altay <al...@google.com> wrote:
>>>>>>
>>>>>>> Hi datapls-engprod,
>>>>>>>
>>>>>>> I have a question. Do you know what would it take to create a new
>>>>>>> gcp project similar to apache-beam-testing for purposes of distributing gcr
>>>>>>> packages? We can use the same billing account.
>>>>>>>
>>>>>>> Hannah, Robert, depending on the complexity of creating another gcp
>>>>>>> project we can go with that, or simply create a new bintray account. Either
>>>>>>> way would give us a clean new project to publish artifacts.
>>>>>>>
>>>>>>> Ahmet
>>>>>>>
>>>>>>> ---------- Forwarded message ---------
>>>>>>> From: Robert Bradshaw <ro...@google.com>
>>>>>>> Date: Tue, Aug 27, 2019 at 6:48 PM
>>>>>>> Subject: Re: Improve container support
>>>>>>> To: dev <de...@beam.apache.org>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 27, 2019 at 6:20 PM Ahmet Altay <al...@google.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw <
>>>>>>> robertwb@google.com> wrote:
>>>>>>> >>
>>>>>>> >> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>>>>>>> hannahjiang@google.com> wrote:
>>>>>>> >> >
>>>>>>> >> > Hi team
>>>>>>> >> >
>>>>>>> >> > I am working on improving docker container support for Beam. We
>>>>>>> would like to publish prebuilt containers for each release version and
>>>>>>> daily snapshot. Current work focuses on release images only and it would be
>>>>>>> part of the release process.
>>>>>>> >>
>>>>>>> >> This would be great!
>>>>>>> >>
>>>>>>> >> > The release images will be pushed to GCR which is publicly
>>>>>>> accessible(pullable). We will use the following locations.
>>>>>>> >> > Repository: gcr.io/beam
>>>>>>> >> > Project: apache-beam-testing
>>>>>>> >>
>>>>>>> >> Given that these are release artifacts, we should use a project
>>>>>>> with
>>>>>>> >> more restricted access than "anyone who opens a PR on github."
>>>>>>> >
>>>>>>> >
>>>>>>> > We have two options:
>>>>>>> > - gcr.io works based on the permissions of the gcs bucket that
>>>>>>> is backing it. GCS supports bucket only permissions. These permissions
>>>>>>> needs to be explicitly granted and the service accounts used by jenkins
>>>>>>> jobs does not have these explicit permissions today.
>>>>>>> > - we can create a new project in gcr, bintray or anything else
>>>>>>> that offers the same service.
>>>>>>>
>>>>>>> I think the cleanest is to simply have a new project whose membership
>>>>>>> consists of (interested) PMC members. If we have to populate this
>>>>>>> manually I think that'd still be OK as the churn is quite low.
>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Gleb
>>>>
>>>
Re: Improve container support
Posted by Sam Bourne <sa...@gmail.com>.
>
> If docker hub is the defacto place let's use that. bintray, or gcr.io (with
> a new GCP project) also sounds like good options. I am impartial to the
> choice of the service. Does anyone have a strong preference here?
Not a strong preference, but it would simplify stuff for us if you choose
dockerhub to limit the number of places we need access to behind our
firewall. Also, the "Official Docker Images" of Flink are there
https://hub.docker.com/_/flink.
On Thu, Aug 29, 2019 at 6:45 AM Hannah Jiang <ha...@google.com> wrote:
> Thanks for letting me know about acces issue and sharing solution.
> Here I created a new one
> <https://docs.google.com/document/d/1IKE_aEkrAzkzUE4pD_r_zVuL5amHGetJ1efnbTfmunM/edit#heading=h.5irk4csrpu0y>
> with gmail.com account.
> Please let me know if you still see the problems.
>
> On Wed, Aug 28, 2019 at 6:08 AM Lukasz Cwik <lc...@google.com> wrote:
>
>> Google locks down docs created wtih @google.com addresses. Hannah please
>> recreate the doc using a non @google.com address and share it with the
>> community. You'll want to replace Google short link with an Apache short
>> link (s.apache.org).
>>
>> On Wed, Aug 28, 2019 at 5:40 AM Gleb Kanterov <gl...@spotify.com> wrote:
>>
>>> Google Doc doesn't seem to be shared with dev@. Can anybody
>>> double-check?
>>>
>>> On Wed, Aug 28, 2019 at 7:36 AM Hannah Jiang <ha...@google.com>
>>> wrote:
>>>
>>>> add dev@
>>>>
>>>> On Tue, Aug 27, 2019 at 9:29 PM Hannah Jiang <ha...@google.com>
>>>> wrote:
>>>>
>>>>> Thanks for commenting and discussions.
>>>>> I created a Google Docs
>>>>> <https://docs.google.com/document/d/1LOraxUHdmuykgOPsjRC00iDqvSapoCuCjZcZm-Vh8o0/edit?usp=sharing> for
>>>>> easy commenting and reviewing. From this moment, all changes will be
>>>>> updated to the Google Docs and I will sync to wiki after finalize all plans.
>>>>>
>>>>> Thanks,
>>>>> Hannah
>>>>>
>>>>> On Tue, Aug 27, 2019 at 9:24 PM Ahmet Altay <al...@google.com> wrote:
>>>>>
>>>>>> Hi datapls-engprod,
>>>>>>
>>>>>> I have a question. Do you know what would it take to create a new gcp
>>>>>> project similar to apache-beam-testing for purposes of distributing gcr
>>>>>> packages? We can use the same billing account.
>>>>>>
>>>>>> Hannah, Robert, depending on the complexity of creating another gcp
>>>>>> project we can go with that, or simply create a new bintray account. Either
>>>>>> way would give us a clean new project to publish artifacts.
>>>>>>
>>>>>> Ahmet
>>>>>>
>>>>>> ---------- Forwarded message ---------
>>>>>> From: Robert Bradshaw <ro...@google.com>
>>>>>> Date: Tue, Aug 27, 2019 at 6:48 PM
>>>>>> Subject: Re: Improve container support
>>>>>> To: dev <de...@beam.apache.org>
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 27, 2019 at 6:20 PM Ahmet Altay <al...@google.com> wrote:
>>>>>> >
>>>>>> > On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw <
>>>>>> robertwb@google.com> wrote:
>>>>>> >>
>>>>>> >> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>>>>>> hannahjiang@google.com> wrote:
>>>>>> >> >
>>>>>> >> > Hi team
>>>>>> >> >
>>>>>> >> > I am working on improving docker container support for Beam. We
>>>>>> would like to publish prebuilt containers for each release version and
>>>>>> daily snapshot. Current work focuses on release images only and it would be
>>>>>> part of the release process.
>>>>>> >>
>>>>>> >> This would be great!
>>>>>> >>
>>>>>> >> > The release images will be pushed to GCR which is publicly
>>>>>> accessible(pullable). We will use the following locations.
>>>>>> >> > Repository: gcr.io/beam
>>>>>> >> > Project: apache-beam-testing
>>>>>> >>
>>>>>> >> Given that these are release artifacts, we should use a project
>>>>>> with
>>>>>> >> more restricted access than "anyone who opens a PR on github."
>>>>>> >
>>>>>> >
>>>>>> > We have two options:
>>>>>> > - gcr.io works based on the permissions of the gcs bucket that is
>>>>>> backing it. GCS supports bucket only permissions. These permissions needs
>>>>>> to be explicitly granted and the service accounts used by jenkins jobs does
>>>>>> not have these explicit permissions today.
>>>>>> > - we can create a new project in gcr, bintray or anything else that
>>>>>> offers the same service.
>>>>>>
>>>>>> I think the cleanest is to simply have a new project whose membership
>>>>>> consists of (interested) PMC members. If we have to populate this
>>>>>> manually I think that'd still be OK as the churn is quite low.
>>>>>>
>>>>>
>>>
>>> --
>>> Cheers,
>>> Gleb
>>>
>>
Re: Improve container support
Posted by Hannah Jiang <ha...@google.com>.
Thanks for letting me know about acces issue and sharing solution.
Here I created a new one with gmail.com account.
Please let me know if you still see the problems.
> On Wed, Aug 28, 2019 at 6:08 AM Lukasz Cwik <lc...@google.com> wrote:
> Google locks down docs created wtih @google.com addresses. Hannah please recreate the doc using a non @google.com address and share it with the community. You'll want to replace Google short link with an Apache short link (s.apache.org).
>
>> On Wed, Aug 28, 2019 at 5:40 AM Gleb Kanterov <gl...@spotify.com> wrote:
>> Google Doc doesn't seem to be shared with dev@. Can anybody double-check?
>>
>>> On Wed, Aug 28, 2019 at 7:36 AM Hannah Jiang <ha...@google.com> wrote:
>>> add dev@
>>>
>>>> On Tue, Aug 27, 2019 at 9:29 PM Hannah Jiang <ha...@google.com> wrote:
>>>> Thanks for commenting and discussions.
>>>> I created a Google Docs for easy commenting and reviewing. From this moment, all changes will be updated to the Google Docs and I will sync to wiki after finalize all plans.
>>>>
>>>> Thanks,
>>>> Hannah
>>>>
>>>>> On Tue, Aug 27, 2019 at 9:24 PM Ahmet Altay <al...@google.com> wrote:
>>>>> Hi datapls-engprod,
>>>>>
>>>>> I have a question. Do you know what would it take to create a new gcp project similar to apache-beam-testing for purposes of distributing gcr packages? We can use the same billing account.
>>>>>
>>>>> Hannah, Robert, depending on the complexity of creating another gcp project we can go with that, or simply create a new bintray account. Either way would give us a clean new project to publish artifacts.
>>>>>
>>>>> Ahmet
>>>>>
>>>>> ---------- Forwarded message ---------
>>>>> From: Robert Bradshaw <ro...@google.com>
>>>>> Date: Tue, Aug 27, 2019 at 6:48 PM
>>>>> Subject: Re: Improve container support
>>>>> To: dev <de...@beam.apache.org>
>>>>>
>>>>>
>>>>> On Tue, Aug 27, 2019 at 6:20 PM Ahmet Altay <al...@google.com> wrote:
>>>>> >
>>>>> > On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw <ro...@google.com> wrote:
>>>>> >>
>>>>> >> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com> wrote:
>>>>> >> >
>>>>> >> > Hi team
>>>>> >> >
>>>>> >> > I am working on improving docker container support for Beam. We would like to publish prebuilt containers for each release version and daily snapshot. Current work focuses on release images only and it would be part of the release process.
>>>>> >>
>>>>> >> This would be great!
>>>>> >>
>>>>> >> > The release images will be pushed to GCR which is publicly accessible(pullable). We will use the following locations.
>>>>> >> > Repository: gcr.io/beam
>>>>> >> > Project: apache-beam-testing
>>>>> >>
>>>>> >> Given that these are release artifacts, we should use a project with
>>>>> >> more restricted access than "anyone who opens a PR on github."
>>>>> >
>>>>> >
>>>>> > We have two options:
>>>>> > - gcr.io works based on the permissions of the gcs bucket that is backing it. GCS supports bucket only permissions. These permissions needs to be explicitly granted and the service accounts used by jenkins jobs does not have these explicit permissions today.
>>>>> > - we can create a new project in gcr, bintray or anything else that offers the same service.
>>>>>
>>>>> I think the cleanest is to simply have a new project whose membership
>>>>> consists of (interested) PMC members. If we have to populate this
>>>>> manually I think that'd still be OK as the churn is quite low.
>>
>>
>> --
>> Cheers,
>> Gleb
Re: Improve container support
Posted by Lukasz Cwik <lc...@google.com>.
Google locks down docs created wtih @google.com addresses. Hannah please
recreate the doc using a non @google.com address and share it with the
community. You'll want to replace Google short link with an Apache short
link (s.apache.org).
On Wed, Aug 28, 2019 at 5:40 AM Gleb Kanterov <gl...@spotify.com> wrote:
> Google Doc doesn't seem to be shared with dev@. Can anybody double-check?
>
> On Wed, Aug 28, 2019 at 7:36 AM Hannah Jiang <ha...@google.com>
> wrote:
>
>> add dev@
>>
>> On Tue, Aug 27, 2019 at 9:29 PM Hannah Jiang <ha...@google.com>
>> wrote:
>>
>>> Thanks for commenting and discussions.
>>> I created a Google Docs
>>> <https://docs.google.com/document/d/1LOraxUHdmuykgOPsjRC00iDqvSapoCuCjZcZm-Vh8o0/edit?usp=sharing> for
>>> easy commenting and reviewing. From this moment, all changes will be
>>> updated to the Google Docs and I will sync to wiki after finalize all plans.
>>>
>>> Thanks,
>>> Hannah
>>>
>>> On Tue, Aug 27, 2019 at 9:24 PM Ahmet Altay <al...@google.com> wrote:
>>>
>>>> Hi datapls-engprod,
>>>>
>>>> I have a question. Do you know what would it take to create a new gcp
>>>> project similar to apache-beam-testing for purposes of distributing gcr
>>>> packages? We can use the same billing account.
>>>>
>>>> Hannah, Robert, depending on the complexity of creating another gcp
>>>> project we can go with that, or simply create a new bintray account. Either
>>>> way would give us a clean new project to publish artifacts.
>>>>
>>>> Ahmet
>>>>
>>>> ---------- Forwarded message ---------
>>>> From: Robert Bradshaw <ro...@google.com>
>>>> Date: Tue, Aug 27, 2019 at 6:48 PM
>>>> Subject: Re: Improve container support
>>>> To: dev <de...@beam.apache.org>
>>>>
>>>>
>>>> On Tue, Aug 27, 2019 at 6:20 PM Ahmet Altay <al...@google.com> wrote:
>>>> >
>>>> > On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw <ro...@google.com>
>>>> wrote:
>>>> >>
>>>> >> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com>
>>>> wrote:
>>>> >> >
>>>> >> > Hi team
>>>> >> >
>>>> >> > I am working on improving docker container support for Beam. We
>>>> would like to publish prebuilt containers for each release version and
>>>> daily snapshot. Current work focuses on release images only and it would be
>>>> part of the release process.
>>>> >>
>>>> >> This would be great!
>>>> >>
>>>> >> > The release images will be pushed to GCR which is publicly
>>>> accessible(pullable). We will use the following locations.
>>>> >> > Repository: gcr.io/beam
>>>> >> > Project: apache-beam-testing
>>>> >>
>>>> >> Given that these are release artifacts, we should use a project with
>>>> >> more restricted access than "anyone who opens a PR on github."
>>>> >
>>>> >
>>>> > We have two options:
>>>> > - gcr.io works based on the permissions of the gcs bucket that is
>>>> backing it. GCS supports bucket only permissions. These permissions needs
>>>> to be explicitly granted and the service accounts used by jenkins jobs does
>>>> not have these explicit permissions today.
>>>> > - we can create a new project in gcr, bintray or anything else that
>>>> offers the same service.
>>>>
>>>> I think the cleanest is to simply have a new project whose membership
>>>> consists of (interested) PMC members. If we have to populate this
>>>> manually I think that'd still be OK as the churn is quite low.
>>>>
>>>
>
> --
> Cheers,
> Gleb
>
Re: Improve container support
Posted by Gleb Kanterov <gl...@spotify.com>.
Google Doc doesn't seem to be shared with dev@. Can anybody double-check?
On Wed, Aug 28, 2019 at 7:36 AM Hannah Jiang <ha...@google.com> wrote:
> add dev@
>
> On Tue, Aug 27, 2019 at 9:29 PM Hannah Jiang <ha...@google.com>
> wrote:
>
>> Thanks for commenting and discussions.
>> I created a Google Docs
>> <https://docs.google.com/document/d/1LOraxUHdmuykgOPsjRC00iDqvSapoCuCjZcZm-Vh8o0/edit?usp=sharing> for
>> easy commenting and reviewing. From this moment, all changes will be
>> updated to the Google Docs and I will sync to wiki after finalize all plans.
>>
>> Thanks,
>> Hannah
>>
>> On Tue, Aug 27, 2019 at 9:24 PM Ahmet Altay <al...@google.com> wrote:
>>
>>> Hi datapls-engprod,
>>>
>>> I have a question. Do you know what would it take to create a new gcp
>>> project similar to apache-beam-testing for purposes of distributing gcr
>>> packages? We can use the same billing account.
>>>
>>> Hannah, Robert, depending on the complexity of creating another gcp
>>> project we can go with that, or simply create a new bintray account. Either
>>> way would give us a clean new project to publish artifacts.
>>>
>>> Ahmet
>>>
>>> ---------- Forwarded message ---------
>>> From: Robert Bradshaw <ro...@google.com>
>>> Date: Tue, Aug 27, 2019 at 6:48 PM
>>> Subject: Re: Improve container support
>>> To: dev <de...@beam.apache.org>
>>>
>>>
>>> On Tue, Aug 27, 2019 at 6:20 PM Ahmet Altay <al...@google.com> wrote:
>>> >
>>> > On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw <ro...@google.com>
>>> wrote:
>>> >>
>>> >> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com>
>>> wrote:
>>> >> >
>>> >> > Hi team
>>> >> >
>>> >> > I am working on improving docker container support for Beam. We
>>> would like to publish prebuilt containers for each release version and
>>> daily snapshot. Current work focuses on release images only and it would be
>>> part of the release process.
>>> >>
>>> >> This would be great!
>>> >>
>>> >> > The release images will be pushed to GCR which is publicly
>>> accessible(pullable). We will use the following locations.
>>> >> > Repository: gcr.io/beam
>>> >> > Project: apache-beam-testing
>>> >>
>>> >> Given that these are release artifacts, we should use a project with
>>> >> more restricted access than "anyone who opens a PR on github."
>>> >
>>> >
>>> > We have two options:
>>> > - gcr.io works based on the permissions of the gcs bucket that is
>>> backing it. GCS supports bucket only permissions. These permissions needs
>>> to be explicitly granted and the service accounts used by jenkins jobs does
>>> not have these explicit permissions today.
>>> > - we can create a new project in gcr, bintray or anything else that
>>> offers the same service.
>>>
>>> I think the cleanest is to simply have a new project whose membership
>>> consists of (interested) PMC members. If we have to populate this
>>> manually I think that'd still be OK as the churn is quite low.
>>>
>>
--
Cheers,
Gleb
Re: Improve container support
Posted by Hannah Jiang <ha...@google.com>.
add dev@
On Tue, Aug 27, 2019 at 9:29 PM Hannah Jiang <ha...@google.com> wrote:
> Thanks for commenting and discussions.
> I created a Google Docs
> <https://docs.google.com/document/d/1LOraxUHdmuykgOPsjRC00iDqvSapoCuCjZcZm-Vh8o0/edit?usp=sharing> for
> easy commenting and reviewing. From this moment, all changes will be
> updated to the Google Docs and I will sync to wiki after finalize all plans.
>
> Thanks,
> Hannah
>
> On Tue, Aug 27, 2019 at 9:24 PM Ahmet Altay <al...@google.com> wrote:
>
>> Hi datapls-engprod,
>>
>> I have a question. Do you know what would it take to create a new gcp
>> project similar to apache-beam-testing for purposes of distributing gcr
>> packages? We can use the same billing account.
>>
>> Hannah, Robert, depending on the complexity of creating another gcp
>> project we can go with that, or simply create a new bintray account. Either
>> way would give us a clean new project to publish artifacts.
>>
>> Ahmet
>>
>> ---------- Forwarded message ---------
>> From: Robert Bradshaw <ro...@google.com>
>> Date: Tue, Aug 27, 2019 at 6:48 PM
>> Subject: Re: Improve container support
>> To: dev <de...@beam.apache.org>
>>
>>
>> On Tue, Aug 27, 2019 at 6:20 PM Ahmet Altay <al...@google.com> wrote:
>> >
>> > On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>> >>
>> >> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com>
>> wrote:
>> >> >
>> >> > Hi team
>> >> >
>> >> > I am working on improving docker container support for Beam. We
>> would like to publish prebuilt containers for each release version and
>> daily snapshot. Current work focuses on release images only and it would be
>> part of the release process.
>> >>
>> >> This would be great!
>> >>
>> >> > The release images will be pushed to GCR which is publicly
>> accessible(pullable). We will use the following locations.
>> >> > Repository: gcr.io/beam
>> >> > Project: apache-beam-testing
>> >>
>> >> Given that these are release artifacts, we should use a project with
>> >> more restricted access than "anyone who opens a PR on github."
>> >
>> >
>> > We have two options:
>> > - gcr.io works based on the permissions of the gcs bucket that is
>> backing it. GCS supports bucket only permissions. These permissions needs
>> to be explicitly granted and the service accounts used by jenkins jobs does
>> not have these explicit permissions today.
>> > - we can create a new project in gcr, bintray or anything else that
>> offers the same service.
>>
>> I think the cleanest is to simply have a new project whose membership
>> consists of (interested) PMC members. If we have to populate this
>> manually I think that'd still be OK as the churn is quite low.
>>
>
Re: Improve container support
Posted by Robert Bradshaw <ro...@google.com>.
On Tue, Aug 27, 2019 at 6:20 PM Ahmet Altay <al...@google.com> wrote:
>
> On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw <ro...@google.com> wrote:
>>
>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com> wrote:
>> >
>> > Hi team
>> >
>> > I am working on improving docker container support for Beam. We would like to publish prebuilt containers for each release version and daily snapshot. Current work focuses on release images only and it would be part of the release process.
>>
>> This would be great!
>>
>> > The release images will be pushed to GCR which is publicly accessible(pullable). We will use the following locations.
>> > Repository: gcr.io/beam
>> > Project: apache-beam-testing
>>
>> Given that these are release artifacts, we should use a project with
>> more restricted access than "anyone who opens a PR on github."
>
>
> We have two options:
> - gcr.io works based on the permissions of the gcs bucket that is backing it. GCS supports bucket only permissions. These permissions needs to be explicitly granted and the service accounts used by jenkins jobs does not have these explicit permissions today.
> - we can create a new project in gcr, bintray or anything else that offers the same service.
I think the cleanest is to simply have a new project whose membership
consists of (interested) PMC members. If we have to populate this
manually I think that'd still be OK as the churn is quite low.
Re: Improve container support
Posted by Ahmet Altay <al...@google.com>.
On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw <ro...@google.com> wrote:
> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com>
> wrote:
> >
> > Hi team
> >
> > I am working on improving docker container support for Beam. We would
> like to publish prebuilt containers for each release version and daily
> snapshot. Current work focuses on release images only and it would be part
> of the release process.
>
> This would be great!
>
> > The release images will be pushed to GCR which is publicly
> accessible(pullable). We will use the following locations.
> > Repository: gcr.io/beam
> > Project: apache-beam-testing
>
> Given that these are release artifacts, we should use a project with
> more restricted access than "anyone who opens a PR on github."
>
We have two options:
- gcr.io works based on the permissions of the gcs bucket that is backing
it. GCS supports bucket only permissions. These permissions needs to be
explicitly granted and the service accounts used by jenkins jobs does not
have these explicit permissions today.
- we can create a new project in gcr, bintray or anything else that offers
the same service.
>
> > More details, including naming and tagging scheme, can be found at wiki
> which is written by several contributors.
>
> Would it make sense to put this in a format more amenable to commenting?
>
> > I would like to discuss these two questions.
> > 1. How many tests do we need to run before pushing images to gcr?
> > Publishing artifacts is the last step of the release process, so at this
> moment, we already verified all codebase. In addition, many Jenkins tests
> use containers, so it is already verified several times. Do we need to run
> it again?
> >
> > 2. How many tests do we need to run to validate pushed images?
> > When we push the images, we assume the images would work and pass all
> the tests. After pushing, we should confirm the images are pullable and
> useable. I suggest we run several tests on dataflow with each pushed image.
> What do you think?
>
> I think the release manager publishing these images as part of the
> release process (just like publishing to the maven repo and svn) and
> validation happens as part of validating the artifacts during the
> vote.
>
Re: Improve container support
Posted by Robert Bradshaw <ro...@google.com>.
On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <ha...@google.com> wrote:
>
> Hi team
>
> I am working on improving docker container support for Beam. We would like to publish prebuilt containers for each release version and daily snapshot. Current work focuses on release images only and it would be part of the release process.
This would be great!
> The release images will be pushed to GCR which is publicly accessible(pullable). We will use the following locations.
> Repository: gcr.io/beam
> Project: apache-beam-testing
Given that these are release artifacts, we should use a project with
more restricted access than "anyone who opens a PR on github."
> More details, including naming and tagging scheme, can be found at wiki which is written by several contributors.
Would it make sense to put this in a format more amenable to commenting?
> I would like to discuss these two questions.
> 1. How many tests do we need to run before pushing images to gcr?
> Publishing artifacts is the last step of the release process, so at this moment, we already verified all codebase. In addition, many Jenkins tests use containers, so it is already verified several times. Do we need to run it again?
>
> 2. How many tests do we need to run to validate pushed images?
> When we push the images, we assume the images would work and pass all the tests. After pushing, we should confirm the images are pullable and useable. I suggest we run several tests on dataflow with each pushed image. What do you think?
I think the release manager publishing these images as part of the
release process (just like publishing to the maven repo and svn) and
validation happens as part of validating the artifacts during the
vote.