[Feature Proposal] Add ARM Support to Beam SDK Container Images

[Celeste Zeng <ce...@gmail.com>]

Hi everyone,

My name is Celeste. I work for the GCP Dataflow team and I am trying to add
ARM support to Beam SDK container images. The ultimate goal is to make the
released Beam SDK container images become multi-arch images, which support
both x86 and ARM. I compiled the following doc to include the feature
overview, my proposed implementation plan, as well as testing plan. And I
appreciate any feedback!

https://docs.google.com/document/d/1ikbEJNsFH1D9HqiMqiVyyMlNpDgSqxXK22nUoetzW6I/edit?usp=sharing

Also, please refer to the pull request to see proposed changes:
https://github.com/apache/beam/pull/27311

Thanks a lot!

Sincerely,
Celeste Zeng
celestezeng33@gmail.com

Re: [Feature Proposal] Add ARM Support to Beam SDK Container Images

[Robert Bradshaw via dev <de...@beam.apache.org>]

Thanks. Left a few comments on the doc. Looking forward to ARM support.

On Tue, Jul 18, 2023 at 3:59 PM Valentyn Tymofieiev via dev <
dev@beam.apache.org> wrote:

> Hi Celeste,
>
> Thanks for the proposal and researching the options. Using multi-arch
> images seems like a good way to reduce the complexity associated with
> correctly selecting  the architecture on the runner. It sounds like there
> may be implications for release process, which future release managers may
> need to be aware of, and there might be an increase in some test suites
> time now once we build ARM images.
>
> Left a few comments on the doc and happy to help with PR review when it is
> ready.
>
> bcc'ing a few folks who might have feedback or to whom this proposal might
> be of interest.
>
> Valentyn
>
>
>
> On Tue, Jul 18, 2023 at 3:12 PM Celeste Zeng <ce...@gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> My name is Celeste. I work for the GCP Dataflow team and I am trying to
>> add ARM support to Beam SDK container images. The ultimate goal is to make
>> the released Beam SDK container images become multi-arch images, which
>> support both x86 and ARM. I compiled the following doc to include the
>> feature overview, my proposed implementation plan, as well as testing plan.
>> And I appreciate any feedback!
>>
>>
>> https://docs.google.com/document/d/1ikbEJNsFH1D9HqiMqiVyyMlNpDgSqxXK22nUoetzW6I/edit?usp=sharing
>>
>> Also, please refer to the pull request to see proposed changes:
>> https://github.com/apache/beam/pull/27311
>>
>> Thanks a lot!
>>
>> Sincerely,
>> Celeste Zeng
>> celestezeng33@gmail.com
>>
>

Re: [Feature Proposal] Add ARM Support to Beam SDK Container Images

[Danny McCormick via dev <de...@beam.apache.org>]

Thanks Celeste! I left a few comments. Overall I like the proposal, but I
think that the open question "If Beam SDK containers are still released by
the release manager, how should we integrate the multiarch containers into
the current Beam container release process?" needs to be answered before I
can be fully +1 on the proposal. Ideally this shouldn't create any special
work for release managers (other than waiting a bit longer for the docker
publish steps to finish).

Thanks,
Danny

On Tue, Jul 18, 2023 at 6:59 PM Valentyn Tymofieiev <va...@google.com>
wrote:

> Hi Celeste,
>
> Thanks for the proposal and researching the options. Using multi-arch
> images seems like a good way to reduce the complexity associated with
> correctly selecting  the architecture on the runner. It sounds like there
> may be implications for release process, which future release managers may
> need to be aware of, and there might be an increase in some test suites
> time now once we build ARM images.
>
> Left a few comments on the doc and happy to help with PR review when it is
> ready.
>
> bcc'ing a few folks who might have feedback or to whom this proposal might
> be of interest.
>
> Valentyn
>
>
>
> On Tue, Jul 18, 2023 at 3:12 PM Celeste Zeng <ce...@gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> My name is Celeste. I work for the GCP Dataflow team and I am trying to
>> add ARM support to Beam SDK container images. The ultimate goal is to make
>> the released Beam SDK container images become multi-arch images, which
>> support both x86 and ARM. I compiled the following doc to include the
>> feature overview, my proposed implementation plan, as well as testing plan.
>> And I appreciate any feedback!
>>
>>
>> https://docs.google.com/document/d/1ikbEJNsFH1D9HqiMqiVyyMlNpDgSqxXK22nUoetzW6I/edit?usp=sharing
>>
>> Also, please refer to the pull request to see proposed changes:
>> https://github.com/apache/beam/pull/27311
>>
>> Thanks a lot!
>>
>> Sincerely,
>> Celeste Zeng
>> celestezeng33@gmail.com
>>
>

Re: [Feature Proposal] Add ARM Support to Beam SDK Container Images

[Valentyn Tymofieiev via dev <de...@beam.apache.org>]

Hi Celeste,

Thanks for the proposal and researching the options. Using multi-arch
images seems like a good way to reduce the complexity associated with
correctly selecting  the architecture on the runner. It sounds like there
may be implications for release process, which future release managers may
need to be aware of, and there might be an increase in some test suites
time now once we build ARM images.

Left a few comments on the doc and happy to help with PR review when it is
ready.

bcc'ing a few folks who might have feedback or to whom this proposal might
be of interest.

Valentyn



On Tue, Jul 18, 2023 at 3:12 PM Celeste Zeng <ce...@gmail.com>
wrote:

> Hi everyone,
>
> My name is Celeste. I work for the GCP Dataflow team and I am trying to
> add ARM support to Beam SDK container images. The ultimate goal is to make
> the released Beam SDK container images become multi-arch images, which
> support both x86 and ARM. I compiled the following doc to include the
> feature overview, my proposed implementation plan, as well as testing plan.
> And I appreciate any feedback!
>
>
> https://docs.google.com/document/d/1ikbEJNsFH1D9HqiMqiVyyMlNpDgSqxXK22nUoetzW6I/edit?usp=sharing
>
> Also, please refer to the pull request to see proposed changes:
> https://github.com/apache/beam/pull/27311
>
> Thanks a lot!
>
> Sincerely,
> Celeste Zeng
> celestezeng33@gmail.com
>