You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Ismaël Mejía <ie...@gmail.com> on 2022/02/08 13:49:59 UTC
Re: Developing on an M1 Mac
For awareness with the just released Beam 2.36.0 Beam works out of the
box to develop on a Mac M1.
I tried Java and Python pipelines with success running locally on both
Flink/Spark runner.
I found one issue using zstd and created [1] that was merged today,
with this the sdks:core tests and Spark runner tests fully pass.
I would see 2.36.0 is the first good enough release for someone
working on a Mac M1 or ARM64 processor.
There are still some missing steps to have full ARM64 [apart of testing it :)]
1. In theory we could run docker x86 images on ARM but those would be
emulated so way slower so it is probably better to support 'native'
CPUs) via multiarchitecture docker images [2].
BEAM-11704 Support Beam docker images on ARM64
I could create the runners images from master, for the SDK containers
there are some issues with hardcoded paths [2] and virtualenv that
probably will be solved once we move to venv, and we will need to
upgrade our release process to include multiarch images (for user
friendliness).
Also golang only supports officially ARM64 starting with version
1.18.0 so we need to move up to that version.
Anyway Beam is in a waaay better shape for ARM64 now than 1y ago when
I created the initial JIRAs.
Ismaël
[1] https://github.com/apache/beam/pull/16755
[2] https://issues.apache.org/jira/browse/BEAM-11704
[3] https://github.com/apache/beam/blob/d1b8e569fd651975f08823a3db49dbee56d491b5/sdks/python/container/Dockerfile#L79
> Could not find protoc-3.14.0-osx-aarch_64.exe
(com.google.protobuf:protoc:3.14.0).
Searched in the following locations:
https://jcenter.bintray.com/com/google/protobuf/protoc/3.14.0/protoc-3.14.0-osx-aarch_64.exe
On Wed, Jan 12, 2022 at 9:53 PM Luke Cwik <lc...@google.com> wrote:
>
> The docker container running in an x86 based cloud machine should work pretty well. This is what Apache Beam's Jenkins setup effectively does.
>
> No experience with developing on an ARM based CPU.
>
> On Wed, Jan 12, 2022 at 9:28 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> Comment from the side - If you use Docker - experience from Airflow -
>> until we will get ARM images, docker experience is next to unusable
>> (docker filesystem slowness + emulation).
>>
>> J.
>>
>> On Wed, Jan 12, 2022 at 6:21 PM Daniel Collins <dp...@google.com> wrote:
>> >
>> > I regularly develop on a non-m1 mac using intellij, which mostly works out of the box. Are you running into any particular issues building or just looking for advice?
>> >
>> > -Daniel
>> >
>> > On Wed, Jan 12, 2022 at 12:16 PM Matt Rudary <Ma...@twosigma.com> wrote:
>> >>
>> >> Does anyone do Beam development on an M1 Mac? Any tips to getting things up and running?
>> >>
>> >>
>> >>
>> >> Alternatively, does anyone have a good “workstation in the cloud” setup?
>> >>
>> >>
>> >>
>> >> Thanks
>> >>
>> >> Matt
Re: Developing on an M1 Mac
Posted by Reuven Lax <re...@google.com>.
What's the current status on developing with M1 machines? I'm considering
upgrading my laptop to an M-1 based laptop, but don't want to regress my
ability to develop and build Beam code.
On Wed, Feb 9, 2022 at 11:51 AM Robert Burke <ro...@frantil.com> wrote:
> For Go we *will* (and do) need to cross compile and in many cases do so
> automatically, for Amd64 instances, but don't currently support Arm64
> workers.
>
> We will likely need to support platform specific containers for all the
> languages, or include boot loaders for all platforms and select which one
> we want via a script.
>
> I have a task to address the user side of this with documentation updates
> to clarify how it works.
>
> On Wed, Feb 9, 2022, 9:55 AM Robert Bradshaw <ro...@google.com> wrote:
>
>> Cool to see all the progress being made with ARM processors.
>>
>> Regarding containers (especially shipping the Go binary), we should
>> make sure to support the case where the pipeline is constructed on ARM
>> (e.g. a new Mac) but submitted to an x86 cluster (currently most of
>> what's out there). And, of course, vice versa. For Python this would
>> involve choosing the right wheels to deploy, Java may have issues with
>> JNI, and for Go we may need to cross-compile.
>>
>>
>> On Wed, Feb 9, 2022 at 12:27 AM Ismaël Mejía <ie...@gmail.com> wrote:
>> >
>> > Thanks for pointing this out Robert, I had somehow in my mind that it
>> > was not official until 1.18 but I forgot to double check, for info I
>> > was able to build the Go SDK container without any trouble so I assume
>> > most of the things are 'ready'.
>> >
>> > It would be great to take a look at fixing the hardcoded arch uses in
>> > paths and other (grep -i -R 'amd64') , don't hesitate to ping me if
>> > you need help with some some test I have easy acces to AWS Graviton
>> > too.
>> >
>> > Of course the Beam community has never mentioned being ARM64
>> > compatible as a target but with the increasing rise of this arch on
>> > the server side and now on the Mac, it would be probably worth.
>> >
>> > In case we decide to do it, Github Actions does not support ARM64 yet,
>> > so other projects validate ARM64 through emulation via qemu, but this
>> > seems to be quite slow.
>> >
>> >
>> >
>> > On Tue, Feb 8, 2022 at 3:58 PM Robert Burke <ro...@frantil.com> wrote:
>> > >
>> > > Go supports ARM64 on Darwin since 1.16, which is the minimum version
>> of Go we currently support.
>> > >
>> > > See https://go.dev/blog/ports
>> > >
>> > > There are definitely some hardcoded paths we'd need to adjust to
>> build boot containers though.
>> > >
>> > > Go 1.18 improves things, and since it has the initial run of Go
>> Generics, we'll likely move to support it pretty quickly.
>> > >
>> > > On Tue, Feb 8, 2022, 6:18 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>> > >>
>> > >> Just for your information: Thanks to that change - i will soon be
>> adding ARM support for Apache Airflow - including building and publishing
>> the images and running our tests (using self-hosted runners).
>> > >> As soon as I get it I will be able to share the code/experiences
>> with you.
>> > >>
>> > >> J
>> > >>
>> > >> On Tue, Feb 8, 2022 at 2:50 PM Ismaël Mejía <ie...@gmail.com>
>> wrote:
>> > >>>
>> > >>> For awareness with the just released Beam 2.36.0 Beam works out of
>> the
>> > >>> box to develop on a Mac M1.
>> > >>>
>> > >>> I tried Java and Python pipelines with success running locally on
>> both
>> > >>> Flink/Spark runner.
>> > >>> I found one issue using zstd and created [1] that was merged today,
>> > >>> with this the sdks:core tests and Spark runner tests fully pass.
>> > >>>
>> > >>> I would see 2.36.0 is the first good enough release for someone
>> > >>> working on a Mac M1 or ARM64 processor.
>> > >>>
>> > >>> There are still some missing steps to have full ARM64 [apart of
>> testing it :)]
>> > >>>
>> > >>> 1. In theory we could run docker x86 images on ARM but those would
>> be
>> > >>> emulated so way slower so it is probably better to support 'native'
>> > >>> CPUs) via multiarchitecture docker images [2].
>> > >>> BEAM-11704 Support Beam docker images on ARM64
>> > >>>
>> > >>> I could create the runners images from master, for the SDK
>> containers
>> > >>> there are some issues with hardcoded paths [2] and virtualenv that
>> > >>> probably will be solved once we move to venv, and we will need to
>> > >>> upgrade our release process to include multiarch images (for user
>> > >>> friendliness).
>> > >>>
>> > >>> Also golang only supports officially ARM64 starting with version
>> > >>> 1.18.0 so we need to move up to that version.
>> > >>>
>> > >>> Anyway Beam is in a waaay better shape for ARM64 now than 1y ago
>> when
>> > >>> I created the initial JIRAs.
>> > >>>
>> > >>> Ismaël
>> > >>>
>> > >>> [1] https://github.com/apache/beam/pull/16755
>> > >>> [2] https://issues.apache.org/jira/browse/BEAM-11704
>> > >>> [3]
>> https://github.com/apache/beam/blob/d1b8e569fd651975f08823a3db49dbee56d491b5/sdks/python/container/Dockerfile#L79
>> > >>>
>> > >>>
>> > >>>
>> > >>> > Could not find protoc-3.14.0-osx-aarch_64.exe
>> > >>> (com.google.protobuf:protoc:3.14.0).
>> > >>> Searched in the following locations:
>> > >>>
>> https://jcenter.bintray.com/com/google/protobuf/protoc/3.14.0/protoc-3.14.0-osx-aarch_64.exe
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>> On Wed, Jan 12, 2022 at 9:53 PM Luke Cwik <lc...@google.com> wrote:
>> > >>> >
>> > >>> > The docker container running in an x86 based cloud machine should
>> work pretty well. This is what Apache Beam's Jenkins setup effectively does.
>> > >>> >
>> > >>> > No experience with developing on an ARM based CPU.
>> > >>> >
>> > >>> > On Wed, Jan 12, 2022 at 9:28 AM Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>> > >>> >>
>> > >>> >> Comment from the side - If you use Docker - experience from
>> Airflow -
>> > >>> >> until we will get ARM images, docker experience is next to
>> unusable
>> > >>> >> (docker filesystem slowness + emulation).
>> > >>> >>
>> > >>> >> J.
>> > >>> >>
>> > >>> >> On Wed, Jan 12, 2022 at 6:21 PM Daniel Collins <
>> dpcollins@google.com> wrote:
>> > >>> >> >
>> > >>> >> > I regularly develop on a non-m1 mac using intellij, which
>> mostly works out of the box. Are you running into any particular issues
>> building or just looking for advice?
>> > >>> >> >
>> > >>> >> > -Daniel
>> > >>> >> >
>> > >>> >> > On Wed, Jan 12, 2022 at 12:16 PM Matt Rudary <
>> Matt.Rudary@twosigma.com> wrote:
>> > >>> >> >>
>> > >>> >> >> Does anyone do Beam development on an M1 Mac? Any tips to
>> getting things up and running?
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >> Alternatively, does anyone have a good “workstation in the
>> cloud” setup?
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >> Thanks
>> > >>> >> >>
>> > >>> >> >> Matt
>>
>
Re: Developing on an M1 Mac
Posted by Robert Burke <ro...@frantil.com>.
For Go we *will* (and do) need to cross compile and in many cases do so
automatically, for Amd64 instances, but don't currently support Arm64
workers.
We will likely need to support platform specific containers for all the
languages, or include boot loaders for all platforms and select which one
we want via a script.
I have a task to address the user side of this with documentation updates
to clarify how it works.
On Wed, Feb 9, 2022, 9:55 AM Robert Bradshaw <ro...@google.com> wrote:
> Cool to see all the progress being made with ARM processors.
>
> Regarding containers (especially shipping the Go binary), we should
> make sure to support the case where the pipeline is constructed on ARM
> (e.g. a new Mac) but submitted to an x86 cluster (currently most of
> what's out there). And, of course, vice versa. For Python this would
> involve choosing the right wheels to deploy, Java may have issues with
> JNI, and for Go we may need to cross-compile.
>
>
> On Wed, Feb 9, 2022 at 12:27 AM Ismaël Mejía <ie...@gmail.com> wrote:
> >
> > Thanks for pointing this out Robert, I had somehow in my mind that it
> > was not official until 1.18 but I forgot to double check, for info I
> > was able to build the Go SDK container without any trouble so I assume
> > most of the things are 'ready'.
> >
> > It would be great to take a look at fixing the hardcoded arch uses in
> > paths and other (grep -i -R 'amd64') , don't hesitate to ping me if
> > you need help with some some test I have easy acces to AWS Graviton
> > too.
> >
> > Of course the Beam community has never mentioned being ARM64
> > compatible as a target but with the increasing rise of this arch on
> > the server side and now on the Mac, it would be probably worth.
> >
> > In case we decide to do it, Github Actions does not support ARM64 yet,
> > so other projects validate ARM64 through emulation via qemu, but this
> > seems to be quite slow.
> >
> >
> >
> > On Tue, Feb 8, 2022 at 3:58 PM Robert Burke <ro...@frantil.com> wrote:
> > >
> > > Go supports ARM64 on Darwin since 1.16, which is the minimum version
> of Go we currently support.
> > >
> > > See https://go.dev/blog/ports
> > >
> > > There are definitely some hardcoded paths we'd need to adjust to build
> boot containers though.
> > >
> > > Go 1.18 improves things, and since it has the initial run of Go
> Generics, we'll likely move to support it pretty quickly.
> > >
> > > On Tue, Feb 8, 2022, 6:18 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >>
> > >> Just for your information: Thanks to that change - i will soon be
> adding ARM support for Apache Airflow - including building and publishing
> the images and running our tests (using self-hosted runners).
> > >> As soon as I get it I will be able to share the code/experiences with
> you.
> > >>
> > >> J
> > >>
> > >> On Tue, Feb 8, 2022 at 2:50 PM Ismaël Mejía <ie...@gmail.com>
> wrote:
> > >>>
> > >>> For awareness with the just released Beam 2.36.0 Beam works out of
> the
> > >>> box to develop on a Mac M1.
> > >>>
> > >>> I tried Java and Python pipelines with success running locally on
> both
> > >>> Flink/Spark runner.
> > >>> I found one issue using zstd and created [1] that was merged today,
> > >>> with this the sdks:core tests and Spark runner tests fully pass.
> > >>>
> > >>> I would see 2.36.0 is the first good enough release for someone
> > >>> working on a Mac M1 or ARM64 processor.
> > >>>
> > >>> There are still some missing steps to have full ARM64 [apart of
> testing it :)]
> > >>>
> > >>> 1. In theory we could run docker x86 images on ARM but those would be
> > >>> emulated so way slower so it is probably better to support 'native'
> > >>> CPUs) via multiarchitecture docker images [2].
> > >>> BEAM-11704 Support Beam docker images on ARM64
> > >>>
> > >>> I could create the runners images from master, for the SDK containers
> > >>> there are some issues with hardcoded paths [2] and virtualenv that
> > >>> probably will be solved once we move to venv, and we will need to
> > >>> upgrade our release process to include multiarch images (for user
> > >>> friendliness).
> > >>>
> > >>> Also golang only supports officially ARM64 starting with version
> > >>> 1.18.0 so we need to move up to that version.
> > >>>
> > >>> Anyway Beam is in a waaay better shape for ARM64 now than 1y ago when
> > >>> I created the initial JIRAs.
> > >>>
> > >>> Ismaël
> > >>>
> > >>> [1] https://github.com/apache/beam/pull/16755
> > >>> [2] https://issues.apache.org/jira/browse/BEAM-11704
> > >>> [3]
> https://github.com/apache/beam/blob/d1b8e569fd651975f08823a3db49dbee56d491b5/sdks/python/container/Dockerfile#L79
> > >>>
> > >>>
> > >>>
> > >>> > Could not find protoc-3.14.0-osx-aarch_64.exe
> > >>> (com.google.protobuf:protoc:3.14.0).
> > >>> Searched in the following locations:
> > >>>
> https://jcenter.bintray.com/com/google/protobuf/protoc/3.14.0/protoc-3.14.0-osx-aarch_64.exe
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Wed, Jan 12, 2022 at 9:53 PM Luke Cwik <lc...@google.com> wrote:
> > >>> >
> > >>> > The docker container running in an x86 based cloud machine should
> work pretty well. This is what Apache Beam's Jenkins setup effectively does.
> > >>> >
> > >>> > No experience with developing on an ARM based CPU.
> > >>> >
> > >>> > On Wed, Jan 12, 2022 at 9:28 AM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >>> >>
> > >>> >> Comment from the side - If you use Docker - experience from
> Airflow -
> > >>> >> until we will get ARM images, docker experience is next to
> unusable
> > >>> >> (docker filesystem slowness + emulation).
> > >>> >>
> > >>> >> J.
> > >>> >>
> > >>> >> On Wed, Jan 12, 2022 at 6:21 PM Daniel Collins <
> dpcollins@google.com> wrote:
> > >>> >> >
> > >>> >> > I regularly develop on a non-m1 mac using intellij, which
> mostly works out of the box. Are you running into any particular issues
> building or just looking for advice?
> > >>> >> >
> > >>> >> > -Daniel
> > >>> >> >
> > >>> >> > On Wed, Jan 12, 2022 at 12:16 PM Matt Rudary <
> Matt.Rudary@twosigma.com> wrote:
> > >>> >> >>
> > >>> >> >> Does anyone do Beam development on an M1 Mac? Any tips to
> getting things up and running?
> > >>> >> >>
> > >>> >> >>
> > >>> >> >>
> > >>> >> >> Alternatively, does anyone have a good “workstation in the
> cloud” setup?
> > >>> >> >>
> > >>> >> >>
> > >>> >> >>
> > >>> >> >> Thanks
> > >>> >> >>
> > >>> >> >> Matt
>
Re: Developing on an M1 Mac
Posted by Robert Bradshaw <ro...@google.com>.
Cool to see all the progress being made with ARM processors.
Regarding containers (especially shipping the Go binary), we should
make sure to support the case where the pipeline is constructed on ARM
(e.g. a new Mac) but submitted to an x86 cluster (currently most of
what's out there). And, of course, vice versa. For Python this would
involve choosing the right wheels to deploy, Java may have issues with
JNI, and for Go we may need to cross-compile.
On Wed, Feb 9, 2022 at 12:27 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
> Thanks for pointing this out Robert, I had somehow in my mind that it
> was not official until 1.18 but I forgot to double check, for info I
> was able to build the Go SDK container without any trouble so I assume
> most of the things are 'ready'.
>
> It would be great to take a look at fixing the hardcoded arch uses in
> paths and other (grep -i -R 'amd64') , don't hesitate to ping me if
> you need help with some some test I have easy acces to AWS Graviton
> too.
>
> Of course the Beam community has never mentioned being ARM64
> compatible as a target but with the increasing rise of this arch on
> the server side and now on the Mac, it would be probably worth.
>
> In case we decide to do it, Github Actions does not support ARM64 yet,
> so other projects validate ARM64 through emulation via qemu, but this
> seems to be quite slow.
>
>
>
> On Tue, Feb 8, 2022 at 3:58 PM Robert Burke <ro...@frantil.com> wrote:
> >
> > Go supports ARM64 on Darwin since 1.16, which is the minimum version of Go we currently support.
> >
> > See https://go.dev/blog/ports
> >
> > There are definitely some hardcoded paths we'd need to adjust to build boot containers though.
> >
> > Go 1.18 improves things, and since it has the initial run of Go Generics, we'll likely move to support it pretty quickly.
> >
> > On Tue, Feb 8, 2022, 6:18 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >> Just for your information: Thanks to that change - i will soon be adding ARM support for Apache Airflow - including building and publishing the images and running our tests (using self-hosted runners).
> >> As soon as I get it I will be able to share the code/experiences with you.
> >>
> >> J
> >>
> >> On Tue, Feb 8, 2022 at 2:50 PM Ismaël Mejía <ie...@gmail.com> wrote:
> >>>
> >>> For awareness with the just released Beam 2.36.0 Beam works out of the
> >>> box to develop on a Mac M1.
> >>>
> >>> I tried Java and Python pipelines with success running locally on both
> >>> Flink/Spark runner.
> >>> I found one issue using zstd and created [1] that was merged today,
> >>> with this the sdks:core tests and Spark runner tests fully pass.
> >>>
> >>> I would see 2.36.0 is the first good enough release for someone
> >>> working on a Mac M1 or ARM64 processor.
> >>>
> >>> There are still some missing steps to have full ARM64 [apart of testing it :)]
> >>>
> >>> 1. In theory we could run docker x86 images on ARM but those would be
> >>> emulated so way slower so it is probably better to support 'native'
> >>> CPUs) via multiarchitecture docker images [2].
> >>> BEAM-11704 Support Beam docker images on ARM64
> >>>
> >>> I could create the runners images from master, for the SDK containers
> >>> there are some issues with hardcoded paths [2] and virtualenv that
> >>> probably will be solved once we move to venv, and we will need to
> >>> upgrade our release process to include multiarch images (for user
> >>> friendliness).
> >>>
> >>> Also golang only supports officially ARM64 starting with version
> >>> 1.18.0 so we need to move up to that version.
> >>>
> >>> Anyway Beam is in a waaay better shape for ARM64 now than 1y ago when
> >>> I created the initial JIRAs.
> >>>
> >>> Ismaël
> >>>
> >>> [1] https://github.com/apache/beam/pull/16755
> >>> [2] https://issues.apache.org/jira/browse/BEAM-11704
> >>> [3] https://github.com/apache/beam/blob/d1b8e569fd651975f08823a3db49dbee56d491b5/sdks/python/container/Dockerfile#L79
> >>>
> >>>
> >>>
> >>> > Could not find protoc-3.14.0-osx-aarch_64.exe
> >>> (com.google.protobuf:protoc:3.14.0).
> >>> Searched in the following locations:
> >>> https://jcenter.bintray.com/com/google/protobuf/protoc/3.14.0/protoc-3.14.0-osx-aarch_64.exe
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Jan 12, 2022 at 9:53 PM Luke Cwik <lc...@google.com> wrote:
> >>> >
> >>> > The docker container running in an x86 based cloud machine should work pretty well. This is what Apache Beam's Jenkins setup effectively does.
> >>> >
> >>> > No experience with developing on an ARM based CPU.
> >>> >
> >>> > On Wed, Jan 12, 2022 at 9:28 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>> >>
> >>> >> Comment from the side - If you use Docker - experience from Airflow -
> >>> >> until we will get ARM images, docker experience is next to unusable
> >>> >> (docker filesystem slowness + emulation).
> >>> >>
> >>> >> J.
> >>> >>
> >>> >> On Wed, Jan 12, 2022 at 6:21 PM Daniel Collins <dp...@google.com> wrote:
> >>> >> >
> >>> >> > I regularly develop on a non-m1 mac using intellij, which mostly works out of the box. Are you running into any particular issues building or just looking for advice?
> >>> >> >
> >>> >> > -Daniel
> >>> >> >
> >>> >> > On Wed, Jan 12, 2022 at 12:16 PM Matt Rudary <Ma...@twosigma.com> wrote:
> >>> >> >>
> >>> >> >> Does anyone do Beam development on an M1 Mac? Any tips to getting things up and running?
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >> Alternatively, does anyone have a good “workstation in the cloud” setup?
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >> Thanks
> >>> >> >>
> >>> >> >> Matt
Re: Developing on an M1 Mac
Posted by Ismaël Mejía <ie...@gmail.com>.
Thanks for pointing this out Robert, I had somehow in my mind that it
was not official until 1.18 but I forgot to double check, for info I
was able to build the Go SDK container without any trouble so I assume
most of the things are 'ready'.
It would be great to take a look at fixing the hardcoded arch uses in
paths and other (grep -i -R 'amd64') , don't hesitate to ping me if
you need help with some some test I have easy acces to AWS Graviton
too.
Of course the Beam community has never mentioned being ARM64
compatible as a target but with the increasing rise of this arch on
the server side and now on the Mac, it would be probably worth.
In case we decide to do it, Github Actions does not support ARM64 yet,
so other projects validate ARM64 through emulation via qemu, but this
seems to be quite slow.
On Tue, Feb 8, 2022 at 3:58 PM Robert Burke <ro...@frantil.com> wrote:
>
> Go supports ARM64 on Darwin since 1.16, which is the minimum version of Go we currently support.
>
> See https://go.dev/blog/ports
>
> There are definitely some hardcoded paths we'd need to adjust to build boot containers though.
>
> Go 1.18 improves things, and since it has the initial run of Go Generics, we'll likely move to support it pretty quickly.
>
> On Tue, Feb 8, 2022, 6:18 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> Just for your information: Thanks to that change - i will soon be adding ARM support for Apache Airflow - including building and publishing the images and running our tests (using self-hosted runners).
>> As soon as I get it I will be able to share the code/experiences with you.
>>
>> J
>>
>> On Tue, Feb 8, 2022 at 2:50 PM Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>> For awareness with the just released Beam 2.36.0 Beam works out of the
>>> box to develop on a Mac M1.
>>>
>>> I tried Java and Python pipelines with success running locally on both
>>> Flink/Spark runner.
>>> I found one issue using zstd and created [1] that was merged today,
>>> with this the sdks:core tests and Spark runner tests fully pass.
>>>
>>> I would see 2.36.0 is the first good enough release for someone
>>> working on a Mac M1 or ARM64 processor.
>>>
>>> There are still some missing steps to have full ARM64 [apart of testing it :)]
>>>
>>> 1. In theory we could run docker x86 images on ARM but those would be
>>> emulated so way slower so it is probably better to support 'native'
>>> CPUs) via multiarchitecture docker images [2].
>>> BEAM-11704 Support Beam docker images on ARM64
>>>
>>> I could create the runners images from master, for the SDK containers
>>> there are some issues with hardcoded paths [2] and virtualenv that
>>> probably will be solved once we move to venv, and we will need to
>>> upgrade our release process to include multiarch images (for user
>>> friendliness).
>>>
>>> Also golang only supports officially ARM64 starting with version
>>> 1.18.0 so we need to move up to that version.
>>>
>>> Anyway Beam is in a waaay better shape for ARM64 now than 1y ago when
>>> I created the initial JIRAs.
>>>
>>> Ismaël
>>>
>>> [1] https://github.com/apache/beam/pull/16755
>>> [2] https://issues.apache.org/jira/browse/BEAM-11704
>>> [3] https://github.com/apache/beam/blob/d1b8e569fd651975f08823a3db49dbee56d491b5/sdks/python/container/Dockerfile#L79
>>>
>>>
>>>
>>> > Could not find protoc-3.14.0-osx-aarch_64.exe
>>> (com.google.protobuf:protoc:3.14.0).
>>> Searched in the following locations:
>>> https://jcenter.bintray.com/com/google/protobuf/protoc/3.14.0/protoc-3.14.0-osx-aarch_64.exe
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Jan 12, 2022 at 9:53 PM Luke Cwik <lc...@google.com> wrote:
>>> >
>>> > The docker container running in an x86 based cloud machine should work pretty well. This is what Apache Beam's Jenkins setup effectively does.
>>> >
>>> > No experience with developing on an ARM based CPU.
>>> >
>>> > On Wed, Jan 12, 2022 at 9:28 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>> >>
>>> >> Comment from the side - If you use Docker - experience from Airflow -
>>> >> until we will get ARM images, docker experience is next to unusable
>>> >> (docker filesystem slowness + emulation).
>>> >>
>>> >> J.
>>> >>
>>> >> On Wed, Jan 12, 2022 at 6:21 PM Daniel Collins <dp...@google.com> wrote:
>>> >> >
>>> >> > I regularly develop on a non-m1 mac using intellij, which mostly works out of the box. Are you running into any particular issues building or just looking for advice?
>>> >> >
>>> >> > -Daniel
>>> >> >
>>> >> > On Wed, Jan 12, 2022 at 12:16 PM Matt Rudary <Ma...@twosigma.com> wrote:
>>> >> >>
>>> >> >> Does anyone do Beam development on an M1 Mac? Any tips to getting things up and running?
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> Alternatively, does anyone have a good “workstation in the cloud” setup?
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> Thanks
>>> >> >>
>>> >> >> Matt
Re: Developing on an M1 Mac
Posted by Robert Burke <ro...@frantil.com>.
Go supports ARM64 on Darwin since 1.16, which is the minimum version of Go
we currently support.
See https://go.dev/blog/ports
There are definitely some hardcoded paths we'd need to adjust to build boot
containers though.
Go 1.18 improves things, and since it has the initial run of Go Generics,
we'll likely move to support it pretty quickly.
On Tue, Feb 8, 2022, 6:18 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Just for your information: Thanks to that change - i will soon be adding
> ARM support for Apache Airflow - including building and publishing the
> images and running our tests (using self-hosted runners).
> As soon as I get it I will be able to share the code/experiences with you.
>
> J
>
> On Tue, Feb 8, 2022 at 2:50 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> For awareness with the just released Beam 2.36.0 Beam works out of the
>> box to develop on a Mac M1.
>>
>> I tried Java and Python pipelines with success running locally on both
>> Flink/Spark runner.
>> I found one issue using zstd and created [1] that was merged today,
>> with this the sdks:core tests and Spark runner tests fully pass.
>>
>> I would see 2.36.0 is the first good enough release for someone
>> working on a Mac M1 or ARM64 processor.
>>
>> There are still some missing steps to have full ARM64 [apart of testing
>> it :)]
>>
>> 1. In theory we could run docker x86 images on ARM but those would be
>> emulated so way slower so it is probably better to support 'native'
>> CPUs) via multiarchitecture docker images [2].
>> BEAM-11704 Support Beam docker images on ARM64
>>
>> I could create the runners images from master, for the SDK containers
>> there are some issues with hardcoded paths [2] and virtualenv that
>> probably will be solved once we move to venv, and we will need to
>> upgrade our release process to include multiarch images (for user
>> friendliness).
>>
>> Also golang only supports officially ARM64 starting with version
>> 1.18.0 so we need to move up to that version.
>>
>> Anyway Beam is in a waaay better shape for ARM64 now than 1y ago when
>> I created the initial JIRAs.
>>
>> Ismaël
>>
>> [1] https://github.com/apache/beam/pull/16755
>> [2] https://issues.apache.org/jira/browse/BEAM-11704
>> [3]
>> https://github.com/apache/beam/blob/d1b8e569fd651975f08823a3db49dbee56d491b5/sdks/python/container/Dockerfile#L79
>>
>>
>>
>> > Could not find protoc-3.14.0-osx-aarch_64.exe
>> (com.google.protobuf:protoc:3.14.0).
>> Searched in the following locations:
>>
>> https://jcenter.bintray.com/com/google/protobuf/protoc/3.14.0/protoc-3.14.0-osx-aarch_64.exe
>>
>>
>>
>>
>>
>> On Wed, Jan 12, 2022 at 9:53 PM Luke Cwik <lc...@google.com> wrote:
>> >
>> > The docker container running in an x86 based cloud machine should work
>> pretty well. This is what Apache Beam's Jenkins setup effectively does.
>> >
>> > No experience with developing on an ARM based CPU.
>> >
>> > On Wed, Jan 12, 2022 at 9:28 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>
>> >> Comment from the side - If you use Docker - experience from Airflow -
>> >> until we will get ARM images, docker experience is next to unusable
>> >> (docker filesystem slowness + emulation).
>> >>
>> >> J.
>> >>
>> >> On Wed, Jan 12, 2022 at 6:21 PM Daniel Collins <dp...@google.com>
>> wrote:
>> >> >
>> >> > I regularly develop on a non-m1 mac using intellij, which mostly
>> works out of the box. Are you running into any particular issues building
>> or just looking for advice?
>> >> >
>> >> > -Daniel
>> >> >
>> >> > On Wed, Jan 12, 2022 at 12:16 PM Matt Rudary <
>> Matt.Rudary@twosigma.com> wrote:
>> >> >>
>> >> >> Does anyone do Beam development on an M1 Mac? Any tips to getting
>> things up and running?
>> >> >>
>> >> >>
>> >> >>
>> >> >> Alternatively, does anyone have a good “workstation in the cloud”
>> setup?
>> >> >>
>> >> >>
>> >> >>
>> >> >> Thanks
>> >> >>
>> >> >> Matt
>>
>
Re: Developing on an M1 Mac
Posted by Jarek Potiuk <ja...@potiuk.com>.
Just for your information: Thanks to that change - i will soon be adding
ARM support for Apache Airflow - including building and publishing the
images and running our tests (using self-hosted runners).
As soon as I get it I will be able to share the code/experiences with you.
J
On Tue, Feb 8, 2022 at 2:50 PM Ismaël Mejía <ie...@gmail.com> wrote:
> For awareness with the just released Beam 2.36.0 Beam works out of the
> box to develop on a Mac M1.
>
> I tried Java and Python pipelines with success running locally on both
> Flink/Spark runner.
> I found one issue using zstd and created [1] that was merged today,
> with this the sdks:core tests and Spark runner tests fully pass.
>
> I would see 2.36.0 is the first good enough release for someone
> working on a Mac M1 or ARM64 processor.
>
> There are still some missing steps to have full ARM64 [apart of testing it
> :)]
>
> 1. In theory we could run docker x86 images on ARM but those would be
> emulated so way slower so it is probably better to support 'native'
> CPUs) via multiarchitecture docker images [2].
> BEAM-11704 Support Beam docker images on ARM64
>
> I could create the runners images from master, for the SDK containers
> there are some issues with hardcoded paths [2] and virtualenv that
> probably will be solved once we move to venv, and we will need to
> upgrade our release process to include multiarch images (for user
> friendliness).
>
> Also golang only supports officially ARM64 starting with version
> 1.18.0 so we need to move up to that version.
>
> Anyway Beam is in a waaay better shape for ARM64 now than 1y ago when
> I created the initial JIRAs.
>
> Ismaël
>
> [1] https://github.com/apache/beam/pull/16755
> [2] https://issues.apache.org/jira/browse/BEAM-11704
> [3]
> https://github.com/apache/beam/blob/d1b8e569fd651975f08823a3db49dbee56d491b5/sdks/python/container/Dockerfile#L79
>
>
>
> > Could not find protoc-3.14.0-osx-aarch_64.exe
> (com.google.protobuf:protoc:3.14.0).
> Searched in the following locations:
>
> https://jcenter.bintray.com/com/google/protobuf/protoc/3.14.0/protoc-3.14.0-osx-aarch_64.exe
>
>
>
>
>
> On Wed, Jan 12, 2022 at 9:53 PM Luke Cwik <lc...@google.com> wrote:
> >
> > The docker container running in an x86 based cloud machine should work
> pretty well. This is what Apache Beam's Jenkins setup effectively does.
> >
> > No experience with developing on an ARM based CPU.
> >
> > On Wed, Jan 12, 2022 at 9:28 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >> Comment from the side - If you use Docker - experience from Airflow -
> >> until we will get ARM images, docker experience is next to unusable
> >> (docker filesystem slowness + emulation).
> >>
> >> J.
> >>
> >> On Wed, Jan 12, 2022 at 6:21 PM Daniel Collins <dp...@google.com>
> wrote:
> >> >
> >> > I regularly develop on a non-m1 mac using intellij, which mostly
> works out of the box. Are you running into any particular issues building
> or just looking for advice?
> >> >
> >> > -Daniel
> >> >
> >> > On Wed, Jan 12, 2022 at 12:16 PM Matt Rudary <
> Matt.Rudary@twosigma.com> wrote:
> >> >>
> >> >> Does anyone do Beam development on an M1 Mac? Any tips to getting
> things up and running?
> >> >>
> >> >>
> >> >>
> >> >> Alternatively, does anyone have a good “workstation in the cloud”
> setup?
> >> >>
> >> >>
> >> >>
> >> >> Thanks
> >> >>
> >> >> Matt
>