You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2022/03/10 11:12:49 UTC
[ANNOUNCEMENT] Support for MacOS M1 in Breeze
Hello Everyone,
Another day, another announcement. I have just merged this PR
https://github.com/apache/airflow/pull/22127 that adds support for
development for Mac OS M1 users (and also anyone who uses ARM-based linux.
The M1 users have (rightfully) complained about rather poor performance for
Dockerized Airflow in Breeze (the intel-based images were used via
emulation and suffered roughly ~ 10x slower speed which made them barely
usable for development).
As of today, our images used for development in Breeze are multi-platform
(amd64/arm64). This is - for now - development only and has some
limitations (mysql and mssql support is missing until mysql and mssql
release arm-based clients and servers - but we have full postgres and
sqlite support - including native ARM postgres server Docker image).
Some initial teething problems have been fixed (thanks Elad/Ephraim :) but
also as a follow up I will need to observe it and implement a few build
optimizations (next few days). So if you see any problems, delays etc, ping
me on #breeze channel in our slack.
Also building ARM images on Intel also suffers from the same 10x slower
emulation problems so we need to modify our CI / self-hosted
infrastructure, but once we do and once we start testing ARM on CI, we will
also be able to release multi-platform "release" images, completing
https://github.com/apache/airflow/issues/15635 and opening up Airflow to be
used on ARM-based linux.
I hope in 2.3.0 we will be able to release the Airflow Production ARM image.
J.
Re: [ANNOUNCEMENT] Support for MacOS M1 in Breeze
Posted by Jarek Potiuk <ja...@potiuk.com>.
Glad you like it. Let me know if you see any problems :)
On Thu, Mar 10, 2022 at 8:50 PM Ping Zhang <pi...@umich.edu> wrote:
> This is HUGE!! thanks Jarek. I have been suffering from slowness for
> months.
>
> Thanks,
>
> Ping
>
>
> On Thu, Mar 10, 2022 at 3:19 AM Kevin Yang <yr...@gmail.com> wrote:
>
>> Excellent! Thanks to Jarek and everyone involved for working on getting
>> us better support on ARM-based Linux. 🙏 Looking forward for the upcoming
>> improvements 👍
>>
>> On Thu, Mar 10, 2022 at 3:13 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> Hello Everyone,
>>>
>>> Another day, another announcement. I have just merged this PR
>>> https://github.com/apache/airflow/pull/22127 that adds support for
>>> development for Mac OS M1 users (and also anyone who uses ARM-based linux.
>>>
>>> The M1 users have (rightfully) complained about rather poor performance
>>> for Dockerized Airflow in Breeze (the intel-based images were used via
>>> emulation and suffered roughly ~ 10x slower speed which made them barely
>>> usable for development).
>>>
>>> As of today, our images used for development in Breeze are
>>> multi-platform (amd64/arm64). This is - for now - development only and has
>>> some limitations (mysql and mssql support is missing until mysql and mssql
>>> release arm-based clients and servers - but we have full postgres and
>>> sqlite support - including native ARM postgres server Docker image).
>>>
>>> Some initial teething problems have been fixed (thanks Elad/Ephraim :)
>>> but also as a follow up I will need to observe it and implement a few
>>> build optimizations (next few days). So if you see any problems, delays
>>> etc, ping me on #breeze channel in our slack.
>>>
>>> Also building ARM images on Intel also suffers from the same 10x slower
>>> emulation problems so we need to modify our CI / self-hosted
>>> infrastructure, but once we do and once we start testing ARM on CI, we will
>>> also be able to release multi-platform "release" images, completing
>>> https://github.com/apache/airflow/issues/15635 and opening up Airflow
>>> to be used on ARM-based linux.
>>>
>>> I hope in 2.3.0 we will be able to release the Airflow Production ARM
>>> image.
>>>
>>> J.
>>>
>>>
Re: [ANNOUNCEMENT] Support for MacOS M1 in Breeze
Posted by Ping Zhang <pi...@umich.edu>.
This is HUGE!! thanks Jarek. I have been suffering from slowness for months.
Thanks,
Ping
On Thu, Mar 10, 2022 at 3:19 AM Kevin Yang <yr...@gmail.com> wrote:
> Excellent! Thanks to Jarek and everyone involved for working on getting us
> better support on ARM-based Linux. 🙏 Looking forward for the upcoming
> improvements 👍
>
> On Thu, Mar 10, 2022 at 3:13 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Hello Everyone,
>>
>> Another day, another announcement. I have just merged this PR
>> https://github.com/apache/airflow/pull/22127 that adds support for
>> development for Mac OS M1 users (and also anyone who uses ARM-based linux.
>>
>> The M1 users have (rightfully) complained about rather poor performance
>> for Dockerized Airflow in Breeze (the intel-based images were used via
>> emulation and suffered roughly ~ 10x slower speed which made them barely
>> usable for development).
>>
>> As of today, our images used for development in Breeze are multi-platform
>> (amd64/arm64). This is - for now - development only and has some
>> limitations (mysql and mssql support is missing until mysql and mssql
>> release arm-based clients and servers - but we have full postgres and
>> sqlite support - including native ARM postgres server Docker image).
>>
>> Some initial teething problems have been fixed (thanks Elad/Ephraim :)
>> but also as a follow up I will need to observe it and implement a few
>> build optimizations (next few days). So if you see any problems, delays
>> etc, ping me on #breeze channel in our slack.
>>
>> Also building ARM images on Intel also suffers from the same 10x slower
>> emulation problems so we need to modify our CI / self-hosted
>> infrastructure, but once we do and once we start testing ARM on CI, we will
>> also be able to release multi-platform "release" images, completing
>> https://github.com/apache/airflow/issues/15635 and opening up Airflow to
>> be used on ARM-based linux.
>>
>> I hope in 2.3.0 we will be able to release the Airflow Production ARM
>> image.
>>
>> J.
>>
>>
Re: [ANNOUNCEMENT] Support for MacOS M1 in Breeze
Posted by Kevin Yang <yr...@gmail.com>.
Excellent! Thanks to Jarek and everyone involved for working on getting us
better support on ARM-based Linux. 🙏 Looking forward for the upcoming
improvements 👍
On Thu, Mar 10, 2022 at 3:13 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hello Everyone,
>
> Another day, another announcement. I have just merged this PR
> https://github.com/apache/airflow/pull/22127 that adds support for
> development for Mac OS M1 users (and also anyone who uses ARM-based linux.
>
> The M1 users have (rightfully) complained about rather poor performance
> for Dockerized Airflow in Breeze (the intel-based images were used via
> emulation and suffered roughly ~ 10x slower speed which made them barely
> usable for development).
>
> As of today, our images used for development in Breeze are multi-platform
> (amd64/arm64). This is - for now - development only and has some
> limitations (mysql and mssql support is missing until mysql and mssql
> release arm-based clients and servers - but we have full postgres and
> sqlite support - including native ARM postgres server Docker image).
>
> Some initial teething problems have been fixed (thanks Elad/Ephraim :) but
> also as a follow up I will need to observe it and implement a few build
> optimizations (next few days). So if you see any problems, delays etc, ping
> me on #breeze channel in our slack.
>
> Also building ARM images on Intel also suffers from the same 10x slower
> emulation problems so we need to modify our CI / self-hosted
> infrastructure, but once we do and once we start testing ARM on CI, we will
> also be able to release multi-platform "release" images, completing
> https://github.com/apache/airflow/issues/15635 and opening up Airflow to
> be used on ARM-based linux.
>
> I hope in 2.3.0 we will be able to release the Airflow Production ARM
> image.
>
> J.
>
>