You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/03 08:44:28 UTC
[GitHub] [airflow] andormarkus opened a new issue #15635: ARM64 support in docker images
andormarkus opened a new issue #15635:
URL: https://github.com/apache/airflow/issues/15635
**Description**
ARM64 support in docker images
**Use case / motivation**
In the recent years ARM based cpu popularity hugely increased. The arm based cpus are offering better price to performance ratio and laptops outstanding performance. It's getting popular to run ARM nodes in your Kubernetes clusterss.
If you want to run airflow on arm node in kubernetes the docker image wont start it wailt fails with `standard_init_linux.go:219: exec user process caused: exec format error`.
ARM based cpu in the database world like [AWS Graviton](https://aws.amazon.com/ec2/graviton/) or in the consumer world like [Apple M1](https://www.apple.com/newsroom/2020/11/apple-unleashes-m1/).
**Related Issues**
In #14796 @potiuk confirmed there is no office support for ARM64 support in docker images
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] FyZzyss edited a comment on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
FyZzyss edited a comment on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-847013588
I managed to run airflow 2.0 in docker on the apple m1 with python 3.8.2.
This was helped by upgrading docker to version [3.3.2](https://docs.docker.com/docker-for-mac/release-notes/#docker-desktop-332) with qemu support and [adding more resources to docker](https://github.com/puckel/docker-airflow/issues/631).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-1005975384
I am getting closer to attempting it. For now I need to merge a series of PRs for docker image optimisation and mostly this one finally - #20664 -> this will change build infrastructure for our image to buildx/buildkit which opens up the built to be mutliplatform. Keep your fingers crossed.
BTW. My new MacPro is bare usable without it so I have some incentive to make it happen :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andormarkus commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-831223265
We could install and run `numpy` on AWS ARM without issue, however installing `pyarrow` 2.0.0 and 3.0.0 was not possible without serious workarounds. Pyarrow 4.0.0 [support](https://issues.apache.org/jira/browse/ARROW-10349) aarch64.
@potiuk Can airflow move to pyarrow 4.0.0 ? Should I create a separate ticket issue for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-831183055
> The biggest obstacle is likely not Airflow itself (which would likely run on ARM just fine), but the many dependencies it has. Either upstreams need to publish ARM wheels (although my understanding is the most difficult-to-build ones already do), or Airflow needs to develop infrastructures to build the images.
Correct. We tried it before and we had limited success previously. The biggest problems were pyarrow/numpy and related especially for the older versions. But with the recent dependency bumps for pyarrow i think this problem might be gone already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Mr-YYM commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
Mr-YYM commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-1001493000
```
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
ce252f9769da airflow_airflow-webserver_1 107.54% 997.1MiB / 5.8GiB 16.79% 1.77MB / 1.21MB 114MB / 24.6kB 41
efeef0b18e3a airflow_airflow-worker_1 94.19% 716.1MiB / 5.8GiB 12.06% 25.8kB / 44kB 36.2MB / 0B 33
45f05042fd21 airflow_airflow-scheduler_1 100.43% 587.2MiB / 5.8GiB 9.89% 767kB / 889kB 32MB / 4.1kB 26
76495df5987e airflow_airflow-triggerer_1 86.23% 264.5MiB / 5.8GiB 4.45% 381kB / 459kB 38.5MB / 0B 9
d50abc5e2641 airflow_flower_1 0.04% 255.9MiB / 5.8GiB 4.31% 26.2kB / 65.5kB 57.4MB / 0B 28
f2d578f6c774 airflow_redis_1 0.08% 7.754MiB / 5.8GiB 0.13% 96.1kB / 33.6kB 12.2MB / 0B 5
916b26b80728 airflow_postgres_1 1.64% 53.78MiB / 5.8GiB 0.91% 2.58MB / 2.92MB 34.8MB / 2.25MB 14
```
when running with emulation, CPU is high
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-831117088
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-1020051090
And also copying my comment from https://github.com/airflow-helm/charts/issues/488#issuecomment-1020015439
Just to explain where it comes from.
The problem is mainly because many of the dependencies are scrambling to get proper "architecture" support. This is a good thing - long term - that Apple switched to ARM (more choice is a good thing) and it pretty much forced all the developers to pay attention.
The ARM support has to - unfortunately - bubble up - from OS support for ARM (already there for linux for quite some time thanks to Android), then low-level libraries in C/C++/Rust, then Python low-level libraries (like Numpy) finally applications and tools like Airflow (or for example MySQL or MSSQL that have to provide clients and images that are also ARM-based). All this is a huge effort by all parties involved - not as much to make the change but mainly to make sure that it works and that they have the right continuous integration in-place to support it, so that they can release the software with confidence after passing all the tests.
I think ~ mid 2022 will be the time where all this effort will be nearly complete and those who won't be there will have to do it as they will lag behind (this is the main reason I deffered buying M1 MacBook as I knew it was very far from being ready when M1 was released first and I hated the touchbar, and lack of HDMI. MagSafe is great BTW and I am glad it is back).
So crossed_fingers that all our dependencies will be ready soon (or at least the crucial ones that will allow Airlfow to reliably and reproducibly build ARM image and add CI harness for it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tikikun commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
tikikun commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-919575669
Hi everyone, is Docker supporting ARM64 now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-1000138886
Thanks @andormarkus - I understand your CI provides ARM (and I am pretty sure those are weeks rather than months that GitHub Actions will provide one - ARM is becoming a "highly reguested" thing. So nice try ;) but GitHub Actions are not a problem here. This is by far the least (and easy to bypass) of our blockers
> As I know this project uses Github actions to build images. Github actions only support amd64 runners does not offer arm64 runner like other CI provider. You can build arm64 images on amd64 platform with the help on Docker BuildX however it will have a 3-4X time penalty compared to a native arm64 runner.
Actually we build our images for CI tests with our self-hosted runners of Amazon. So we could set the pipelines on GitHub actions using those. We will just need to build a bit of infrastructure for that.
The actual blocker here is native support for ARM for a number of libraries and tools we are using. Airflow has 550+ pyhon dependencies, where some of those have native c/C++ libraries and need precompiled versions. We have complex cross-dependencies which simply make it complex to get the right set that will work.
For example one of the things that "blocks us" to hav full ARM support is lack of ARM images for MySQL https://bugs.mysql.com/bug.php?id=103462.
However, while we won't be able to get "full" support" for ARM yet for all the dependencies I am pretty confident we can get "selective" support. Python 3.9 + Postgres + subset of providers should actually work.
I am going to take a stab on it after Xmas, however. I even have a new MacBook PRO to play a bit with that. This will require a few upgrades to our build infrastructure:
* adapt our Dockerfiles to support multi-platform builds
* changing build infrastructure to use BUILDX and produce multiplatform images (including proper caching)
* adding posibility to select test scope depending on the Architecture
* adding possibility to select and build constraints that will depend on the architecture (I am pretty sure we will have different set of constraints for our dependencies for different architectures)
* converting our self-hosted AMIs to be ARM-based
All that is needed to make it works. The final "result" that you see as an image seems like "simple" thing to do. But in order do that a set of prerequisites need to be fulfilled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-831185123
Just checked pyarrow 4.0.0 has aarch wheels. We are using 3.0 now so maybe it's not yet solved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-1020003975
FYI. One of the last "serious" blockers for ARM image is `numpy`. The <1.21 limit added by Apache Beam is going to be removed once Apache Beam releases next version, which is planned for first week of February https://github.com/apache/airflow/issues/19059#issuecomment-1019579091
I could - potentialy - exclude Beam (we did that in the past), but since we know it will be fixed in weeks, we should wait. I will attempt to prepare ARM image right after (cc: @dstandish :))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zeevo commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
zeevo commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-865450843
> I managed to run airflow 2.0 in docker on the apple m1 with python 3.8.2.
> This was helped by upgrading docker to version [3.3.2](https://docs.docker.com/docker-for-mac/release-notes/#docker-desktop-332) with qemu support and [adding more resources to docker](https://github.com/puckel/docker-airflow/issues/631).
Woah! I'm trying to run Airflow on arm64 docker. How did you do it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-831184024
> Airflow needs to develop infrastructures to build the images.
Correct. We have credits from GCP that we will be able to use to run Runners in the ARM infrastructure. We are now running out of S3 credits (trying to get some more) and I am working with Ash, Bin and Claudio on switching to GCP. I think infra won't be a problem :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zeevo commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
zeevo commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-865450843
> I managed to run airflow 2.0 in docker on the apple m1 with python 3.8.2.
> This was helped by upgrading docker to version [3.3.2](https://docs.docker.com/docker-for-mac/release-notes/#docker-desktop-332) with qemu support and [adding more resources to docker](https://github.com/puckel/docker-airflow/issues/631).
Woah! I'm trying to run Airflow on arm64 docker. How did you do it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-847015843
Ah cool. Any special considerations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andormarkus commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-1020027012
@potiuk We are waiting for this almost a year, plus weeks does not matters.
Will this be part of 2.2.4 or we need to wait till 2.2.5?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-831154219
Absolutely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-1000138886
Thanks @andormarkus - I understand your CI provides ARM. And I am pretty sure those are weeks rather than months that GitHub Actions will provide one - ARM is becoming a "highly reguested" thing. So nice try ;) but GitHub Actions are not a problem here. This is by far the least (and easy to bypass) of our blockers
> As I know this project uses Github actions to build images. Github actions only support amd64 runners does not offer arm64 runner like other CI provider. You can build arm64 images on amd64 platform with the help on Docker BuildX however it will have a 3-4X time penalty compared to a native arm64 runner.
Actually we build our images for CI tests with our self-hosted runners of Amazon. So we could set the pipelines on GitHub actions using those. We will just need to build a bit of infrastructure for that.
The actual blocker here is native support for ARM for a number of libraries and tools we are using. Airflow has 550+ pyhon dependencies, where some of those have native c/C++ libraries and need precompiled versions. We have complex cross-dependencies which simply make it complex to get the right set that will work.
Another thing that "blocks us" to hav full ARM support is lack of ARM images for MySQL https://bugs.mysql.com/bug.php?id=103462.
However, while we won't be able to get "full" support" for ARM yet for all the dependencies I am pretty confident we can get "selective" support. Python 3.9 + Postgres + subset of providers should actually work.
I am going to take a stab on it after Xmas, however. I even have a new MacBook PRO to play a bit with that. This will require a few upgrades to our build infrastructure:
* adapt our Dockerfiles to support multi-platform builds
* changing build infrastructure to use BUILDX and produce multiplatform images (including proper caching)
* adding posibility to select test scope depending on the Architecture
* adding possibility to select and build constraints that will depend on the architecture (I am pretty sure we will have different set of constraints for our dependencies for different architectures)
* converting our self-hosted AMIs to be ARM-based
All that is needed to make it works. The final "result" that you see as an image seems like "simple" thing to do. But in order do that a set of prerequisites need to be fulfilled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zeevo commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
zeevo commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-1000043024
Does anyone have an ideas or Dockerfiles to share for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] FyZzyss commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
FyZzyss commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-847025278
> Ah cool. Any special considerations?
There were no more problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-920158459
Not yet (natively). The Docker on MacOS with ARM however should work with emulation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-831155734
Been on my radar for quite some time. We were mostly waiting for getting the docker into stable version for MacOS. And yeah I think this is the next 'big' thing for the images once we get full PIP21 compatibility and python 3.9 support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tikikun commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
tikikun commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-1005236541
Has anyone done this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] FyZzyss edited a comment on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
FyZzyss edited a comment on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-847013588
I managed to run airflow 2.0 on the apple m1 with python 3.8.2.
This was helped by upgrading docker to version [3.3.2](https://docs.docker.com/docker-for-mac/release-notes/#docker-desktop-332) with qemu support and [adding more resources to docker](https://github.com/puckel/docker-airflow/issues/631).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-1020049959
I think 2.3.0, realistically
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zeevo commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
zeevo commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-1000566891
@potiuk
> I am going to take a stab on it after Xmas, however. I even have a new MacBook PRO to play a bit with that. This will require a few upgrades to our build infrastructure:
🙏 Thanks and good luck!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andormarkus commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-1000126474
As I know this project uses Github actions to build images. Github actions only support amd64 runners does not offer arm64 runner like other CI provider. You can build arm64 images on amd64 platform with the help on Docker BuildX however it will have a 3-4X time penalty compared to a native arm64 runner.
To make sure Airflow running on arm64 without hiccup you needs to rerun all of the test on arm64 runner as well. I don't know how do that on Github with amd64 runner. Our CI provider has amd64 and arm64 runners and we double run our test to make sure both platform is fully supported.
Our CI provider has a very generous open source but migrating this project to other CI provider would be a lot of work. To run Airflow on Amazon Graviton (other public provider does not have arm offering yet) would not have significant impact on our monthly bill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr edited a comment on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
uranusjr edited a comment on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-831158297
The biggest obstacle is likely not Airflow itself (which would likely run on ARM just fine), but the many dependencies it has. Either upstreams need to publish ARM wheels (although my understanding is the most difficult-to-build ones already do), or Airflow needs to develop infrastructures to build the images.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-831158297
The biggest obstacle is likely not Airflow itself (which would likely run on ARM just fine), but the many dependencies it has. For macOS, upstreams need to publish ARM wheels (although my understanding is the most difficult-to-build ones already do), and for Docker Airflow will need to either push the upstream for support or develop infrastructures to build the images.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-1000138886
Thanks @andormarkus - I understand your CI provides ARM. And I am pretty sure those are weeks rather than months that GitHub Actions will provide one - ARM is becoming a "highly reguested" thing. So nice try ;) but GitHub Actions are not a problem here. This is by far the least (and easy to bypass) of our blockers
> As I know this project uses Github actions to build images. Github actions only support amd64 runners does not offer arm64 runner like other CI provider. You can build arm64 images on amd64 platform with the help on Docker BuildX however it will have a 3-4X time penalty compared to a native arm64 runner.
Actually we build our images for CI tests with our self-hosted runners of Amazon. So we could set the pipelines on GitHub actions using those. We will just need to build a bit of infrastructure for that.
The actual blocker here is native support for ARM for a number of libraries and tools we are using. Airflow has 550+ pyhon dependencies, where some of those have native c/C++ libraries and need precompiled versions. We have complex cross-dependencies which simply make it complex to get the right set that will work.
For example one of the things that "blocks us" to hav full ARM support is lack of ARM images for MySQL https://bugs.mysql.com/bug.php?id=103462.
However, while we won't be able to get "full" support" for ARM yet for all the dependencies I am pretty confident we can get "selective" support. Python 3.9 + Postgres + subset of providers should actually work.
I am going to take a stab on it after Xmas, however. I even have a new MacBook PRO to play a bit with that. This will require a few upgrades to our build infrastructure:
* adapt our Dockerfiles to support multi-platform builds
* changing build infrastructure to use BUILDX and produce multiplatform images (including proper caching)
* adding posibility to select test scope depending on the Architecture
* adding possibility to select and build constraints that will depend on the architecture (I am pretty sure we will have different set of constraints for our dependencies for different architectures)
* converting our self-hosted AMIs to be ARM-based
All that is needed to make it works. The final "result" that you see as an image seems like "simple" thing to do. But in order do that a set of prerequisites need to be fulfilled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tikikun edited a comment on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
tikikun edited a comment on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-919575669
Hi everyone, is Airflow supporting ARM64 now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] FyZzyss commented on issue #15635: ARM64 support in docker images
Posted by GitBox <gi...@apache.org>.
FyZzyss commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-847013588
I managed to run airflow 2.0 on the apple m1.
This was helped by upgrading docker to version [3.3.2](https://docs.docker.com/docker-for-mac/release-notes/#docker-desktop-332) with qemu support and [adding more resources to docker](https://github.com/puckel/docker-airflow/issues/631).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org