[GitHub] [airflow] potiuk commented on issue #15635: ARM64 support in docker images

[GitBox <gi...@apache.org>]

potiuk commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-1000138886


   Thanks @andormarkus  - I understand your CI provides ARM (and I am pretty sure those are weeks rather than months that GitHub Actions will provide one - ARM is becoming a "highly reguested" thing. So nice try ;) but GitHub Actions are not a problem here. This is by far the least (and easy to bypass) of our blockers
   
   > As I know this project uses Github actions to build images. Github actions only support amd64 runners does not offer arm64 runner like other CI provider. You can build arm64 images on amd64 platform with the help on Docker BuildX however it will have a 3-4X time penalty compared to a native arm64 runner.
   
   Actually we build our images for CI tests with our self-hosted runners of Amazon. So we could set the pipelines on GitHub actions using those. We will just need to build a bit of infrastructure for that.
    
   The actual blocker here is native support for ARM for a number of libraries and tools we are using. Airflow has 550+ pyhon dependencies, where some of those have native c/C++ libraries and need precompiled versions. We have complex cross-dependencies which simply make it complex to get the right set that will work.
   
   For example one of the things that "blocks us" to hav full ARM support is lack of ARM images for MySQL https://bugs.mysql.com/bug.php?id=103462.
   
   However, while we won't be able to get "full" support" for ARM yet for all the dependencies I am pretty confident we can get "selective" support. Python 3.9 + Postgres + subset of providers should actually work.
   
   I am going to take a stab on it after Xmas, however. I even have a new MacBook PRO to play a bit with that. This will require a few upgrades to our build infrastructure:
   
   * adapt our Dockerfiles to support multi-platform builds
   * changing build infrastructure to use BUILDX and produce multiplatform images (including proper caching)
   * adding posibility to select test scope depending on the Architecture
   * adding possibility to select and build constraints that will depend on the architecture (I am pretty sure we will have different set of constraints for our dependencies for different architectures)
   * converting our self-hosted AMIs to be ARM-based
   
   All that is needed to make it works. The final "result" that you see as an image seems like "simple" thing to do. But in order do that a set of prerequisites need to be fulfilled.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org