You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/03 11:12:20 UTC
[GitHub] [airflow] potiuk opened a new pull request, #24813: Run ARM building way faster
potiuk opened a new pull request, #24813:
URL: https://github.com/apache/airflow/pull/24813
Since we are now building ARM images in parallel, We need more powerful
machines and we implemented in-memory docker, similarly as in our AMD
instances. The m6g.2xlarge are quite a bit better than c6g.xlarge for
our case:
1) They have 8 vCPUs
2) They have 32 GB memory (should be enough to build 4 ARM CI images
3) Thye are Memory-optimised, and since docker is build in memory
the memory speed is the most important factor
This also allows to switch building all images (including cache)
in parallel - so that we can have 1 job instead of 4 - similarly
as we have in case of regular AMD builds.
Another advantage of it is that we loose far less time of the
AMD instance which "triggers" the docker build, because this instance
will control 4 parallel builds at a time effectively, which will
decrease a lot of overhead connected with running the instance mostly
idle during the build (and since the builds will be generally
faster, the overhead will be even smaller).
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24813: Prepare ARM images much faster and cheaper
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24813:
URL: https://github.com/apache/airflow/pull/24813#issuecomment-1173126070
Also Preparing cache will be much faster (same reason) - the "release builds" should also run quite a bit faster but they are already using cache so I expect those speedups to be smaller (12 minutes instead of 16 more or less).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #24813: Prepare ARM images much faster and cheaper
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #24813:
URL: https://github.com/apache/airflow/pull/24813
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24813: Prepare ARM images much faster and cheaper
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24813:
URL: https://github.com/apache/airflow/pull/24813#issuecomment-1173125841
This change heavily optimize our ARM building experience (especially for regular PRs that will change setup.py).
Currently ARM image building is performed in those cases:
* In regular PRs when they are changing dependencies
* In main build (to make sure that dependency upgrades did not break anything)
* In "Production image release"
I've used similar approach as we have for our AMD machines. We are using bigger instances (8 vCPUS + 32 GB RAM) and building docker is fully performed in RAM. That gives tremendous improvements comparing to the previous (4 CPUS and using disks) for two reasons:
* Docker does not touch disk when writing layers during the build
* there are a number of cases where dependencies are build for ARM (for example oracle python libraries) and they use paralell compilation in memory which is way faster. (during package instalation, the machines wiht 8 vCPUS are mostly 60% - 90% busy)
You can see result of it here: https://github.com/apache/airflow/runs/7169575771?check_suite_focus=true. It took 17 minutes (compared to ~ 1h 20 before). The machines are just 2x more expensive, but they are needed for 4x shorter time, so we will pay 50% of cost with 4x shorter time to build.
Wheninstead of 1h parallel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org