[GitHub] [airflow] potiuk commented on pull request #24813: Prepare ARM images much faster and cheaper

[GitBox <gi...@apache.org>]

potiuk commented on PR #24813:
URL: https://github.com/apache/airflow/pull/24813#issuecomment-1173125841

   This change heavily optimize our ARM building experience (especially for regular PRs that will change setup.py).
   
   Currently ARM image building is performed in those cases:
   
   * In regular PRs when they are changing dependencies
   * In main build (to make sure that dependency upgrades did not break anything)
   * In "Production image release"
   
   I've used similar approach as we have for our AMD machines. We are using bigger instances (8 vCPUS + 32 GB RAM) and building docker is fully performed  in RAM. That gives tremendous improvements comparing to the previous (4 CPUS and using disks) for two reasons:
   
   * Docker does not touch disk when writing layers during the build
   * there are a number of cases where dependencies are build for ARM (for example oracle python libraries) and they use paralell compilation in memory which is way faster. (during package instalation, the machines wiht 8 vCPUS are mostly 60% - 90% busy) 
   
   You can see result of it here: https://github.com/apache/airflow/runs/7169575771?check_suite_focus=true.  It took 17 minutes (compared to ~ 1h 20 before). The machines are just 2x more expensive, but they are needed for 4x shorter time, so we will pay 50% of cost with 4x shorter time to build. 
   
   Wheninstead of 1h parallel


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org