[GitHub] [airflow] potiuk commented on pull request #30705: Optimize parallel test execution for Google, Amazon, WWW tests

["potiuk (via GitHub)" <gi...@apache.org>]

potiuk commented on PR #30705:
URL: https://github.com/apache/airflow/pull/30705#issuecomment-1512973322

   This one is the next optimisation after the #29223 - and it should bring our "main" test execution time on sel-hosted runners below 20 minutes (I hope - let's see). 
   
   
   Over the last couple of months Amazon tests grew, and with moto being powerful (but rather slow) mocking framework, they started to take the biggest chunk of Provider test time turning Providers test type into by far longest test type to run, when tests are run in parallel on our self-hosted runners. I saw that when looking at the test-output that often last few minutes of the tests were only "Providers" test type running, while all the others were completeed. This brought utilisation of our 8 CPU machines down quite a lot and increased elapsed time of time execution.
   
   
   By adding this change, where I separate amazon, google  out from "other" provider tests we should get back to the situation that the test types are more-or-less even (With amazon still being slowest likely) so that the utilization of the machines might get back to close 100% all the time and elapsed time should go down.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org