[GitHub] [airflow] potiuk commented on pull request #15062: Parallelize build of documentation.

[GitBox <gi...@apache.org>]

potiuk commented on pull request #15062:
URL: https://github.com/apache/airflow/pull/15062#issuecomment-809171652


   @ashb @kaxil @mik-laj. Good news. Seems that I managed to get doc build parallelization under control. It required dockerizing the build (autoapi relies on _api generated in the sources and the only way I could achieve paralelisation was to isolate sources of Airflow between the parallel docs via docker containers. 
   
   I will still have to work a little on output /formatting and stability but seems that with build parallelisation we can get vast speed improvements as well in doc building time:
   *  we have about 8 minutes now for Self-hosted runners (exluding time needed for setting up the venv which will be removed because of venv caching).
   * we have about 12 minutes for the GitHub Runners. I still have to make sure the GitHub Runners are not outliers  (and possibly I will figure out why the build in Self-hosted runnners is not even faster). 
   
   But anyhow this speedup is significant and we will have further optimisation of our CI infrastructure/cost.
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org