You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/19 21:51:55 UTC

[GitHub] [airflow] potiuk opened a new pull request, #25169: Move javascript compilation to host

potiuk opened a new pull request, #25169:
URL: https://github.com/apache/airflow/pull/25169

   Previously, in order to keep consistent development environment
   we've compiled javascript in the CI image. However we can utilise
   power of pre-commmit for setting the node environment for all
   contributors automatically. Instead of compiling the javascript
   in the image, we can compile it via pre-commit in the host.
   
   This can be done thanks to the new python breeze which is far more
   flexible and can now add execution of compilation of the javascript when
   needed and using pre-commit environments.
   
   Thanks to that, we can vastly simplify the Dockerfiles and scripts that
   are used to automatically build or signal that the assets need
   recompilation. We can basically assume that the assets were prepared
   outside of the image building (and breeze makes sure it happens)
   
   The changes:
   
   * node.js is not needed in images (neither PROD build nor CI)
   * no need for multiple asset compilation scripts. All is done
     via pre-commit environment with `breeze compile-www-assets``
     command
   * lint checks for UI do not need the docker image any more
     (they are also based on pre-commit environment)
   * no more checks/warnings when you enter the image
   * start-airflow command builds the compilation before entering
   * prepare-airflow-package runs asset compilation before entering
     docker airflow building.
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #25169: Move javascript compilation to host

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25169:
URL: https://github.com/apache/airflow/pull/25169#issuecomment-1190732766

   PR here: https://github.com/apache/airflow/pull/25201
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #25169: Move javascript compilation to host

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25169:
URL: https://github.com/apache/airflow/pull/25169#issuecomment-1190557789

   Yep. All good with public runners too. Merging and refreshing the images. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #25169: Move javascript compilation to host

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25169:
URL: https://github.com/apache/airflow/pull/25169#issuecomment-1190397652

   Let's see now :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] bbovenzi commented on pull request #25169: Move javascript compilation to host

Posted by GitBox <gi...@apache.org>.
bbovenzi commented on PR #25169:
URL: https://github.com/apache/airflow/pull/25169#issuecomment-1190372063

   > Hm. Interestingly enough - we seem to have not enough space on our machines to get THREE yarn installs :)
   
   Are we not clearing the cache each time?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #25169: Move javascript compilation to host

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25169:
URL: https://github.com/apache/airflow/pull/25169#issuecomment-1190758347

   Merged. Should be fixed now @MatrixManAtYrService 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #25169: Move javascript compilation to host

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25169:
URL: https://github.com/apache/airflow/pull/25169#issuecomment-1189593748

   cc: @pierrejeambrun @bbovenzi - please take a look, I think you will find this change really cool.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] MatrixManAtYrService commented on pull request #25169: Move javascript compilation to host

Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService commented on PR #25169:
URL: https://github.com/apache/airflow/pull/25169#issuecomment-1190716270

   I have some automation which runs `python setup.py compile_assets bdist_wheel` in service of creating an airflow image for test.  Following this commit, it fails with:
   
   ```
   [Errno 2] No such file or directory: './airflow/www/compile_assets.sh': './airflow/www/compile_assets.sh'
   ```
   
   Is that a bug, or is there a better way for me to be building this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #25169: Move javascript compilation to host

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25169:
URL: https://github.com/apache/airflow/pull/25169#issuecomment-1190158580

   Cool. Yeah. I will . :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #25169: Move javascript compilation to host

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25169:
URL: https://github.com/apache/airflow/pull/25169#issuecomment-1190725953

   A bug. fix is coming 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk merged pull request #25169: Move javascript compilation to host

Posted by GitBox <gi...@apache.org>.
potiuk merged PR #25169:
URL: https://github.com/apache/airflow/pull/25169


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #25169: Move javascript compilation to host

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25169:
URL: https://github.com/apache/airflow/pull/25169#issuecomment-1190374928

   Not when we have three independent yarn builds run in pre-commits - the envs are created in ~/.cache which is rather small it seems, But I am looking into it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #25169: Move javascript compilation to host

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25169:
URL: https://github.com/apache/airflow/pull/25169#issuecomment-1190479489

   Now we are cooking with gas. I will just re-run it with public runners to be absolutely sure the yarn modules will not make it run out of the capacity either.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #25169: Move javascript compilation to host

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25169:
URL: https://github.com/apache/airflow/pull/25169#issuecomment-1190350413

   Hm. Interestingly enough - we seem to have not enough space on our machines to get THREE yarn installs :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #25169: Move javascript compilation to host

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25169:
URL: https://github.com/apache/airflow/pull/25169#issuecomment-1190160454

   Thanks for testing @bbovenzi  - > I think this will help a bit with faster iterations and maybe we could also add some more dev-friendly javascript tooling around to attract more contributors and make it easier to start/test/debug Javascript code. Especially with the typescript and React, I have a feeling we could add some tooling to breeze to improve "first-time-experience" for new users. Happy to brainstorm on it and implement any improvements there :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #25169: Move javascript compilation to host

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25169:
URL: https://github.com/apache/airflow/pull/25169#issuecomment-1190398148

   Added more disk space for the instance (later I can optimize it even more).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org