You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/23 23:01:35 UTC
[GitHub] [airflow] potiuk opened a new pull request, #23195: Nicer handling of cached Breeze parameters
potiuk opened a new pull request, #23195:
URL: https://github.com/apache/airflow/pull/23195
The cached parameters in Python Breeze were largely based on
Bash implementation. They did the job but required pretty
cumbersome synchronization of cached values with parameters
passed and it was easy to forget about this as you had to
do it sepearately in each method that had potentially
cacheable parameters.
In this PR we take advantage of the Click class hiarchy and their
extendability. We've already extended Click Choice parameter
to be much better formatted for long list of choices but we take
this a bit further now with adding new type of parameter that
can cache the values between runs.
This has multiple advantages:
* we do not have to remember about synchroniation - parameters
automatically read/write their values from cache as they are
used
* we can automatically set parameters to default when wrong
value is passed. This is nice as user does not have to
re-run the command because the values are corrected
on-the-flight.
* the parameters can have their default values displayed in
help screen (we use sentinel default that we can use to
detect if value was passed from parameter or taken from default)
* the parameters can also have their <current> values marked in
list of choices - so the user in the help screen can see not
only the default but also which value is currently selected
and will be used if you do not pass any parameter.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragement file, named `{pr_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #23195: Nicer handling of cached Breeze parameters
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23195:
URL: https://github.com/apache/airflow/pull/23195#issuecomment-1108154399
merged as part of #23205
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #23195: Nicer handling of cached Breeze parameters
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23195:
URL: https://github.com/apache/airflow/pull/23195#issuecomment-1107663019
This has been bothered me a bit - our implementation of parameter caching with synchronization was a bit convoluted (mostly because it was based on the Bash implementation).
I took a stab at it and it turned out that we can rather easily do it "properly" i.e. extending Click `Choice` to make the parameters much "smarter" -now it is enough do declare the parameter as "CacheableChoice" and it does all the between-session caching of last used values- and it even can nicely integrate with the help printed by rich-click in the way that we can see both - default and current values in the --help output"
@Bowrna -> you might want to take a look and see that this is a really nice improvement :).
@blag -> you might also take some inspiration when convert Airlfow CLI. The Click + Rich Click combination is awesome and wonderfully and easily extensible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #23195: Nicer handling of cached Breeze parameters
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #23195:
URL: https://github.com/apache/airflow/pull/23195#discussion_r857043943
##########
dev/breeze/tests/test_build_image.py:
##########
@@ -1,60 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
Those tests can be deleted now. We completely decoupled caching parameters from creating BuildProdParams, BuildCIParams - by the time they get created, the cache retrieval/writing is already processed via `CacheableChoice` and those tests make no sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #23195: Nicer handling of cached Breeze parameters
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #23195: Nicer handling of cached Breeze parameters
URL: https://github.com/apache/airflow/pull/23195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #23195: Nicer handling of cached Breeze parameters
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23195:
URL: https://github.com/apache/airflow/pull/23195#issuecomment-1108307291
Already merged as part of #23205
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org