[I] Secret caching is skipped when DAGs are parsed in Airflow CLI [airflow]

["Birne94 (via GitHub)" <gi...@apache.org>]

Birne94 opened a new issue, #37543:
URL: https://github.com/apache/airflow/issues/37543

   ### Apache Airflow version
   
   main (development)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   Airflow supports the [`secrets.use_cache`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#use-cache) option to speed up DAG file processing. According to the documentation, it "enables local caching of Variables, when parsing DAGs only". After enabling this option we found that the DAG processing is still slow when using the Airflow CLI. Looking at the code, the cache is not used at all because `SecretCache.init()` is only called [as part of DAG processing job](https://github.com/apache/airflow/blob/f2ea8a3e1753012bfe0d529c9c8be66cf55ca28f/airflow/dag_processing/manager.py#L1066C9-L1066C27), not however [when fetching DAGs from the CLI](https://github.com/apache/airflow/blob/f2ea8a3e1753012bfe0d529c9c8be66cf55ca28f/airflow/cli/commands/dag_command.py#L454).
   
   ### What you think should happen instead?
   
   When enabling secret caching, I would expect it to be applied for the CLI as well.
   
   ### How to reproduce
   
   1. enable `secrets.use_cache` in airflow config
   2. have a DAG that accesses variables in top-level code
   3. run `airflow dags list-import-errors`
   4. see that it takes very long to execute (20s in my case)
   
   ### Operating System
   
   Amazon Linux
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Amazon (AWS) MWAA
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Secret caching is skipped when DAGs are parsed in Airflow CLI [airflow]

["eladkal (via GitHub)" <gi...@apache.org>]

eladkal commented on issue #37543:
URL: https://github.com/apache/airflow/issues/37543#issuecomment-1975229021

   As explained in https://github.com/apache/airflow/pull/37551#pullrequestreview-1891141876 its not a bug. It's by design.
   Since its about experimental feature and requires dev list discussion I am closing the issue as the community have no further task here (we try to keep our Github issuew actionable). I encourgae you to follow the instructions left on the PR comment and start a discussion on the mailing list. If your ideas is accepted you can continue working on the PR (no need to have Github issue along side)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Secret caching is skipped when DAGs are parsed in Airflow CLI [airflow]

["boring-cyborg[bot] (via GitHub)" <gi...@apache.org>]

boring-cyborg[bot] commented on issue #37543:
URL: https://github.com/apache/airflow/issues/37543#issuecomment-1952369603

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org