You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/30 15:31:45 UTC

[GitHub] [airflow] thcidale0808 opened a new pull request #21223: Added Feature: Create Pools from Env Vars.

thcidale0808 opened a new pull request #21223:
URL: https://github.com/apache/airflow/pull/21223

<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.

Feel free to ping committers for the review!

In case of existing issue, reference it using one of the following:

closes: #18582
related: #18582

How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
This PR is to create pools when the an environment variable is defined with the standard structure: ```AIRFLOW__<SECTION>__<KEY>``` where the section is ```pool``` and key is the name of the pool.

**^ Add meaningful description above**

Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed pull request #21223: Added Feature: Create Pools from Env Vars.

Posted by GitBox <gi...@apache.org>.

potiuk closed pull request #21223:
URL: https://github.com/apache/airflow/pull/21223


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] thcidale0808 commented on pull request #21223: Added Feature: Create Pools from Env Vars.

Posted by GitBox <gi...@apache.org>.

thcidale0808 commented on pull request #21223:
URL: https://github.com/apache/airflow/pull/21223#issuecomment-1037443067

@potiuk , please find below a proposed solution and answers to your questions:

## Solution:

Create a state file and a pool config sync cli and api to sync to database and config. The state file will be the source of true. When user call the config sync using the cli or api, the core flow will be: [state file → DB (models), state file → conf (hot reload)].

The state file will be generated when airflow is initiated based on the config and env variables.

The state file will be updated when the user create or change a pool using the current UI.

For changes of the environment variables, the user will also need to call the sync process using the new api or cli and pass the boolean argument —env-var. This will update the state file based on the env vars and then execute the core flow.

Example: When user create or update a pool using the UI or the importer, will update the state file and then call the sync process.

## Questions:
Based on the solution above, please find below answer to your questions:

- I updated the environment value and the pool has not been updated ? Why? Is it explained somewhere? How can I update it?

R: Call pool config sync cli or api passing the argument —env-var. This will update the state file and execute the core flow.

- I removed the pool from environment but it has not been removed? Why ? How can I remove the pool without using CLI or UI or API since I could create one ?

R: Similar with previous question. Call the pool config sync cli or api passing the argument —env-var

- How can I rename a pool by changing the environment/config ?

R: For env variables changes, call pool config sync cli or api passing the argument —env-var. This will update the state file and execute the core flow. Similarly, for config change, call the sync using —config

- How can I update pool value without using CLI or UI or API ?

R: Using the current UI. This will also update the state file which will be the source of the true for the sync process.

- If I already updated my pool value via DB, what happens if I change the value in the environment or conf?

R: This will a sync issue. So upon executing the dag use the pool, log a warning letting the user know that there’s a mismatch between the DB and the state file and asking to change the values using the UI or env variables and execute the sync process.