You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "juditnovak (via GitHub)" <gi...@apache.org> on 2025/12/13 16:27:52 UTC
[PR] [TESTENV] Pytest plugin allowing to load custom Airflow config [airflow]
juditnovak opened a new pull request, #59389:
URL: https://github.com/apache/airflow/pull/59389
---
## Proposal: Airflow Pytest Plugin allowing to load custom configuration
Currently the Airflow confguration workflows, combined with the Airflow Pytest Plugin restrain test execution very strictly to pre-configured defaults.
While unit tests can easily overcome this issue (using the `conf_vars` fixture) there is no workaround when using Airflow System Tests.
It's highly understandable to make sure no mistake can happen across test and perhaps other deployment configuration settings.
On the other hand, a clearly indicated pytest option (only impacting the scope of the given test run) may be a sufficiently safe bet to avoid interference with non-testing Airflow runs. The advantage of the pytest CLI argument is an elegant, clean usage on pipelines.
(Currently the same impact can only be achieved by re-loading the desired config within test space.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [TESTENV] Pytest plugin allowing to load custom Airflow config [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #59389:
URL: https://github.com/apache/airflow/pull/59389#issuecomment-3952705275
I think that one is pretty abstract - I have no idea exactly how this one solves any problem. Good example (possibly added in the docs) explaining example usage would be helpful. Generally when we add a feature to our test framework, we also add description in the contributiing documentation explaining how to use it and where it is helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [TESTENV] Pytest plugin allowing to load custom Airflow config [airflow]
Posted by "juditnovak (via GitHub)" <gi...@apache.org>.
juditnovak commented on PR #59389:
URL: https://github.com/apache/airflow/pull/59389#issuecomment-4010965831
Dear @potiuk , when you get a moment please let me know what do you think of my response on your question. In case further details/examples may be needed, I'm glad to go into more detail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [TESTENV] Pytest plugin allowing to load custom Airflow config [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #59389:
URL: https://github.com/apache/airflow/pull/59389#issuecomment-4033375885
Can you please describe it in the documentation where we explain system tests ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [TESTENV] Pytest plugin allowing to load custom Airflow config [airflow]
Posted by "juditnovak (via GitHub)" <gi...@apache.org>.
juditnovak commented on PR #59389:
URL: https://github.com/apache/airflow/pull/59389#issuecomment-3953139073
I referred to this proposal in comment https://github.com/apache/airflow/pull/59390#discussion_r2616410567 (I marked it resolved after the approval I recall).
In PR https://github.com/apache/airflow/pull/59390, I added to the System Test documentation a new section: [Custom configuration file](https://github.com/apache/airflow/blob/main/contributing-docs/testing/system_tests.rst#custom-configuration-file), which currently presents the single option of a "hard-coded" way to load configuration dedicated for System Tests.
This proposal is to offer a more elegant and handy solution for this purpose.
Use case: ability to load a custom, system-test-specific Airflow configuration file from a specific location when invoking the execution of the System Tests.
(Existing example: in the System Test pipelines for project https://github.com/juditnovak/airflow-provider-nomad I'm currently using the ['hard-coded' solution](https://github.com/juditnovak/airflow-provider-nomad/blob/main/tests/system/nomad/conftest.py#L303)...)
Assuming that other devs may have also come across the issue, it may be convenient to have the option enabled by the Airflow System Tests plugin itself.
(Also, an advantage of this solution is that it is different from any existing options for loading Airflow configuration files that apply to production Airflow instances, ensuring to avoid "potential mistakes".)
Of course, if this proposal is to be taken onboard, I'm adding to this very same PR the corresponding docs changes as well, targeting the section mentioned above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [TESTENV] Pytest plugin allowing to load custom Airflow config [airflow]
Posted by "juditnovak (via GitHub)" <gi...@apache.org>.
juditnovak commented on PR #59389:
URL: https://github.com/apache/airflow/pull/59389#issuecomment-3950806189
Dear @potiuk , This proposal is linked with https://github.com/apache/airflow/pull/59390 that has been merged lately.
No urgency, please let me know your thoughts when you've got a moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [TESTENV] Pytest plugin allowing to load custom Airflow config [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #59389:
URL: https://github.com/apache/airflow/pull/59389#issuecomment-3808243401
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [TESTENV] Pytest plugin allowing to load custom Airflow config [airflow]
Posted by "juditnovak (via GitHub)" <gi...@apache.org>.
juditnovak commented on code in PR #59389:
URL: https://github.com/apache/airflow/pull/59389#discussion_r2616397877
##########
devel-common/src/tests_common/pytest_plugin.py:
##########
@@ -415,6 +421,18 @@ def initialize_airflow_tests(request):
print(f"Home of the user: {home}\nAirflow home {airflow_home}")
+ if request.config.option.load_config:
+ from airflow.configuration import conf, load_standard_airflow_configuration, AIRFLOW_CONFIG
+ saved_af_conf = AIRFLOW_CONFIG
+ AIRFLOW_CONFIG = request.config.option.load_config
+ try:
+ load_standard_airflow_configuration(conf)
+ except:
+ raise
+ finally:
+ AIRFLOW_CONFIG = saved_af_conf
+ conf.validate()
Review Comment:
The snippet above is just one possible solution, to demonstrate the base concept (and may not be the best -- feedback is welcome).
On my current pipelines I'm using `conf.read_file()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [TESTENV] Pytest plugin allowing to load custom Airflow config [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #59389:
URL: https://github.com/apache/airflow/pull/59389
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org