You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dominikhei (via GitHub)" <gi...@apache.org> on 2023/12/18 20:33:18 UTC
[PR] Added functionality to automatically ingest custom airflow.cfg file upon startup [airflow]
dominikhei opened a new pull request, #36289:
URL: https://github.com/apache/airflow/pull/36289
closes: #35812
I have made the necessary changes to the docker-compose.yaml file as proposed within the issue. The relevant line is currently outcommented, but the documentation in the index.rst and comments above the outcommented line tell users
what to do. If one deletes the comment, an airflow.cfg file within ``${AIRFLOW_PROJ_DIR:-.}`` is now added to ``/opt/airflow/airflow.cfg``within the containers, once ``docker compose up``has been executed. The new line also should make it clear where the airflow.cfg is stored in the containers, as brought up by the issue.
As mentioned, I also added documentation for the changes in the index.rst file, such that users know how to automatically ingest their own airflow.cfg upon startup.
@potiuk Can you maybe have a look at this, since you also approved the original issue?
If I should change anything please let me know :)
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added functionality to automatically ingest custom airflow.cfg file upon startup [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #36289:
URL: https://github.com/apache/airflow/pull/36289#discussion_r1430709131
##########
docs/apache-airflow/howto/docker-compose/docker-compose.yaml:
##########
@@ -61,6 +61,7 @@ x-airflow-common:
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session'
+ TEST:
Review Comment:
Some redundant variable?
##########
docs/apache-airflow/howto/docker-compose/docker-compose.yaml:
##########
@@ -74,6 +75,10 @@ x-airflow-common:
- ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs
- ${AIRFLOW_PROJ_DIR:-.}/config:/opt/airflow/config
- ${AIRFLOW_PROJ_DIR:-.}/plugins:/opt/airflow/plugins
+ #The following line can be used to use a custom airflow.cfg, stored in the AIRFLOW_PROJ_DIR
+ #Uncomment the following line to make use of this
+ #- ${AIRFLOW_PROJ_DIR:-.}/airflow.cfg:/opt/airflow/airflow.cfg
+
Review Comment:
Another option place config, e.g. `awesome.cfg` into the `config` directory and setup [`AIRFLOW_CONFIG`](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#envvar-AIRFLOW_CONFIG) environment variable to `/opt/airflow/config/awesome.cfg`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added functionality to automatically ingest custom airflow.cfg file upon startup [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #36289:
URL: https://github.com/apache/airflow/pull/36289#issuecomment-1861569432
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added functionality to automatically ingest custom airflow.cfg file upon startup [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36289:
URL: https://github.com/apache/airflow/pull/36289#discussion_r1430720155
##########
docs/apache-airflow/howto/docker-compose/docker-compose.yaml:
##########
@@ -74,6 +75,10 @@ x-airflow-common:
- ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs
- ${AIRFLOW_PROJ_DIR:-.}/config:/opt/airflow/config
- ${AIRFLOW_PROJ_DIR:-.}/plugins:/opt/airflow/plugins
+ #The following line can be used to use a custom airflow.cfg, stored in the AIRFLOW_PROJ_DIR
+ #Uncomment the following line to make use of this
+ #- ${AIRFLOW_PROJ_DIR:-.}/airflow.cfg:/opt/airflow/airflow.cfg
+
Review Comment:
Yep. Lokely better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added functionality to automatically ingest custom airflow.cfg file upon startup [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #36289:
URL: https://github.com/apache/airflow/pull/36289#issuecomment-1862700441
Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added functionality to automatically ingest custom airflow.cfg file upon startup [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #36289:
URL: https://github.com/apache/airflow/pull/36289#discussion_r1431244401
##########
docs/apache-airflow/howto/docker-compose/docker-compose.yaml:
##########
@@ -69,11 +69,15 @@ x-airflow-common:
# WARNING: Use _PIP_ADDITIONAL_REQUIREMENTS option ONLY for a quick checks
# for other purpose (development, test and especially production usage) build/extend Airflow image.
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
+ #The following line can be used to set a custom config file, stored in the local config folder
+ #If you want to use it, outcomment it and replace airflow.cfg with the name of your config file
+ #AIRFLOW_CONFIG: '/opt/airflow/config/airflow.cfg'
volumes:
- ${AIRFLOW_PROJ_DIR:-.}/dags:/opt/airflow/dags
- ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs
- ${AIRFLOW_PROJ_DIR:-.}/config:/opt/airflow/config
- ${AIRFLOW_PROJ_DIR:-.}/plugins:/opt/airflow/plugins
+
Review Comment:
```suggestion
```
##########
docs/apache-airflow/howto/docker-compose/docker-compose.yaml:
##########
@@ -69,11 +69,15 @@ x-airflow-common:
# WARNING: Use _PIP_ADDITIONAL_REQUIREMENTS option ONLY for a quick checks
# for other purpose (development, test and especially production usage) build/extend Airflow image.
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
+ #The following line can be used to set a custom config file, stored in the local config folder
+ #If you want to use it, outcomment it and replace airflow.cfg with the name of your config file
+ #AIRFLOW_CONFIG: '/opt/airflow/config/airflow.cfg'
Review Comment:
```suggestion
# The following line can be used to set a custom config file, stored in the local config folder
# If you want to use it, outcomment it and replace airflow.cfg with the name of your config file
# AIRFLOW_CONFIG: '/opt/airflow/config/airflow.cfg'
```
##########
docs/apache-airflow/howto/docker-compose/docker-compose.yaml:
##########
@@ -41,7 +41,7 @@
# A better way is to build a custom image or extend the official image
# as described in https://airflow.apache.org/docs/docker-stack/build.html.
# Default: ''
-#
+#
Review Comment:
```suggestion
#
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added functionality to automatically ingest custom airflow.cfg file upon startup [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #36289:
URL: https://github.com/apache/airflow/pull/36289
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added functionality to automatically ingest custom airflow.cfg file upon startup [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #36289:
URL: https://github.com/apache/airflow/pull/36289#discussion_r1431294413
##########
docs/apache-airflow/howto/docker-compose/index.rst:
##########
@@ -340,6 +340,19 @@ that conflicts with the version of apache-airflow that you are using.
Run ``docker compose build`` to build the image, or add ``--build`` flag to ``docker compose up`` or
``docker compose run`` commands to build the image automatically as needed.
+Special case - Adding a custom config file
+==========================================
+
+If you have a custom config file and wish to use it in your Airflow instance, you need to perform the following steps:
+
+1) Remove comment from the ``AIRFLOW_CONFIG: '/opt/airflow/config/airflow.cfg'`` line
+ in the ``docker-compose.yaml`` file.
Review Comment:
```suggestion
1) Remove comment from the ``AIRFLOW_CONFIG: '/opt/airflow/config/airflow.cfg'`` line
in the ``docker-compose.yaml`` file.
```
##########
docs/apache-airflow/howto/docker-compose/index.rst:
##########
@@ -340,6 +340,19 @@ that conflicts with the version of apache-airflow that you are using.
Run ``docker compose build`` to build the image, or add ``--build`` flag to ``docker compose up`` or
``docker compose run`` commands to build the image automatically as needed.
+Special case - Adding a custom config file
+==========================================
+
+If you have a custom config file and wish to use it in your Airflow instance, you need to perform the following steps:
+
+1) Remove comment from the ``AIRFLOW_CONFIG: '/opt/airflow/config/airflow.cfg'`` line
+ in the ``docker-compose.yaml`` file.
+
+2) Place your custom ``airflow.cfg`` file in the local config folder.
+
+3) If your config file has a different name than ``airflow.cfg``, adjust the filename in
Review Comment:
```suggestion
3) If your config file has a different name than ``airflow.cfg``, adjust the filename in
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org