You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/04 10:52:58 UTC
[GitHub] [airflow] potiuk opened a new pull request #17409: Improves image building documentation for new users
potiuk opened a new pull request #17409:
URL: https://github.com/apache/airflow/pull/17409
This PR improves documentation for building images of airflow,
specifically targetting users who do not have big experience with
building the images. It shows examples on how custom image building
can be easily used to upgrade provider packages as well as how
image building can be easily integrated in quick-start using
docker-compose.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17409: Improve image building documentation for new users
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17409:
URL: https://github.com/apache/airflow/pull/17409#issuecomment-892942516
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #17409: Improve image building documentation for new users
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #17409:
URL: https://github.com/apache/airflow/pull/17409#discussion_r682855059
##########
File path: docs/apache-airflow/start/docker.rst
##########
@@ -81,6 +81,18 @@ If you need install a new Python library or system library, you can :doc:`build
.. _initializing_docker_compose_environment:
+Using custom images
+===================
+
+When you want to run Airflow locally, you might want to use extended image, containing some additional dependencies - for
+example you might add new python packages, or upgrade airflow providers to a later version. This can be done very easily
+by placing your custom Dockerfile. Then you can use `docker-compose build` command to build your image (you need to
+do it only once) and you can also add `--build` to your `docker-compose` commands with flag to rebuild the images
+on-the-fly when you run other `docker-compose` commands.
+
+The examples of how you can extend the image with custom providers, python packages,
Review comment:
```suggestion
Examples of how you can extend the image with custom providers, python packages,
```
##########
File path: docs/docker-stack/build.rst
##########
@@ -250,6 +250,19 @@ You should be aware, about a few things:
Examples of image extending
---------------------------
+Example of upgrading Airflow Provider packages
+..............................................
+
+The :ref:`Airflow Providers <providers:community-maintained-providers>` are released independently of the main
Review comment:
```suggestion
The :ref:`Airflow Providers <providers:community-maintained-providers>` are released independently of core
```
##########
File path: docs/apache-airflow/start/docker.rst
##########
@@ -81,6 +81,18 @@ If you need install a new Python library or system library, you can :doc:`build
.. _initializing_docker_compose_environment:
+Using custom images
+===================
+
+When you want to run Airflow locally, you might want to use extended image, containing some additional dependencies - for
Review comment:
```suggestion
When you want to run Airflow locally, you might want to use an extended image, containing some additional dependencies - for
```
##########
File path: docs/apache-airflow/start/docker.rst
##########
@@ -81,6 +81,18 @@ If you need install a new Python library or system library, you can :doc:`build
.. _initializing_docker_compose_environment:
+Using custom images
+===================
+
+When you want to run Airflow locally, you might want to use extended image, containing some additional dependencies - for
+example you might add new python packages, or upgrade airflow providers to a later version. This can be done very easily
+by placing your custom Dockerfile. Then you can use `docker-compose build` command to build your image (you need to
+do it only once) and you can also add `--build` to your `docker-compose` commands with flag to rebuild the images
Review comment:
```suggestion
do it only once). You can also add the `--build` flag to your `docker-compose` commands to rebuild the images
```
##########
File path: docs/apache-airflow/start/docker.rst
##########
@@ -81,6 +81,18 @@ If you need install a new Python library or system library, you can :doc:`build
.. _initializing_docker_compose_environment:
+Using custom images
+===================
+
+When you want to run Airflow locally, you might want to use extended image, containing some additional dependencies - for
+example you might add new python packages, or upgrade airflow providers to a later version. This can be done very easily
+by placing your custom Dockerfile. Then you can use `docker-compose build` command to build your image (you need to
Review comment:
```suggestion
by placing a custom Dockerfile alongside your `docker-compose.yaml`. Then you can use `docker-compose build` command to build your image (you need to
```
##########
File path: docs/docker-stack/build.rst
##########
@@ -250,6 +250,19 @@ You should be aware, about a few things:
Examples of image extending
---------------------------
+Example of upgrading Airflow Provider packages
+..............................................
+
+The :ref:`Airflow Providers <providers:community-maintained-providers>` are released independently of the main
+Airflow and sometimes you might want to upgrade specific providers only to fix some problems or
+use features available in that provider. Here is an example of how you can do it
Review comment:
```suggestion
use features available in that provider version. Here is an example of how you can do it
```
Maybe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #17409: Improve image building documentation for new users
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #17409:
URL: https://github.com/apache/airflow/pull/17409#discussion_r682855059
##########
File path: docs/apache-airflow/start/docker.rst
##########
@@ -81,6 +81,18 @@ If you need install a new Python library or system library, you can :doc:`build
.. _initializing_docker_compose_environment:
+Using custom images
+===================
+
+When you want to run Airflow locally, you might want to use extended image, containing some additional dependencies - for
+example you might add new python packages, or upgrade airflow providers to a later version. This can be done very easily
+by placing your custom Dockerfile. Then you can use `docker-compose build` command to build your image (you need to
+do it only once) and you can also add `--build` to your `docker-compose` commands with flag to rebuild the images
+on-the-fly when you run other `docker-compose` commands.
+
+The examples of how you can extend the image with custom providers, python packages,
Review comment:
```suggestion
Examples of how you can extend the image with custom providers, python packages,
```
##########
File path: docs/docker-stack/build.rst
##########
@@ -250,6 +250,19 @@ You should be aware, about a few things:
Examples of image extending
---------------------------
+Example of upgrading Airflow Provider packages
+..............................................
+
+The :ref:`Airflow Providers <providers:community-maintained-providers>` are released independently of the main
Review comment:
```suggestion
The :ref:`Airflow Providers <providers:community-maintained-providers>` are released independently of core
```
##########
File path: docs/apache-airflow/start/docker.rst
##########
@@ -81,6 +81,18 @@ If you need install a new Python library or system library, you can :doc:`build
.. _initializing_docker_compose_environment:
+Using custom images
+===================
+
+When you want to run Airflow locally, you might want to use extended image, containing some additional dependencies - for
Review comment:
```suggestion
When you want to run Airflow locally, you might want to use an extended image, containing some additional dependencies - for
```
##########
File path: docs/apache-airflow/start/docker.rst
##########
@@ -81,6 +81,18 @@ If you need install a new Python library or system library, you can :doc:`build
.. _initializing_docker_compose_environment:
+Using custom images
+===================
+
+When you want to run Airflow locally, you might want to use extended image, containing some additional dependencies - for
+example you might add new python packages, or upgrade airflow providers to a later version. This can be done very easily
+by placing your custom Dockerfile. Then you can use `docker-compose build` command to build your image (you need to
+do it only once) and you can also add `--build` to your `docker-compose` commands with flag to rebuild the images
Review comment:
```suggestion
do it only once). You can also add the `--build` flag to your `docker-compose` commands to rebuild the images
```
##########
File path: docs/apache-airflow/start/docker.rst
##########
@@ -81,6 +81,18 @@ If you need install a new Python library or system library, you can :doc:`build
.. _initializing_docker_compose_environment:
+Using custom images
+===================
+
+When you want to run Airflow locally, you might want to use extended image, containing some additional dependencies - for
+example you might add new python packages, or upgrade airflow providers to a later version. This can be done very easily
+by placing your custom Dockerfile. Then you can use `docker-compose build` command to build your image (you need to
Review comment:
```suggestion
by placing a custom Dockerfile alongside your `docker-compose.yaml`. Then you can use `docker-compose build` command to build your image (you need to
```
##########
File path: docs/docker-stack/build.rst
##########
@@ -250,6 +250,19 @@ You should be aware, about a few things:
Examples of image extending
---------------------------
+Example of upgrading Airflow Provider packages
+..............................................
+
+The :ref:`Airflow Providers <providers:community-maintained-providers>` are released independently of the main
+Airflow and sometimes you might want to upgrade specific providers only to fix some problems or
+use features available in that provider. Here is an example of how you can do it
Review comment:
```suggestion
use features available in that provider version. Here is an example of how you can do it
```
Maybe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #17409: Improve image building documentation for new users
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #17409:
URL: https://github.com/apache/airflow/pull/17409
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #17409: Improve image building documentation for new users
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #17409:
URL: https://github.com/apache/airflow/pull/17409
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17409: Improve image building documentation for new users
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17409:
URL: https://github.com/apache/airflow/pull/17409#issuecomment-892942516
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org