You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/13 15:42:16 UTC

[GitHub] [airflow] mik-laj opened a new pull request #13655: Adds recipes for installing a few common tools in Docker image

mik-laj opened a new pull request #13655:
URL: https://github.com/apache/airflow/pull/13655

hello,

Some developers ask how to install additional tools in the image. I have prepared instructions for a few common tools.

Best regards,
Kamil Breguła

<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.

Feel free to ping committers for the review!

In case of existing issue, reference it using one of the following:

closes: #ISSUE
related: #ISSUE

How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->

---
**^ Add meaningful description above**

Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj merged pull request #13655: Add recipes for installing a few common tools in Docker image

Posted by GitBox <gi...@apache.org>.

mik-laj merged pull request #13655:
URL: https://github.com/apache/airflow/pull/13655


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #13655: Adds recipes for installing a few common tools in Docker image

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #13655:
URL: https://github.com/apache/airflow/pull/13655#issuecomment-759618848


   > @potiuk I agree with Kamil, I think adding it now wouldn't help our CI :D Let's really wait for the CI to be more stable.
   
   Agree. Hopefully soon! :crossed_fingers: 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #13655: Adds recipes for installing a few common tools in Docker image

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on pull request #13655:
URL: https://github.com/apache/airflow/pull/13655#issuecomment-759665196


   The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest master or amend the last commit of the PR, and push it with --force-with-lease.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] feluelle commented on pull request #13655: Adds recipes for installing a few common tools in Docker image

Posted by GitBox <gi...@apache.org>.

feluelle commented on pull request #13655:
URL: https://github.com/apache/airflow/pull/13655#issuecomment-759610471


   @mik-laj SGTM 👍 
   
   @potiuk I agree with Kamil, I think adding it now wouldn't help our CI :D Let's really wait for the CI to be more stable.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj edited a comment on pull request #13655: Adds recipes for installing a few common tools in Docker image

Posted by GitBox <gi...@apache.org>.

mik-laj edited a comment on pull request #13655:
URL: https://github.com/apache/airflow/pull/13655#issuecomment-759568405

@feluelle I would like it to be in one place along with the documentation for Docker image. It's not a complicated code, so I don't think there are any problems with maintaining it. These are code snippets from several images that we use in production at Polidea. These tools are required by Airflow operators, so I think, we should describe how to install these additional optional components in the image as this is not always obvious.

As for a separate repository, we agreed in the community that we would maintain a monorepo. I was wondering if this should be in the documentation for `apache-airflow`, but we don't have separate documentation for the Docker image yet. For now, this is the only place that has end-user documentation for docker image. In the future, we can create a new documentation package, but it is an extra effort. This may be a done as part of https://github.com/apache/airflow/issues/11740 , where we will have to do some work on better documentation for the image.

Other projects that have separate documentation about Docker have such sections in the documentation.
https://jupyter-docker-stacks.readthedocs.io/en/latest/using/recipes.html

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj edited a comment on pull request #13655: Adds recipes for installing a few common tools in Docker image

Posted by GitBox <gi...@apache.org>.

mik-laj edited a comment on pull request #13655:
URL: https://github.com/apache/airflow/pull/13655#issuecomment-759568405

Other projects that have separate documentation about Docker have such sections in the documentation.
https://jupyter-docker-stacks.readthedocs.io/en/latest/using/recipes.html

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #13655: Adds recipes for installing a few common tools in Docker image

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #13655:
URL: https://github.com/apache/airflow/pull/13655#issuecomment-759716675


   > Do you think would be a good idea to have also docker-compose-recipes doc?
   
   @mik-laj works on a whole eeasy-to-generate docker-compose example to become part of the repo https://github.com/apache/airflow/issues/8605#issuecomment-759469443
   
   All comments are welcome :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #13655: Adds recipes for installing a few common tools in Docker image

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on pull request #13655:
URL: https://github.com/apache/airflow/pull/13655#issuecomment-759754878


   [The Workflow run](https://github.com/apache/airflow/actions/runs/483162856) is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj edited a comment on pull request #13655: Adds recipes for installing a few common tools in Docker image

Posted by GitBox <gi...@apache.org>.

mik-laj edited a comment on pull request #13655:
URL: https://github.com/apache/airflow/pull/13655#issuecomment-759602802


   @potiuk we have a bit overloaded our CI system, so I don't think that adding more types of tests will be a good idea, especially since building an image is not too light operation and is also crucial for the viability of our project. What do you think I'd have created a separate ticket and added these tests in the future when our CI would be a little more stable? I don't think those images would get damaged during this time. Not many people will contribute to these files, and on the other hand, external services, if they have problems with stability, will not affect our CI.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #13655: Adds recipes for installing a few common tools in Docker image

Posted by GitBox <gi...@apache.org>.

mik-laj commented on pull request #13655:
URL: https://github.com/apache/airflow/pull/13655#issuecomment-759568405

@feluelle I would like it to be in one place along with the documentation for Docker image. It's not a complicated code, so I don't think there are any problems with maintaining it. These are code snippets from several images that we use in production at Polidea. These tools are often required by various Airflow operators, so we should describe how to install these additional optional components in the image as this is not always obvious.

Other projects that have separate documentation about Docker have such sections in the documentation.
https://jupyter-docker-stacks.readthedocs.io/en/latest/using/recipes.html

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #13655: Adds recipes for installing a few common tools in Docker image

Posted by GitBox <gi...@apache.org>.

mik-laj commented on pull request #13655:
URL: https://github.com/apache/airflow/pull/13655#issuecomment-759602802


   @potiuk we have a bit overloaded with our CI system, so I don't think that adding more types of tests will be a good idea, especially since building an image is not too light operation and is also crucial for the viability of our project. What do you think I'd have created a separate ticket and added these tests in the future when our CI would be a little more stable? I don't think those images would get damaged during this time. Not many people will contribute to these files, and on the other hand, external services, if they have problems with stability, will not affect our CI.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] JavierLopezT commented on pull request #13655: Adds recipes for installing a few common tools in Docker image

Posted by GitBox <gi...@apache.org>.

JavierLopezT commented on pull request #13655:
URL: https://github.com/apache/airflow/pull/13655#issuecomment-759713120


   Do you think would be a good idea to have also docker-compose-recipes doc? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj edited a comment on pull request #13655: Adds recipes for installing a few common tools in Docker image

Posted by GitBox <gi...@apache.org>.

mik-laj edited a comment on pull request #13655:
URL: https://github.com/apache/airflow/pull/13655#issuecomment-759602802


   @potiuk we have a bit overloaded our CI system, so I don't think that adding more types of tests will be a good idea, especially since building an image is not too light operation and is also crucial for the viability of our project. What do you think I'd have created a separate ticket and added these tests in the future when our CI would be a little more stable? I don't think those images would get damaged during this time. Not many people will contribute to these files, and on the other hand, external services, if they have problems with stability, will not affect our CI.
   
   If we had tests that are run much less frequently, e.g. once a week, and not with any changes in the project, I would be glad to add these tests.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #13655: Adds recipes for installing a few common tools in Docker image

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #13655:
URL: https://github.com/apache/airflow/pull/13655#issuecomment-759596643


   > Please let me know what you think.
   
   I think it's  good idea to add it. This is mostly documentation. however I'd add a test to build those images in CI. This way we will be sure it is up-to date. @mik-laj WDYT?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org