You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/29 16:06:37 UTC
[GitHub] [airflow] turbaszek opened a new pull request #12706: Add .airflowignore to example dags
turbaszek opened a new pull request #12706:
URL: https://github.com/apache/airflow/pull/12706
Kubernetes is not required for running Airflow. If users will have
load_examples=True then they will see warning as described in #12705.
This is the simplest possible solution I can think of. The examples should render in docs (if used) and are not listed in DAGs:
```
DAG ID Filepath Owner
example_bash_operator /opt/airflow/airflow/example_dags/example_bash_operator.py airflow
example_branch_dop_operator_v3 /opt/airflow/airflow/example_dags/example_branch_python_dop_operator_3.py airflow
example_branch_operator /opt/airflow/airflow/example_dags/example_branch_operator.py airflow
example_complex /opt/airflow/airflow/example_dags/example_complex.py airflow
example_dag_decorator /opt/airflow/airflow/example_dags/example_dag_decorator.py airflow
example_external_task_marker_child /opt/airflow/airflow/example_dags/example_external_task_marker_dag.py airflow
example_external_task_marker_parent /opt/airflow/airflow/example_dags/example_external_task_marker_dag.py airflow
example_nested_branch_dag /opt/airflow/airflow/example_dags/example_nested_branch_dag.py airflow
example_passing_params_via_test_command /opt/airflow/airflow/example_dags/example_passing_params_via_test_command.py airflow
example_python_operator /opt/airflow/airflow/example_dags/example_python_operator.py airflow
example_short_circuit_operator /opt/airflow/airflow/example_dags/example_short_circuit_operator.py airflow
example_skip_dag /opt/airflow/airflow/example_dags/example_skip_dag.py airflow
example_subdag_operator /opt/airflow/airflow/example_dags/example_subdag_operator.py airflow
example_subdag_operator.section-1 /opt/airflow/airflow/example_dags/example_subdag_operator.py airflow
example_subdag_operator.section-2 /opt/airflow/airflow/example_dags/example_subdag_operator.py airflow
example_task_group /opt/airflow/airflow/example_dags/example_task_group.py airflow
example_trigger_controller_dag /opt/airflow/airflow/example_dags/example_trigger_controller_dag.py airflow
example_trigger_target_dag /opt/airflow/airflow/example_dags/example_trigger_target_dag.py airflow
example_xcom /opt/airflow/airflow/example_dags/example_xcom.py airflow
example_xcom_args /opt/airflow/airflow/example_dags/example_xcomargs.py airflow
example_xcom_args_with_operators /opt/airflow/airflow/example_dags/example_xcomargs.py airflow
latest_only /opt/airflow/airflow/example_dags/example_latest_only.py airflow
latest_only_with_trigger /opt/airflow/airflow/example_dags/example_latest_only_with_trigger.py airflow
test_utils /opt/airflow/airflow/example_dags/test_utils.py airflow
tutorial /opt/airflow/airflow/example_dags/tutorial.py airflow
tutorial_etl_dag /opt/airflow/airflow/example_dags/tutorial_etl_dag.py airflow
tutorial_taskflow_api_etl_dag /opt/airflow/airflow/example_dags/tutorial_taskflow_api_etl.py airflow
```
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12706: Add .airflowignore to example dags
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12706:
URL: https://github.com/apache/airflow/pull/12706#issuecomment-735453844
Oh lol. Too many PRs, can't even remember that I already approved one to fix it :grin:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12706: Add .airflowignore to example dags
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12706:
URL: https://github.com/apache/airflow/pull/12706#issuecomment-735448860
How about just putting the "entireity" of the dag file in a `try`/`except ImportError` -- that way if they have installed the kube extra they do actually get the example dags.
Cos with it in the airflow ignore, we may as well not have those as example dags -- they would never be runnable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12706: Add .airflowignore to example dags
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12706:
URL: https://github.com/apache/airflow/pull/12706#issuecomment-735451875
Other option is to do some special casing when loading examples, and ignore these two files when kube modules can't be loaded?
Or more perhaps more generaically, when loading _any_ example dag, if we get an ImportError ignore that file and move on? That might be a bit overkill
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12706: Add .airflowignore to example dags
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12706:
URL: https://github.com/apache/airflow/pull/12706#issuecomment-735420883
[The Workflow run](https://github.com/apache/airflow/actions/runs/390124993) is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek edited a comment on pull request #12706: Add .airflowignore to example dags
Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on pull request #12706:
URL: https://github.com/apache/airflow/pull/12706#issuecomment-735453294
Oh, there's already try/except 😄 #12494
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #12706: Add .airflowignore to example dags
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #12706:
URL: https://github.com/apache/airflow/pull/12706#issuecomment-735452987
Ok, let's do try/except
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek closed pull request #12706: Add .airflowignore to example dags
Posted by GitBox <gi...@apache.org>.
turbaszek closed pull request #12706:
URL: https://github.com/apache/airflow/pull/12706
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek edited a comment on pull request #12706: Add .airflowignore to example dags
Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on pull request #12706:
URL: https://github.com/apache/airflow/pull/12706#issuecomment-735452240
> So the one that is in the docs should maybe be moved somewhere else (outside of airflow/example_dags/`).
In fact that's what we should do in general - put "non core" DAGs somewhere else. But then they will not show in examples. In this PR I wanted to do a "quick fix" to avoid surprises like in #12705.
General problem was reported in https://github.com/apache/airflow/issues/11435
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12706: Add .airflowignore to example dags
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12706:
URL: https://github.com/apache/airflow/pull/12706#issuecomment-735452462
Kube _Executor_ example is sort of core. It's one of those awkward grey areas.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #12706: Add .airflowignore to example dags
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #12706:
URL: https://github.com/apache/airflow/pull/12706#issuecomment-735453929
Yup, already fixed 🚀
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #12706: Add .airflowignore to example dags
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #12706:
URL: https://github.com/apache/airflow/pull/12706#issuecomment-735453294
Oh, there's already try/except 😄
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12706: Add .airflowignore to example dags
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12706:
URL: https://github.com/apache/airflow/pull/12706#issuecomment-735451700
Yeah, try-except is ugly I'll grant you. Putting them in an airflowignore feels like a hack.
So the one that is in the docs should maybe be moved somewhere else (outside of airflow/example_dags/`).
What's the other one?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #12706: Add .airflowignore to example dags
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #12706:
URL: https://github.com/apache/airflow/pull/12706#issuecomment-735452240
> So the one that is in the docs should maybe be moved somewhere else (outside of airflow/example_dags/`).
In fact that's we should do in general - put "non core" DAGs somewhere else. But then they will not show in examples. In this PR I wanted to do a "quick fix" to avoid surprises like in #12705.
General problem was reported in https://github.com/apache/airflow/issues/11435
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #12706: Add .airflowignore to example dags
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #12706:
URL: https://github.com/apache/airflow/pull/12706#issuecomment-735451482
> How about just putting the "entireity" of the dag file in a `try`/`except ImportError` -- that way if they have installed the kube extra they do actually get the example dags.
>
> Cos with it in the airflow ignore, we may as well not have those as example dags -- they would never be runnable.
The try/except was the first idea I had but... well you know it's whole file in try/except. To be honest I'm not sure how many people will run these two examples. One of them is described in docs so people will learn how-to.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org