You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/21 16:06:04 UTC
[GitHub] [airflow] baryluk opened a new issue #17140: dag.cli() issues in 1.10.15
baryluk opened a new issue #17140:
URL: https://github.com/apache/airflow/issues/17140
**Apache Airflow version**:
1.10.15
**Environment**:
- **OS** (e.g. from /etc/os-release): Linux, amd64, Fedora 34
- **Install tools**: pyenv virtualenv, pip (with constraints).
**What happened**:
I made a testing script in preparation for migration to Airflow 2.x, from 1.10.12. After addressing some early discovered issues, I managed to update to 1.10.15.
The testing script uses pattern like this `python3 - tasks list < ./dags/primary/examples/tutorial.py` (with `AIRFLOW_HOME` and `PYTHONPATH` setup so the imports from `common/` works properly. I do not pass the script directly to python, because that messes up `sys.path` by adding base directory of the script too, which is not what I want).
This script works with 2.1.2, but despite 1.10.12 having some CLI commands backported, it does not work in 1.10.12.
Example:
```python3
$ cat ./dags/primary/examples/tutorial.py
from datetime import timedelta
import airflow
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
# These args will get passed on to each operator
# You can override them on a per-task basis during operator initialization
default_args = {
"owner": "Airflow",
"depends_on_past": False,
"start_date": airflow.utils.dates.days_ago(2),
"email": ["airflow@example.com"],
"email_on_failure": False,
"email_on_retry": False,
"retries": 1,
"retry_delay": timedelta(minutes=5),
}
dag = DAG(
"tutorial",
default_args=default_args,
description="A simple tutorial DAG",
schedule_interval=timedelta(days=1),
dagrun_timeout=timedelta(seconds=20),
)
# t1, t2 and t3 are examples of tasks created by instantiating operators
t1 = BashOperator(
task_id="print_date",
bash_command="date",
dag=dag,
)
t2 = BashOperator(
task_id="sleep",
depends_on_past=False,
bash_command="sleep 10",
dag=dag,
)
t3 = BashOperator(
task_id="echo1",
depends_on_past=False,
bash_command="echo 1",
params={"my_param": "Parameter I passed in"},
dag=dag,
)
t1 >> [t2, t3]
if __name__ == "__main__":
dag.cli()
$
```
Airflow 1.10.15
`tasks list`:
```
(airflow1.10.15) $ export AIRFLOW_HOME="${HOME}/airflow-testing/airflow"
(airflow1.10.15) $ PYTHONPATH="$(pwd)" python3 - tasks list < ./dags/primary/examples/tutorial.py
Traceback (most recent call last):
File "<stdin>", line 119, in <module>
File "/home/user/airflow-testing/venv/lib/python3.9/site-packages/airflow/models/dag.py", line 1439, in cli
parser = cli.CLIFactory.get_parser(dag_parser=True)
File "/home/user/airflow-testing/venv/lib/python3.9/site-packages/airflow/bin/cli.py", line 4216, in get_parser
action = _add_command(subparsers, ALL_COMMANDS_DICT[sub_name])
KeyError: 'backfill'
(airflow1.10.15) ~/code/airflow7$
```
Same when calling `tasks`, or no arguments::
```
(airflow1.10.15) $ PYTHONPATH="$(pwd)" python3 - < ./dags/primary/examples/tutorial.py
Traceback (most recent call last):
File "<stdin>", line 119, in <module>
File "/home/witek/airflow-testing/venv/lib/python3.9/site-packages/airflow/models/dag.py", line 1439, in cli
parser = cli.CLIFactory.get_parser(dag_parser=True)
File "/home/witek/airflow-testing/venv/lib/python3.9/site-packages/airflow/bin/cli.py", line 4216, in get_parser
action = _add_command(subparsers, ALL_COMMANDS_DICT[sub_name])
KeyError: 'backfill'
(airflow1.10.15) $
```
```
(airflow1.10.15) $ python3
Python 3.9.6 (default, Jul 16 2021, 00:00:00)
[GCC 11.1.1 20210531 (Red Hat 11.1.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import airflow
>>> airflow.version.version
'1.10.15'
```
(same with python 3.7.9)
(note the `airflow dags list` itself for example works, and retrives dags from the configured database). The issue is only with `dag.cli()` path).
**What you expected to happen**:
Should work same as in airflow 2.x (here using python 3.7.9):
```
(airflow2.1.2) $ PYTHONPATH=$(pwd) python3 - < dags/primary/examples/tutorial.py
<stdin>:29 DeprecationWarning: This module is deprecated. Please use `airflow.operators.bash`.
usage: airflow [-h] GROUP_OR_COMMAND ...
positional arguments:
GROUP_OR_COMMAND
Groups:
dags Manage DAGs
tasks Manage tasks
Commands:
optional arguments:
-h, --help show this help message and exit
airflow command error: the following arguments are required: GROUP_OR_COMMAND, see help above.
```
And other commands:
```
(airflow2.1.2) $ PYTHONPATH=$(pwd) python3 - tasks list < dags/primary/examples/tutorial.py
<stdin>:29 DeprecationWarning: This module is deprecated. Please use `airflow.operators.bash`.
print_date
sleep
templated
(airflow2.1.2) $
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #17140: dag.cli() issues in 1.10.15
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #17140:
URL: https://github.com/apache/airflow/issues/17140
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17140: dag.cli() issues in 1.10.15
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17140:
URL: https://github.com/apache/airflow/issues/17140#issuecomment-884342913
We are not going to release any more changes to 1.10.* line - it has reached end of life. Some of the commands might indeed work differently (there are couple of others) but I am afraid this is the small remaining pain in the migration that you have to handle, the overhead of fixing and releasing 1.10.15 is far too big to be justified here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org