You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/30 19:55:36 UTC
[GitHub] [airflow] jceresini opened a new issue #19903: Error setting dependencies on task_group defined using the decorator
jceresini opened a new issue #19903:
URL: https://github.com/apache/airflow/issues/19903
### Apache Airflow version
2.2.2 (latest released)
### Operating System
MacOS 11.6.1
### Versions of Apache Airflow Providers
$ pip freeze | grep airflow
apache-airflow==2.2.2
apache-airflow-providers-celery==2.1.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-sqlite==2.0.1
### Deployment
Other
### Deployment details
`airflow standalone`
### What happened
```
AttributeError: 'NoneType' object has no attribute 'update_relative'
```
### What you expected to happen
Task group should be set as downstream of `start` task, and upstream of `end` task
### How to reproduce
* Add the following code to dags folder
```python
from datetime import datetime
from airflow.decorators import dag, task, task_group
@dag(start_date=datetime(2023, 1, 1), schedule_interval="@once")
def test_dag_1():
@task
def start():
pass
@task
def do_thing(x):
print(x)
@task_group
def do_all_things():
do_thing(1)
do_thing(2)
@task
def end():
pass
start() >> do_all_things() >> end()
test_dag_1()
```
* Run `airflow standalone`
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on issue #19903: Error setting dependencies on task_group defined using the decorator
Posted by GitBox <gi...@apache.org>.
subkanthi commented on issue #19903:
URL: https://github.com/apache/airflow/issues/19903#issuecomment-1004105943
Im able to reproduce this in `standalone` mode even without the taskgroup ` start() >> end()`
```
@dag(start_date=datetime(2023, 1, 1), schedule_interval="@once")
def test_dag_1():
@task
def start():
pass
@task
def do_thing(x):
print(x)
@task_group
def do_all_things():
do_thing(1)
do_thing(2)
@task
def end():
pass
start() >> do_all_things() >> end()
test_dag_1()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] moshederri removed a comment on issue #19903: Error setting dependencies on task_group defined using the decorator
Posted by GitBox <gi...@apache.org>.
moshederri removed a comment on issue #19903:
URL: https://github.com/apache/airflow/issues/19903#issuecomment-1003771492
Quick question about this - i tried changing the content of the function to something other than `pass` or `print` but for some reason i am still getting the same exception (i'm probably doing something wrong). But regardless - working with the dag decorator, isn't the rshift misused here?
i mean, shouldn't:
`start() >> do_all_things() >> end() `
just be:
```
start()
do_all_things()
end()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on issue #19903: Error setting dependencies on task_group defined using the decorator
Posted by GitBox <gi...@apache.org>.
subkanthi commented on issue #19903:
URL: https://github.com/apache/airflow/issues/19903#issuecomment-1002780046
Looks like the problem is here for both the tasks, with pass or print, the task runs are not created, but replacing it with sleep, a task run is created and added to the DB.
` @task
def end():
pass`
`@task
def end():
pass`
It might be useful to indicate this to the user, still trying to figure out where this logic is performed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] moshederri commented on issue #19903: Error setting dependencies on task_group defined using the decorator
Posted by GitBox <gi...@apache.org>.
moshederri commented on issue #19903:
URL: https://github.com/apache/airflow/issues/19903#issuecomment-1003771492
Quick question about this - i tried changing the content of the function to something other than `pass` or `print` but for some reason i am still getting the same exception (i'm probably doing something wrong). But regardless - working with the dag decorator, isn't the rshift misused here?
i mean, shouldn't:
`start() >> do_all_things() >> end() `
be just:
```
start()
do_all_things()
end()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on issue #19903: Error setting dependencies on task_group defined using the decorator
Posted by GitBox <gi...@apache.org>.
subkanthi commented on issue #19903:
URL: https://github.com/apache/airflow/issues/19903#issuecomment-1002342857
Looking at this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #19903: Error setting dependencies on task_group defined using the decorator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #19903:
URL: https://github.com/apache/airflow/issues/19903#issuecomment-1005849107
Actually there is a way to make this work under the current `@task_group` functionality; you need to do this:
```python
@task_group
def do_all_things():
return [do_thing(1), do_thing(2)]
```
But this is totally not obvious and probably does not look nice, so #20671 is still useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #19903: Error setting dependencies on task_group defined using the decorator
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #19903:
URL: https://github.com/apache/airflow/issues/19903
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi edited a comment on issue #19903: Error setting dependencies on task_group defined using the decorator
Posted by GitBox <gi...@apache.org>.
subkanthi edited a comment on issue #19903:
URL: https://github.com/apache/airflow/issues/19903#issuecomment-1002780046
Looks like the problem is here for both the tasks, with pass or print, the task instances are not created, but replacing it with sleep, a task instance is created and added to the DB.
` @task
def end():
pass`
`@task
def end():
pass`
It might be useful to indicate this to the user, still trying to figure out where this logic is performed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] moshederri edited a comment on issue #19903: Error setting dependencies on task_group defined using the decorator
Posted by GitBox <gi...@apache.org>.
moshederri edited a comment on issue #19903:
URL: https://github.com/apache/airflow/issues/19903#issuecomment-1003771492
Quick question about this - i tried changing the content of the function to something other than `pass` or `print` but for some reason i am still getting the same exception (i'm probably doing something wrong). But regardless - working with the dag decorator, isn't the rshift misused here?
i mean, shouldn't:
`start() >> do_all_things() >> end() `
just be:
```
start()
do_all_things()
end()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #19903: Error setting dependencies on task_group defined using the decorator
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #19903:
URL: https://github.com/apache/airflow/issues/19903
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #19903: Error setting dependencies on task_group defined using the decorator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #19903:
URL: https://github.com/apache/airflow/issues/19903#issuecomment-1005535302
Oh I see what’s going on. This has nothing to do with the task function, but the task _group_ function. Currently, the `@task_group` decorator is intended to be used like this:
https://github.com/apache/airflow/blob/e8b5ab9efe93f826a2b521b5d1cac0404354c3b4/airflow/example_dags/example_task_group_decorator.py#L60-L63
Note how the task group function returns `task_3(...)`, which produces a BaseOperator.
In your example, however, the task group function does not return anything, i.e. implicitly returns None. This means `start() >> do_all_things()` becomes `DummyOperator(...) >> None`, which fails with the exception.
However, I don’t think the existing function-chaining syntax suits the need here, since `do_thing(1)` and `do_thing(2)` are supposed to be independent (and both depend on `start()` and depended by `end()`). So we need to extend the `@task_group` decorator a bit to make this work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr edited a comment on issue #19903: Error setting dependencies on task_group defined using the decorator
Posted by GitBox <gi...@apache.org>.
uranusjr edited a comment on issue #19903:
URL: https://github.com/apache/airflow/issues/19903#issuecomment-1003874514
I don’t recall any logic checking a function’s content, so this is very surprising. I can’t reproduce though; what’s a minimal example of this happening?
This seems to work correctly for me:
```python
import datetime
from airflow.decorators import dag, task
@dag(start_date=datetime.datetime(2022, 1, 1))
def empty_task_dag():
@task
def empty_task():
pass
empty_task()
dag = empty_task_dag()
```
A task instance of DAG ID `empty_task_dag` and task ID `empty_task` is shown on the UI when I manually trigger a DAG run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #19903: Error setting dependencies on task_group defined using the decorator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #19903:
URL: https://github.com/apache/airflow/issues/19903#issuecomment-1003874514
I don’t recall any logic checking a function’s content, so this is very surprising. I can reproduce though; it seems like if the function does not have a body, even the _task_ itself is not created in the DAG.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi edited a comment on issue #19903: Error setting dependencies on task_group defined using the decorator
Posted by GitBox <gi...@apache.org>.
subkanthi edited a comment on issue #19903:
URL: https://github.com/apache/airflow/issues/19903#issuecomment-1002342857
Looking at this issue.
The problem is here, for some when the right shift is called for start, the other value is None, its almost like the task group is not created properly.
```
def __rshift__(self, other: Union["DependencyMixin", Sequence["DependencyMixin"]]):
"""Implements Task >> Task"""
self.set_downstream(other)
return other
```
```
if not isinstance(task_or_task_list, Sequence):
task_or_task_list = [task_or_task_list]
task_list: List["BaseOperator"] = []
for task_object in task_or_task_list:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org