You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dstandish (via GitHub)" <gi...@apache.org> on 2023/07/25 20:00:00 UTC
[GitHub] [airflow] dstandish opened a new issue, #32836: dag.test() may not work with mapped tasks
dstandish opened a new issue, #32836:
URL: https://github.com/apache/airflow/issues/32836
### Body
repro
```python
with DAG(dag_id="example_setup_one_to_many") as dag:
@setup
def my_setup():
print("setting up multiple things")
return [1, 2, 3]
@task
def my_work(val):
print(f"doing work with multiple things: {val}")
raise ValueError("fail!")
return val
@teardown
def my_teardown(val):
print(f"teardown: {val}")
s = my_setup()
t = my_teardown.expand(val=s)
with t:
w = my_work(s)
dag.test()
```
can verify more easily if you change `test` to return the dag run obj. then call `dr.get_task_instances` and you will see you only get 3 instances -- the but there sholud be more by mapping.
@uranusjr
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #32836: dag.test() does not propagate upstream_failed state correctly
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on issue #32836:
URL: https://github.com/apache/airflow/issues/32836#issuecomment-1650546357
updated issue with simpler example and more accurate info
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #32836: dag.test() does not work with mapped tasks
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on issue #32836:
URL: https://github.com/apache/airflow/issues/32836#issuecomment-1650205134
hmm... i tested with
```python
tis = {x.task_id: x for x in dr.get_task_instances()}
```
and only got 3 tis ... but the problem may instead be with this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish closed issue #32836: dag.test() does not propagate upstream_failed state correctly
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish closed issue #32836: dag.test() does not propagate upstream_failed state correctly
URL: https://github.com/apache/airflow/issues/32836
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #32836: dag.test() may not work with mapped tasks
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on issue #32836:
URL: https://github.com/apache/airflow/issues/32836#issuecomment-1650465117
ok actually @uranusjr i was able to confirm that many -> one mapping when there's a failure in mapped tasks, upstream failed does not propagate correctly beyond the first downstream relative when using dag.test() or info.scheduling_decisions approach
here is example test for repro
```python
def get_states(dr):
ti_dict = defaultdict(dict)
for ti in dr.get_task_instances():
if ti.map_index == -1:
ti_dict[ti.task_id] = ti.state
else:
ti_dict[ti.task_id][ti.map_index] = ti.state
return ti_dict
def test_many_one(dag_maker, session):
with dag_maker(dag_id="many_one") as dag:
@task
def my_setup(val):
print("setup")
raise ValueError("hi")
@task
def my_work():
print("work")
@task
def my_teardown():
print("teardown")
s = my_setup.expand(val=[1, 2])
t = my_teardown()
s >> my_work() >> t
dr = dag_maker.create_dagrun()
while True:
info = dr.task_instance_scheduling_decisions()
if not info.schedulable_tis:
break
for ti in info.schedulable_tis:
try:
ti.run()
except Exception as e:
print(e)
session.commit()
info.unfinished_tis
# dr = dag.test()
states = get_states(dr)
expected = {
"my_setup": {0: "failed", 1: "failed"},
"my_teardown": None,
"my_work": "upstream_failed",
}
assert states == expected
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #32836: dag.test() may not work with mapped tasks
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on issue #32836:
URL: https://github.com/apache/airflow/issues/32836#issuecomment-1650231696
false alarm, closing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish closed issue #32836: dag.test() may not work with mapped tasks
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish closed issue #32836: dag.test() may not work with mapped tasks
URL: https://github.com/apache/airflow/issues/32836
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org