You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dstandish (via GitHub)" <gi...@apache.org> on 2023/07/25 20:00:00 UTC

[GitHub] [airflow] dstandish opened a new issue, #32836: dag.test() may not work with mapped tasks

dstandish opened a new issue, #32836:
URL: https://github.com/apache/airflow/issues/32836

   ### Body
   
   repro
   ```python
       with DAG(dag_id="example_setup_one_to_many") as dag:
           @setup
           def my_setup():
               print("setting up multiple things")
               return [1, 2, 3]
   
           @task
           def my_work(val):
               print(f"doing work with multiple things: {val}")
               raise ValueError("fail!")
               return val
   
           @teardown
           def my_teardown(val):
               print(f"teardown: {val}")
   
           s = my_setup()
           t = my_teardown.expand(val=s)
           with t:
               w = my_work(s)
       dag.test()
   ```
   
   can verify more easily if you change `test` to return the dag run obj.  then call `dr.get_task_instances` and you will see you only get 3 instances -- the but there sholud be more by mapping.
   
   @uranusjr 
   
   ### Committer
   
   - [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on issue #32836: dag.test() does not propagate upstream_failed state correctly

Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on issue #32836:
URL: https://github.com/apache/airflow/issues/32836#issuecomment-1650546357

   updated issue with simpler example and more accurate info


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on issue #32836: dag.test() does not work with mapped tasks

Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on issue #32836:
URL: https://github.com/apache/airflow/issues/32836#issuecomment-1650205134

   hmm... i tested with
   ```python
   tis = {x.task_id: x for x in dr.get_task_instances()}
   ```
   and only got 3 tis ... but the problem may instead be with this method 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish closed issue #32836: dag.test() does not propagate upstream_failed state correctly

Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish closed issue #32836: dag.test() does not propagate upstream_failed state correctly
URL: https://github.com/apache/airflow/issues/32836


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on issue #32836: dag.test() may not work with mapped tasks

Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on issue #32836:
URL: https://github.com/apache/airflow/issues/32836#issuecomment-1650465117

   ok actually @uranusjr i was able to confirm that many -> one mapping when there's a failure in mapped tasks, upstream failed does not propagate correctly beyond the first downstream relative when using dag.test() or info.scheduling_decisions approach
   
   here is example test for repro
   ```python
   def get_states(dr):
       ti_dict = defaultdict(dict)
       for ti in dr.get_task_instances():
           if ti.map_index == -1:
               ti_dict[ti.task_id] = ti.state
           else:
               ti_dict[ti.task_id][ti.map_index] = ti.state
       return ti_dict
   
   def test_many_one(dag_maker, session):
       with dag_maker(dag_id="many_one") as dag:
   
           @task
           def my_setup(val):
               print("setup")
               raise ValueError("hi")
   
           @task
           def my_work():
               print("work")
   
           @task
           def my_teardown():
               print("teardown")
   
           s = my_setup.expand(val=[1, 2])
           t = my_teardown()
           s >> my_work() >> t
       dr = dag_maker.create_dagrun()
       while True:
           info = dr.task_instance_scheduling_decisions()
           if not info.schedulable_tis:
               break
           for ti in info.schedulable_tis:
               try:
                   ti.run()
               except Exception as e:
                   print(e)
       session.commit()
       info.unfinished_tis
       # dr = dag.test()
       states = get_states(dr)
       expected = {
           "my_setup": {0: "failed", 1: "failed"},
           "my_teardown": None,
           "my_work": "upstream_failed",
       }
       assert states == expected
   
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on issue #32836: dag.test() may not work with mapped tasks

Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on issue #32836:
URL: https://github.com/apache/airflow/issues/32836#issuecomment-1650231696

   false alarm, closing


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish closed issue #32836: dag.test() may not work with mapped tasks

Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish closed issue #32836: dag.test() may not work with mapped tasks
URL: https://github.com/apache/airflow/issues/32836


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org