You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "MarcusCramer91 (via GitHub)" <gi...@apache.org> on 2025/08/21 12:24:32 UTC

[I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

MarcusCramer91 opened a new issue, #54779:
URL: https://github.com/apache/airflow/issues/54779

   ### Apache Airflow version
   
   3.0.5
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   On v3.0.4. 
   
   Cannot restart a single mapped task. It always restarts all sibling mapped tasks as well. I found this old issue https://github.com/apache/airflow/issues/24699 so I thought it should be working, but doesn't for me. We have multiple issues here:
   
   1. When clearing the individual task it shows all siblings in the "Downstream" selection.
   2. It re-runs all sibling tasks. Cannot tell for certain this isn't just a UI bug in this simple example. But in my real-world use case, where Airflow triggers Google Cloud Run Jobs, it does spawn n Cloud Run Jobs upon restart.
   3. It also re-runs all downstream tasks for all map indices.
   4. For previous task executions, upon UI reload, I only get "::group::Log message source details: sources=["Could not read served logs: 'TaskInstanceHistory' object has no attribute 'triggerer_job'"]" and not the actual logs.
   
   
   Or am I misunderstanding this feature?
   
   ### What you think should happen instead?
   
   Only the selected mapped task should be restarted. 
   
   The logs should properly show up in the UI for previous task tries. 
   
   Only downstream tasks with the same map index should be re-run (not sure this is easily possible or intended, but without it I don't see how restarting individual tasks would be of much use). 
   
   ### How to reproduce
   
   ```
   @task
   def task1(val: str):
       print("Task 2" + val)
   
   @task
   def task2(val: str):
       print("Task 2" + val)
   
   @dag(dag_id="test")
   def get_dag() -> None:
       t1 = task1.expand(val=["foo", "bar"])
       t2 = task2.expand(val=["foo", "bar"])
       t1 >> t2
   
   get_dag()
   ```
   
   Run once. Then re-run an individual task of task1.
   
   ### Operating System
   
   Debian GNU/Linux 12 (bookworm)
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

Posted by "eric-workweek (via GitHub)" <gi...@apache.org>.
eric-workweek commented on issue #54779:
URL: https://github.com/apache/airflow/issues/54779#issuecomment-3471100203

   <img width="884" height="619" alt="Image" src="https://github.com/user-attachments/assets/c11ff5ad-d368-4baf-b1c7-321d5c35131a" />
   
   It seems like this is also a problem in version 2.11.0 (`Airflow 2.11.0+astro.3` specifically)?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

Posted by "matthieuauger (via GitHub)" <gi...@apache.org>.
matthieuauger commented on issue #54779:
URL: https://github.com/apache/airflow/issues/54779#issuecomment-3461199792

   @uranusjr Just tried on 3.1.1 but restarting create_job[0] also trigger all indexes for downstream tasks within the group.
   
   <img width="516" height="396" alt="Image" src="https://github.com/user-attachments/assets/610e6371-8229-4249-a45a-599eb595e5d1" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr closed issue #54779: Restarting a single mapped task restarts all sibling tasks as well
URL: https://github.com/apache/airflow/issues/54779


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

Posted by "MarcusCramer91 (via GitHub)" <gi...@apache.org>.
MarcusCramer91 commented on issue #54779:
URL: https://github.com/apache/airflow/issues/54779#issuecomment-3833659309

   I can confirm what @matthieuauger said. Any chance you could have a second look @uranusjr ? :) 
   
   I am not 100% certain how exactly it would work, since it somehow must be specified that a downstream mapped task refers to the same mapping indices as the retried task. But it is a very common use case to map multiple tasks to the same array as stated above.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

Posted by "MarcusCramer91 (via GitHub)" <gi...@apache.org>.
MarcusCramer91 commented on issue #54779:
URL: https://github.com/apache/airflow/issues/54779#issuecomment-3290959295

   Any updates here? :) This is really making production use for us more cumbersome right now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

Posted by "kevinhongzl (via GitHub)" <gi...@apache.org>.
kevinhongzl commented on issue #54779:
URL: https://github.com/apache/airflow/issues/54779#issuecomment-3352987552

   Hi @MarcusCramer91
   I tried restarting a single mapped task by unselecting all the options (past, future, upstream, etc.) above: 
   
   <img width="756" height="491" alt="Image" src="https://github.com/user-attachments/assets/bda5b9f2-13d9-4c2a-ba9b-a69ae0aaa2c0" />
   
   and it worked as expected. No other task instances were restarted except the one I clicked on. Would you mind checking this to see if it works for you?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on issue #54779:
URL: https://github.com/apache/airflow/issues/54779#issuecomment-3459970408

   Can any of you try 3.1.1? I believe this is a duplicate to #56289.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on issue #54779:
URL: https://github.com/apache/airflow/issues/54779#issuecomment-3467549297

   Cool, I’ll take a stab at this and hopefully fix the downstream issue in the next release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #54779:
URL: https://github.com/apache/airflow/issues/54779#issuecomment-3211954453

   I am able to reproduce this behavior. However, I'd like to address this point below:
   
   > Only downstream tasks with the same map index should be re-run (not sure this is easily possible or intended, but without it I don't see how restarting individual tasks would be of much use).
   
   In this case, if I cleared the state of `task1[0]`, I would expect both `task2[0]` and `task1[0]` to be re-run, not just `task2[0]`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

Posted by "MarcusCramer91 (via GitHub)" <gi...@apache.org>.
MarcusCramer91 commented on issue #54779:
URL: https://github.com/apache/airflow/issues/54779#issuecomment-3397522000

   @kevinhongzl Exactly what @matthieuauger said :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

Posted by "matthieuauger (via GitHub)" <gi...@apache.org>.
matthieuauger commented on issue #54779:
URL: https://github.com/apache/airflow/issues/54779#issuecomment-3662792446

   Hello,
   
   I tried this on last Airflow version but this still does not work as described here: https://github.com/apache/airflow/issues/54779#issuecomment-3353662574
   
   If you restart a mapped task, it doesn't include its siblings.
   
   But it still includes all the downstream task siblings.
   
   In my screenshot for example, when I restart the create_job n°25 I still get all the mapped tasks for downstream task request_report_creation.
   
   Could we reopen this issue?
   
   <img width="885" height="586" alt="Image" src="https://github.com/user-attachments/assets/9d8811af-1421-4e65-9b35-a1e6701491ad" />
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

Posted by "MarcusCramer91 (via GitHub)" <gi...@apache.org>.
MarcusCramer91 commented on issue #54779:
URL: https://github.com/apache/airflow/issues/54779#issuecomment-3237451148

   > I am able to reproduce this behavior. However, I'd like to address this point below:
   > 
   > > Only downstream tasks with the same map index should be re-run (not sure this is easily possible or intended, but without it I don't see how restarting individual tasks would be of much use).
   > 
   > In this case, if I cleared the state of `task1[0]`, I would expect both `task2[0]` and `task1[0]` to be re-run, not just `task2[0]`.
   
   Agree! But I wouldn't expect task2[1] and task1[1] to be re-run, right?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

Posted by "matthieuauger (via GitHub)" <gi...@apache.org>.
matthieuauger commented on issue #54779:
URL: https://github.com/apache/airflow/issues/54779#issuecomment-3353662574

   Hello,
   
   I think the initial issue is related to "Downstream" related tasks and depth-first workflows.
   
   Use case is when you have multiple mapped tasks in a workflow, for example in a group.
   My personal is use case is that I download data for each of my users with mapped tasks.
   
   My DAG looks like this
   
   - TaskGroup (mapped to 10 users)
     - Task 1 : create_job (mapped to 10 users with the group)
     - Task 2 : download_data (mapped to 10 users with the group)
     - Task 3 : validate_data (mapped to 10 users with the group)
   
   Let's assume everything went well for 9 out of 10 users (index 0 to 8) but failed for last user (mapped task index 9).
   
   I would like to be able to restart task instance create_job[9], click downstream and only see download_data[9] and validate_date[9].
   
   This is really useful because currently if a mapped task fails,  you need to re-run everything.
   
   I tried again with new version but issue still persist
   
   <img width="1228" height="663" alt="Image" src="https://github.com/user-attachments/assets/11b6109b-bab2-44bf-acd4-54b7aeb165ce" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

Posted by "matthieuauger (via GitHub)" <gi...@apache.org>.
matthieuauger commented on issue #54779:
URL: https://github.com/apache/airflow/issues/54779#issuecomment-3467386753

   Indeed good catch! I just retried it and it seems to be an improvement from previous situation


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

Posted by "matthieuauger (via GitHub)" <gi...@apache.org>.
matthieuauger commented on issue #54779:
URL: https://github.com/apache/airflow/issues/54779#issuecomment-3467679843

   That would be a really huge improvement from my perspective, real one-piece-flow enabled on my DAGs. Can't wait


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Restarting a single mapped task restarts all sibling tasks as well [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on issue #54779:
URL: https://github.com/apache/airflow/issues/54779#issuecomment-3466029632

   So now it correctly only triggers `create_job[0]` (instead of all `create_job` instances), but the downstream handling is still not correct?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org