[GitHub] [airflow] JGoldman110 commented on issue #20143: task_instance_mutation_hook not applied every time

[GitBox <gi...@apache.org>]

JGoldman110 commented on issue #20143:
URL: https://github.com/apache/airflow/issues/20143#issuecomment-1221600291

   We are currently trying to use the `task_instance_mutation_hook` as a way to have manual task runs sent to a specific queue. We have found that using the `task_instance_mutation_hook` to achieve this works well for manual and scheduled runs. However, when clearing a task, `task_instance_mutation_hook` is not called.
   
   When a task is cleared we found that original queue assigned to that task from the `task_instance_mutation_hook ` is not kept. Instead the [`refresh_from_task`](https://github.com/apache/airflow/blob/2.1.3/airflow/models/taskinstance.py#L658) method is called which sets the queue based on the task object.
   
   ```
   def refresh_from_task(self, task, pool_override=None):
       """
       Copy common attributes from the given task.
       :param task: The task object to copy from
       :type task: airflow.models.BaseOperator
       :param pool_override: Use the pool_override instead of task's pool
       :type pool_override: str
       """
       self.queue = task.queue # -------------------------> Original task assigned by task_instance_mutation_hook is reset
       self.pool = pool_override or task.pool
       self.pool_slots = task.pool_slots
       self.priority_weight = task.priority_weight_total
       self.run_as_user = task.run_as_user
       self.max_tries = task.retries
       self.executor_config = task.executor_config
       self.operator = task.task_type
   ```
   
   If the `task_instance_mutation_hook` is only called when the task instance is first created, we would expect that the task instance original assigned queue should be preserved on clear. Or, the `task_instance_mutation_hook` should be called again on clearing a task.
   
   @dstandish any thoughts on preserving the queue on clear or calling `task_instance_mutation_hook` on clear?
   
   Airflow Version: 2.1.3


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org