You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/01 18:18:27 UTC

[GitHub] [airflow] aaronmielke opened a new issue, #24788: Make task_instance priority_weight dag_run/execution_date dependent

aaronmielke opened a new issue, #24788:
URL: https://github.com/apache/airflow/issues/24788

   ### Description
   
   We want to weight a task's priority based on execution date. In general it would be nice to have the option to make the priority weight of a task instance dynamic based on the dag_run (from which execution_date and possibly other parameters can be derived)
   
   ### Use case/motivation
   
   We have a pool to run a set of tasks. Sometime we rerun a lot of old dag instances. This queues a LOT of task instances. Those dag rerun task instances end up delaying "today's" dag_instance tasks. If we weight today's dag_instance tasks with a higher priority weight this wouldn't happen.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #24788: Make task_instance priority_weight dag_run/execution_date dependent

Posted by GitBox <gi...@apache.org>.

potiuk closed issue #24788: Make task_instance priority_weight dag_run/execution_date dependent
URL: https://github.com/apache/airflow/issues/24788


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #24788: Make task_instance priority_weight dag_run/execution_date dependent

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #24788:
URL: https://github.com/apache/airflow/issues/24788#issuecomment-1174466564

   This would be a major change in the architectuer of scheduler and executor. I think it is not enough to make it a feature. If someone would like to implement such prioritisation that would require major re-architecting of how schedulers and executors work. That requires AIP https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals and extended discussion on devlist. converting this into discussion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #24788: Make task_instance priority_weight dag_run/execution_date dependent

Posted by GitBox <gi...@apache.org>.

boring-cyborg[bot] commented on issue #24788:
URL: https://github.com/apache/airflow/issues/24788#issuecomment-1172601900

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org