You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/04 10:03:08 UTC
[GitHub] [airflow] honarkhah opened a new issue #12807: add weight_rule to config
honarkhah opened a new issue #12807:
URL: https://github.com/apache/airflow/issues/12807
**Description**
It would be nice if the weight_rule default value could be managed by a global config
suggested config:
```
# Weighting method used for the effective total priority weight of the task.
# Options are: { downstream | upstream | absolute } default is
weight_rule = downstream
```
**Use case / motivation**
In some pipeline, you really need to have absolute weight, and then you have to add a line in each task definition which is annoying
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #12807: add weight_rule to config
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12807:
URL: https://github.com/apache/airflow/issues/12807#issuecomment-773412937
> Guess that is the point of issue!
You described that you want to avoid specifying the same value of `weight_rule` per task.
For that you can define it once in `default_args`.
Though it's not exactly what you are describing maybe [task policy](https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#task-level-cluster-policy) can solve some of it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #12807: add weight_rule to config
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #12807:
URL: https://github.com/apache/airflow/issues/12807#issuecomment-773412937
> Guess that is the point of issue!
You described that you want to avoid specifying the same value of `weight_rule` per task.
For that you can define it once in `default_args`.
Maybe [task policy](https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#task-level-cluster-policy) can solve some of it? (Though not exactly the same as this overwrites values provided by the user)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #12807: add weight_rule to config
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #12807:
URL: https://github.com/apache/airflow/issues/12807#issuecomment-773412937
> Guess that is the point of issue!
You described that you want to avoid specifying the same value of `weight_rule` per task.
For that you can define it once in `default_args`.
Maybe [task policy](https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#task-level-cluster-policy) can solve some of the problem you described? (Though not exactly the same as this overwrites values provided by the user)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #12807: add weight_rule to config
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #12807:
URL: https://github.com/apache/airflow/issues/12807#issuecomment-773412937
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #12807: add weight_rule to config
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12807:
URL: https://github.com/apache/airflow/issues/12807#issuecomment-773373670
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] honarkhah commented on issue #12807: add weight_rule to config
Posted by GitBox <gi...@apache.org>.
honarkhah commented on issue #12807:
URL: https://github.com/apache/airflow/issues/12807#issuecomment-773415168
I really think having that one configurable for entire cluster could be
helpful!
On Thu, Feb 4, 2021, 4:56 PM eladkal <no...@github.com> wrote:
> Guess that is the point of issue!
>
> You described that you want to avoid specifying the same value of
> weight_rule per task.
> For that you can define it once in default_args.
>
> Though it's not exactly what you are describing maybe task policy
> <https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#task-level-cluster-policy>
> can solve some of it.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/issues/12807#issuecomment-773412937>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAG447JD64NOSWACDNYS2ADS5K7TJANCNFSM4UNG56PA>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] honarkhah commented on issue #12807: add weight_rule to config
Posted by GitBox <gi...@apache.org>.
honarkhah commented on issue #12807:
URL: https://github.com/apache/airflow/issues/12807#issuecomment-773394570
Then it needs to be done in each dag, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] honarkhah commented on issue #12807: add default weight_rule to airflow.cfg
Posted by GitBox <gi...@apache.org>.
honarkhah commented on issue #12807:
URL: https://github.com/apache/airflow/issues/12807#issuecomment-933241449
thanks for this feature ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #12807: add weight_rule to config
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12807:
URL: https://github.com/apache/airflow/issues/12807#issuecomment-773373670
>In some pipeline, you really need to have absolute weight, and then you have to add a line in each task definition which is annoying
Why not define it once in `default_args`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #12807: add weight_rule to config
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #12807:
URL: https://github.com/apache/airflow/issues/12807#issuecomment-773412937
> Guess that is the point of issue!
You described that you want to avoid specifying the same value of `weight_rule` per task.
For that you can define it once in `default_args`.
Though it's not exactly what you are describing maybe [task policy](https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#task-level-cluster-policy) can solve some of it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] honarkhah commented on issue #12807: add weight_rule to config
Posted by GitBox <gi...@apache.org>.
honarkhah commented on issue #12807:
URL: https://github.com/apache/airflow/issues/12807#issuecomment-773394570
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] honarkhah commented on issue #12807: add weight_rule to config
Posted by GitBox <gi...@apache.org>.
honarkhah commented on issue #12807:
URL: https://github.com/apache/airflow/issues/12807#issuecomment-773405722
Guess that is the point of issue!
On Thu, Feb 4, 2021 at 4:37 PM eladkal <no...@github.com> wrote:
> Then it needs to be done in each dag, right?
>
> Yes
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/issues/12807#issuecomment-773399489>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAG447LNSTCWEVHQ57KA7SDS5K5LDANCNFSM4UNG56PA>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #12807: add weight_rule to config
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12807:
URL: https://github.com/apache/airflow/issues/12807#issuecomment-773399489
> Then it needs to be done in each dag, right?
Yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #12807: add default weight_rule to airflow.cfg
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #12807:
URL: https://github.com/apache/airflow/issues/12807
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org