You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dstandish (via GitHub)" <gi...@apache.org> on 2024/10/21 19:51:27 UTC
[D] Should you be able to override pools in backfill? [airflow]
GitHub user dstandish created a discussion: Should you be able to override pools in backfill?
Note: I originally posted this on slack but I'm reposting here for posterity
So i was going to implement the "pool override" behavior on backfill.
But it strikes me that, the option to **provide one "pool override" that overrides pool for all tasks in the backfill**, that just doesn't make a lot of sense.
Obviously different tasks may use different pools. so all-or-nothing override doesn't make a lot of sense.
And note, in **the old backfill, it just straight up ignored pools unless you provided an override** --- by default just no pools were used. this was surprising to me.
But ok, i don't really like that option.
so what's the alternative? we could **allow user to provide a dict of tasks or something** where you optionally override the pool for any specific task.
and then if no override is provided, then use the normal pool. this would work and make more sense.
but it seems like a rather kludgy interface.
WDYT?
Separately though, **does anyone feel that backfill should still completely ignore pools by default?** Anyone think that behavior should be preserved?
Responses from slack:
>
> raphaelauv
> Yesterday at 1:22 PM
> Backfill was ignoring pools , that's a bug
> Backfill is a trick on time , not on the code of the dag , airflow should respect the code of the dag no matter what
>
>
> Daniel Standish
> [Yesterday at 1:24 PM](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729455867348749?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> And your opinion concerning a possible interface for overriding pools?
> [1:24](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729455889539649?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> Maybe it’s not really necessary, or not as important, now that we will have separate concurrency controls for backfill?
>
>
> Jarek Potiuk
> Yesterday at 2:54 PM
> Yeah. Backfill ignoring pools was a bug.
> I think we should not override pools at all with backfill - see the discusion on backfill vs. deferrable. There is a reason a task needs a pool, and backfill ignoring that is a bad idea. (edited)
> :heavy_plus_sign:
> 2
>
> [2:56](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729461380822389?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> There are probably a couple of “override” scenarios, but - paraphrasing your request
> [@dstandish](https://apache-airflow.slack.com/team/UG7RJQEV9)
> from the backfill/deferrable discussion - we should only allow overridability if someone comes up with - maybe a made up but realistic - case where it makes sense (edited)
>
>
> raphaelauv
> Yesterday at 5:05 PM
> a possible interface for overriding pools
> maybe some people would expect this feature ( and I don't see what are the purpose of pools in that case in this scenario ) , but they can always
> copy paste the dag
> sufix the name
> edit the pool usage in the tasks
> trigger the backill of the dag
>
>
> Daniel Standish
> [6 minutes ago](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729539875337429?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> and I don't see what are the purpose of pools in that case in this scenario
> [12:44](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729539888781359?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> pools are a bit of an enigma
> [12:45](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729539918844969?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> alright, well, if people say no override, then no override there shall be
> [12:45](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729539926740689?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> i will make a lazy consensus vote
>
>
> raphaelauv
> 4 minutes ago
> pools are a bit of an enigma
> I only use them as cross-dag semaphore
>
>
GitHub link: https://github.com/apache/airflow/discussions/43235
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Should you be able to override pools in backfill? [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
GitHub user dstandish edited a discussion: Should you be able to override pools in backfill?
Note: I originally posted this on slack but I'm reposting here for posterity
So i was going to implement the "pool override" behavior on backfill.
But it strikes me that, the option to **provide one "pool override" that overrides pool for all tasks in the backfill**, that just doesn't make a lot of sense.
Obviously different tasks may use different pools. so all-or-nothing override doesn't make a lot of sense.
And note, in **the old backfill, it just straight up ignored pools unless you provided an override** --- by default just no pools were used. this was surprising to me.
But ok, i don't really like that option.
so what's the alternative? we could **allow user to provide a dict of tasks or something** where you optionally override the pool for any specific task.
and then if no override is provided, then use the normal pool. this would work and make more sense.
but it seems like a rather kludgy interface.
WDYT?
Separately though, **does anyone feel that backfill should still completely ignore pools by default?** Anyone think that behavior should be preserved?
GitHub link: https://github.com/apache/airflow/discussions/43235
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Should you be able to override pools in backfill? [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
GitHub user dstandish edited a comment on the discussion: Should you be able to override pools in backfill?
> I feel either users never fully grasped what to expect from the backfills or just were happy with the behavior - I suspect it's the former.
yeah or maybe folks did not use backfill as much or did not use pools as much as we might think 🤷
backfill only makes sense when your dag is driven by `execution_date`. when i was airflow user, i almost never used execution date.
GitHub link: https://github.com/apache/airflow/discussions/43235#discussioncomment-11063260
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Should you be able to override pools in backfill? [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
GitHub user dstandish closed a discussion: Should you be able to override pools in backfill?
Note: I originally posted this on slack but I'm reposting here for posterity
So i was going to implement the "pool override" behavior on backfill.
But it strikes me that, the option to **provide one "pool override" that overrides pool for all tasks in the backfill**, that just doesn't make a lot of sense.
Obviously different tasks may use different pools. so all-or-nothing override doesn't make a lot of sense.
And note, in **the old backfill, it just straight up ignored pools unless you provided an override** --- by default just no pools were used. this was surprising to me.
But ok, i don't really like that option.
so what's the alternative? we could **allow user to provide a dict of tasks or something** where you optionally override the pool for any specific task.
and then if no override is provided, then use the normal pool. this would work and make more sense.
but it seems like a rather kludgy interface.
WDYT?
Separately though, **does anyone feel that backfill should still completely ignore pools by default?** Anyone think that behavior should be preserved?
Responses from slack:
>
> raphaelauv
> Yesterday at 1:22 PM
> Backfill was ignoring pools , that's a bug
> Backfill is a trick on time , not on the code of the dag , airflow should respect the code of the dag no matter what
>
>
> Daniel Standish
> [Yesterday at 1:24 PM](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729455867348749?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> And your opinion concerning a possible interface for overriding pools?
> [1:24](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729455889539649?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> Maybe it’s not really necessary, or not as important, now that we will have separate concurrency controls for backfill?
>
>
> Jarek Potiuk
> Yesterday at 2:54 PM
> Yeah. Backfill ignoring pools was a bug.
> I think we should not override pools at all with backfill - see the discusion on backfill vs. deferrable. There is a reason a task needs a pool, and backfill ignoring that is a bad idea. (edited)
> :heavy_plus_sign:
> 2
>
> [2:56](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729461380822389?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> There are probably a couple of “override” scenarios, but - paraphrasing your request
> [@dstandish](https://apache-airflow.slack.com/team/UG7RJQEV9)
> from the backfill/deferrable discussion - we should only allow overridability if someone comes up with - maybe a made up but realistic - case where it makes sense (edited)
>
>
> raphaelauv
> Yesterday at 5:05 PM
> a possible interface for overriding pools
> maybe some people would expect this feature ( and I don't see what are the purpose of pools in that case in this scenario ) , but they can always
> copy paste the dag
> sufix the name
> edit the pool usage in the tasks
> trigger the backill of the dag
>
>
> Daniel Standish
> [6 minutes ago](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729539875337429?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> and I don't see what are the purpose of pools in that case in this scenario
> [12:44](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729539888781359?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> pools are a bit of an enigma
> [12:45](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729539918844969?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> alright, well, if people say no override, then no override there shall be
> [12:45](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729539926740689?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> i will make a lazy consensus vote
>
>
> raphaelauv
> 4 minutes ago
> pools are a bit of an enigma
> I only use them as cross-dag semaphore
>
>
GitHub link: https://github.com/apache/airflow/discussions/43235
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Should you be able to override pools in backfill? [airflow]
Posted by "shubham22 (via GitHub)" <gi...@apache.org>.
GitHub user shubham22 closed the discussion with a comment: Should you be able to override pools in backfill?
Probably, I do see "backfill" mentioned often in internal slack channels, and most ask about the CLI command or try to understand best practices.
GitHub link: https://github.com/apache/airflow/discussions/43235#discussioncomment-11063446
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Should you be able to override pools in backfill? [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
GitHub user dstandish edited a comment on the discussion: Should you be able to override pools in backfill?
> I feel either users never fully grasped what to expect from the backfills or just were happy with the behavior - I suspect it's the former.
yeah or maybe folks did not use backfill as much or did not use pools as much as we might think
GitHub link: https://github.com/apache/airflow/discussions/43235#discussioncomment-11063260
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Should you be able to override pools in backfill? [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
GitHub user dstandish closed the discussion with a comment: Should you be able to override pools in backfill?
Responses from slack:
>
> raphaelauv
> Yesterday at 1:22 PM
> Backfill was ignoring pools , that's a bug
> Backfill is a trick on time , not on the code of the dag , airflow should respect the code of the dag no matter what
>
>
> Daniel Standish
> [Yesterday at 1:24 PM](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729455867348749?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> And your opinion concerning a possible interface for overriding pools?
> [1:24](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729455889539649?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> Maybe it’s not really necessary, or not as important, now that we will have separate concurrency controls for backfill?
>
>
> Jarek Potiuk
> Yesterday at 2:54 PM
> Yeah. Backfill ignoring pools was a bug.
> I think we should not override pools at all with backfill - see the discusion on backfill vs. deferrable. There is a reason a task needs a pool, and backfill ignoring that is a bad idea. (edited)
> :heavy_plus_sign:
> 2
>
> [2:56](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729461380822389?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> There are probably a couple of “override” scenarios, but - paraphrasing your request
> [@dstandish](https://apache-airflow.slack.com/team/UG7RJQEV9)
> from the backfill/deferrable discussion - we should only allow overridability if someone comes up with - maybe a made up but realistic - case where it makes sense (edited)
>
>
> raphaelauv
> Yesterday at 5:05 PM
> a possible interface for overriding pools
> maybe some people would expect this feature ( and I don't see what are the purpose of pools in that case in this scenario ) , but they can always
> copy paste the dag
> sufix the name
> edit the pool usage in the tasks
> trigger the backill of the dag
>
>
> Daniel Standish
> [6 minutes ago](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729539875337429?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> and I don't see what are the purpose of pools in that case in this scenario
> [12:44](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729539888781359?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> pools are a bit of an enigma
> [12:45](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729539918844969?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> alright, well, if people say no override, then no override there shall be
> [12:45](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729539926740689?thread_ts=1729436754.752759&cid=C06K9Q5G2UA)
> i will make a lazy consensus vote
>
>
> raphaelauv
> 4 minutes ago
> pools are a bit of an enigma
> I only use them as cross-dag semaphore
>
>
GitHub link: https://github.com/apache/airflow/discussions/43235#discussioncomment-11009376
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Should you be able to override pools in backfill? [airflow]
Posted by "shubham22 (via GitHub)" <gi...@apache.org>.
GitHub user shubham22 closed the discussion with a comment: Should you be able to override pools in backfill?
To some extent, I am kind of surprised that no user screamed about pools being ignored during backfill, at least I didn't find an open issue on a quick search. I feel either users never fully grasped what to expect from the backfills or just were happy with the behavior - I suspect it's the former.
Overall, I agree with the direction to treat pools as an integral part of DAG that should be respected during backfills. Backfills should only be modifying the data interval, and the rest of the behavior should mimic the regular DAG run. This option also keeps the interface less complex and more predictable. If there is a need for override, which I doubt, it's much easier to add later when we have concrete use cases.
GitHub link: https://github.com/apache/airflow/discussions/43235#discussioncomment-11063048
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Should you be able to override pools in backfill? [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
GitHub user dstandish edited a comment on the discussion: Should you be able to override pools in backfill?
> I feel either users never fully grasped what to expect from the backfills or just were happy with the behavior - I suspect it's the former.
yeah or maybe folks did not use backfill as much or did not use pools as much as we might think 🤷
GitHub link: https://github.com/apache/airflow/discussions/43235#discussioncomment-11063260
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Should you be able to override pools in backfill? [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
GitHub user dstandish closed the discussion with a comment: Should you be able to override pools in backfill?
>yeah or maybe people don't use backfill much
GitHub link: https://github.com/apache/airflow/discussions/43235#discussioncomment-11063260
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org