You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Blain David <da...@infrabel.be> on 2026/10/01 08:00:21 UTC
[VOTE] AIP-104 Iterable Tasks (IT) and Task Spreading (TS)
Hi everyone,
I would like to restart the vote on AIP-104, now titled Iterable Tasks (IT) and Task Spreading (TS):
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
The previous vote, under the name "Task Iteration and Dynamic Task Batching", ran during the summer and did not get much engagement:
https://lists.apache.org/thread/x5bd26zfqkkhp7sx7nb0crjlsctf2z5j
The AIP has changed since then, so please vote on the current page rather than on what you may remember:
- Renamed "Task Iteration (TI)" to "Iterable Tasks (IT)", as TI is already widely used for Task Instance (as remarked by Daniel).
- Renamed "Dynamic Task Batching (DTB)" to "Task Spreading (TS)". The primitive is now .spread(across=N) instead of .batch(size=N), as "size" read like a chunk length while it is the number of Task Instances. Items are distributed round robin over those N Task Instances. The new name was discussed and voted on in separate threads (linked below).
- Iterable Tasks now checkpoint progress per item through AIP-103 Task State. On a retry, or after a worker crash, items that already finished are not run again; only the items that failed or were in flight are.
- The page now documents task_concurrency, the setting that controls how many items run at once inside one Task Instance.
- The page now lists the current limits: deferrable operators, reschedule-mode sensors and TriggerDagRunOperator are not supported inside .iterate() yet (deferrable support is planned in a separate PR); execution_timeout applies per item only to async operators; multiple_outputs is ignored; and pools count the Task Instance rather than its items.
The previous discussion threads can be found here:
https://lists.apache.org/thread/ztnfsqolow4v1zsv4pkpnxc1fk0hbf2p
https://lists.apache.org/thread/tr3x4sx55k0fk2coq28xs2648m1p65cc
The naming of the spread primitive:
https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2 (discussion)
https://lists.apache.org/thread/rrc9bmhh5dcpyj4rcpjr2lj1q8vjdoxp (vote)
Thank you again to everyone who reviewed the AIP and took part in the discussions. A special thank you to Kaxil for the many detailed reviews, and to TP, Jens, Jarek and Ash for their comments. Thanks as well to Daniel, Jedidiah and Amogh for their more recent reviews on the AIP page, and to everyone who voted on the naming of the spread primitive.
The vote will remain open for 5 days and will close on Wednesday, 7 October 2026, at 00:00 GMT.
Everyone is encouraged to vote, although only votes from PMC members and committers are considered binding.
Please vote accordingly:
[ ] +1 Approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason
Please consider this my +1 (binding).
Regards,
David
General (Internal Property)
Re: [VOTE] AIP-104 Iterable Tasks (IT) and Task Spreading (TS)
Posted by Ash Berlin-Taylor <as...@apache.org>.
Thanks for the changes and refocusing. +1!
> On 1 Oct 2026, at 09:00, Blain David <da...@infrabel.be> wrote:
>
> Hi everyone,
>
> I would like to restart the vote on AIP-104, now titled Iterable Tasks (IT) and Task Spreading (TS):
>
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
>
> The previous vote, under the name "Task Iteration and Dynamic Task Batching", ran during the summer and did not get much engagement:
>
> https://lists.apache.org/thread/x5bd26zfqkkhp7sx7nb0crjlsctf2z5j
>
> The AIP has changed since then, so please vote on the current page rather than on what you may remember:
>
> - Renamed "Task Iteration (TI)" to "Iterable Tasks (IT)", as TI is already widely used for Task Instance (as remarked by Daniel).
> - Renamed "Dynamic Task Batching (DTB)" to "Task Spreading (TS)". The primitive is now .spread(across=N) instead of .batch(size=N), as "size" read like a chunk length while it is the number of Task Instances. Items are distributed round robin over those N Task Instances. The new name was discussed and voted on in separate threads (linked below).
> - Iterable Tasks now checkpoint progress per item through AIP-103 Task State. On a retry, or after a worker crash, items that already finished are not run again; only the items that failed or were in flight are.
> - The page now documents task_concurrency, the setting that controls how many items run at once inside one Task Instance.
> - The page now lists the current limits: deferrable operators, reschedule-mode sensors and TriggerDagRunOperator are not supported inside .iterate() yet (deferrable support is planned in a separate PR); execution_timeout applies per item only to async operators; multiple_outputs is ignored; and pools count the Task Instance rather than its items.
>
> The previous discussion threads can be found here:
>
> https://lists.apache.org/thread/ztnfsqolow4v1zsv4pkpnxc1fk0hbf2p
> https://lists.apache.org/thread/tr3x4sx55k0fk2coq28xs2648m1p65cc
>
> The naming of the spread primitive:
>
> https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2 (discussion)
> https://lists.apache.org/thread/rrc9bmhh5dcpyj4rcpjr2lj1q8vjdoxp (vote)
>
> Thank you again to everyone who reviewed the AIP and took part in the discussions. A special thank you to Kaxil for the many detailed reviews, and to TP, Jens, Jarek and Ash for their comments. Thanks as well to Daniel, Jedidiah and Amogh for their more recent reviews on the AIP page, and to everyone who voted on the naming of the spread primitive.
>
> The vote will remain open for 5 days and will close on Wednesday, 7 October 2026, at 00:00 GMT.
>
> Everyone is encouraged to vote, although only votes from PMC members and committers are considered binding.
>
> Please vote accordingly:
>
> [ ] +1 Approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
> Please consider this my +1 (binding).
>
> Regards,
> David
>
> General (Internal Property)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [VOTE] AIP-104 Iterable Tasks (IT) and Task Spreading (TS)
Posted by Jarek Potiuk <ja...@potiuk.com>.
+1 - looking forward to some optimized flows it will enable!
On Mon, Oct 5, 2026 at 8:41 AM Amogh Desai <am...@apache.org> wrote:
>
> +1 binding.
>
> Real gap.
>
> On the feature name itself, I don't really care much if we call it Task
> Spreading or
> something else, like Ash was suggesting.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Thu, Oct 1, 2026 at 1:30 PM Blain David <da...@infrabel.be> wrote:
>
> > Hi everyone,
> >
> > I would like to restart the vote on AIP-104, now titled Iterable Tasks
> > (IT) and Task Spreading (TS):
> >
> >
> > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
> >
> > The previous vote, under the name "Task Iteration and Dynamic Task
> > Batching", ran during the summer and did not get much engagement:
> >
> > https://lists.apache.org/thread/x5bd26zfqkkhp7sx7nb0crjlsctf2z5j
> >
> > The AIP has changed since then, so please vote on the current page rather
> > than on what you may remember:
> >
> > - Renamed "Task Iteration (TI)" to "Iterable Tasks (IT)", as TI is already
> > widely used for Task Instance (as remarked by Daniel).
> > - Renamed "Dynamic Task Batching (DTB)" to "Task Spreading (TS)". The
> > primitive is now .spread(across=N) instead of .batch(size=N), as "size"
> > read like a chunk length while it is the number of Task Instances. Items
> > are distributed round robin over those N Task Instances. The new name was
> > discussed and voted on in separate threads (linked below).
> > - Iterable Tasks now checkpoint progress per item through AIP-103 Task
> > State. On a retry, or after a worker crash, items that already finished are
> > not run again; only the items that failed or were in flight are.
> > - The page now documents task_concurrency, the setting that controls how
> > many items run at once inside one Task Instance.
> > - The page now lists the current limits: deferrable operators,
> > reschedule-mode sensors and TriggerDagRunOperator are not supported inside
> > .iterate() yet (deferrable support is planned in a separate PR);
> > execution_timeout applies per item only to async operators;
> > multiple_outputs is ignored; and pools count the Task Instance rather than
> > its items.
> >
> > The previous discussion threads can be found here:
> >
> > https://lists.apache.org/thread/ztnfsqolow4v1zsv4pkpnxc1fk0hbf2p
> > https://lists.apache.org/thread/tr3x4sx55k0fk2coq28xs2648m1p65cc
> >
> > The naming of the spread primitive:
> >
> > https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2
> > (discussion)
> > https://lists.apache.org/thread/rrc9bmhh5dcpyj4rcpjr2lj1q8vjdoxp (vote)
> >
> > Thank you again to everyone who reviewed the AIP and took part in the
> > discussions. A special thank you to Kaxil for the many detailed reviews,
> > and to TP, Jens, Jarek and Ash for their comments. Thanks as well to
> > Daniel, Jedidiah and Amogh for their more recent reviews on the AIP page,
> > and to everyone who voted on the naming of the spread primitive.
> >
> > The vote will remain open for 5 days and will close on Wednesday, 7
> > October 2026, at 00:00 GMT.
> >
> > Everyone is encouraged to vote, although only votes from PMC members and
> > committers are considered binding.
> >
> > Please vote accordingly:
> >
> > [ ] +1 Approve
> > [ ] +0 no opinion
> > [ ] -1 disapprove with the reason
> >
> > Please consider this my +1 (binding).
> >
> > Regards,
> > David
> >
> > General (Internal Property)
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [VOTE] AIP-104 Iterable Tasks (IT) and Task Spreading (TS)
Posted by Amogh Desai <am...@apache.org>.
+1 binding.
Real gap.
On the feature name itself, I don't really care much if we call it Task
Spreading or
something else, like Ash was suggesting.
Thanks & Regards,
Amogh Desai
On Thu, Oct 1, 2026 at 1:30 PM Blain David <da...@infrabel.be> wrote:
> Hi everyone,
>
> I would like to restart the vote on AIP-104, now titled Iterable Tasks
> (IT) and Task Spreading (TS):
>
>
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
>
> The previous vote, under the name "Task Iteration and Dynamic Task
> Batching", ran during the summer and did not get much engagement:
>
> https://lists.apache.org/thread/x5bd26zfqkkhp7sx7nb0crjlsctf2z5j
>
> The AIP has changed since then, so please vote on the current page rather
> than on what you may remember:
>
> - Renamed "Task Iteration (TI)" to "Iterable Tasks (IT)", as TI is already
> widely used for Task Instance (as remarked by Daniel).
> - Renamed "Dynamic Task Batching (DTB)" to "Task Spreading (TS)". The
> primitive is now .spread(across=N) instead of .batch(size=N), as "size"
> read like a chunk length while it is the number of Task Instances. Items
> are distributed round robin over those N Task Instances. The new name was
> discussed and voted on in separate threads (linked below).
> - Iterable Tasks now checkpoint progress per item through AIP-103 Task
> State. On a retry, or after a worker crash, items that already finished are
> not run again; only the items that failed or were in flight are.
> - The page now documents task_concurrency, the setting that controls how
> many items run at once inside one Task Instance.
> - The page now lists the current limits: deferrable operators,
> reschedule-mode sensors and TriggerDagRunOperator are not supported inside
> .iterate() yet (deferrable support is planned in a separate PR);
> execution_timeout applies per item only to async operators;
> multiple_outputs is ignored; and pools count the Task Instance rather than
> its items.
>
> The previous discussion threads can be found here:
>
> https://lists.apache.org/thread/ztnfsqolow4v1zsv4pkpnxc1fk0hbf2p
> https://lists.apache.org/thread/tr3x4sx55k0fk2coq28xs2648m1p65cc
>
> The naming of the spread primitive:
>
> https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2
> (discussion)
> https://lists.apache.org/thread/rrc9bmhh5dcpyj4rcpjr2lj1q8vjdoxp (vote)
>
> Thank you again to everyone who reviewed the AIP and took part in the
> discussions. A special thank you to Kaxil for the many detailed reviews,
> and to TP, Jens, Jarek and Ash for their comments. Thanks as well to
> Daniel, Jedidiah and Amogh for their more recent reviews on the AIP page,
> and to everyone who voted on the naming of the spread primitive.
>
> The vote will remain open for 5 days and will close on Wednesday, 7
> October 2026, at 00:00 GMT.
>
> Everyone is encouraged to vote, although only votes from PMC members and
> committers are considered binding.
>
> Please vote accordingly:
>
> [ ] +1 Approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
> Please consider this my +1 (binding).
>
> Regards,
> David
>
> General (Internal Property)
>
Re: [VOTE] AIP-104 Iterable Tasks (IT) and Task Spreading (TS)
Posted by Wei Lee <we...@apache.org>.
+1 (binding)
Best,
Wei
On 2026/10/06 16:37:17 Vikram Koka via dev wrote:
> +1 (binding) on the feature.
>
> I have no opinion on the naming either.
>
> Vikram
>
> On Tue, Oct 6, 2026 at 5:59 AM Dheeraj Turaga <tu...@apache.org> wrote:
>
> > +1 (binding)
> >
> > On Mon, Oct 5, 2026 at 3:38 PM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > +1 on the features. No opinion on naming
> > >
> > > On Mon, 5 Oct 2026 at 18:08, Zach Gottesman via dev <
> > > dev@airflow.apache.org>
> > > wrote:
> > >
> > > > +1 (non-binding). Coincidentally I was just discussing the need for
> > > > something similar to this in a meeting earlier today. Would love to see
> > > it
> > > > as part of the Airflow public interface.
> > > >
> > > > On Mon, Oct 5, 2026 at 9:36 AM Jens Scheffler via dev <
> > > > dev@airflow.apache.org> wrote:
> > > >
> > > > > +1
> > > > >
> > > > > On 05.10.26 11:45, Shahar Epstein wrote:
> > > > > > +1 (binding), thanks for leading this effort!
> > > > > >
> > > > > >
> > > > > > Shahar
> > > > > >
> > > > > > On Thu, Oct 1, 2026, 11:00 Blain David <da...@infrabel.be>
> > > > wrote:
> > > > > >
> > > > > >> Hi everyone,
> > > > > >>
> > > > > >> I would like to restart the vote on AIP-104, now titled Iterable
> > > Tasks
> > > > > >> (IT) and Task Spreading (TS):
> > > > > >>
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
> > > > > >>
> > > > > >> The previous vote, under the name "Task Iteration and Dynamic Task
> > > > > >> Batching", ran during the summer and did not get much engagement:
> > > > > >>
> > > > > >> https://lists.apache.org/thread/x5bd26zfqkkhp7sx7nb0crjlsctf2z5j
> > > > > >>
> > > > > >> The AIP has changed since then, so please vote on the current page
> > > > > rather
> > > > > >> than on what you may remember:
> > > > > >>
> > > > > >> - Renamed "Task Iteration (TI)" to "Iterable Tasks (IT)", as TI is
> > > > > already
> > > > > >> widely used for Task Instance (as remarked by Daniel).
> > > > > >> - Renamed "Dynamic Task Batching (DTB)" to "Task Spreading (TS)".
> > > The
> > > > > >> primitive is now .spread(across=N) instead of .batch(size=N), as
> > > > "size"
> > > > > >> read like a chunk length while it is the number of Task Instances.
> > > > Items
> > > > > >> are distributed round robin over those N Task Instances. The new
> > > name
> > > > > was
> > > > > >> discussed and voted on in separate threads (linked below).
> > > > > >> - Iterable Tasks now checkpoint progress per item through AIP-103
> > > Task
> > > > > >> State. On a retry, or after a worker crash, items that already
> > > > finished
> > > > > are
> > > > > >> not run again; only the items that failed or were in flight are.
> > > > > >> - The page now documents task_concurrency, the setting that
> > controls
> > > > how
> > > > > >> many items run at once inside one Task Instance.
> > > > > >> - The page now lists the current limits: deferrable operators,
> > > > > >> reschedule-mode sensors and TriggerDagRunOperator are not
> > supported
> > > > > inside
> > > > > >> .iterate() yet (deferrable support is planned in a separate PR);
> > > > > >> execution_timeout applies per item only to async operators;
> > > > > >> multiple_outputs is ignored; and pools count the Task Instance
> > > rather
> > > > > than
> > > > > >> its items.
> > > > > >>
> > > > > >> The previous discussion threads can be found here:
> > > > > >>
> > > > > >> https://lists.apache.org/thread/ztnfsqolow4v1zsv4pkpnxc1fk0hbf2p
> > > > > >> https://lists.apache.org/thread/tr3x4sx55k0fk2coq28xs2648m1p65cc
> > > > > >>
> > > > > >> The naming of the spread primitive:
> > > > > >>
> > > > > >> https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2
> > > > > >> (discussion)
> > > > > >> https://lists.apache.org/thread/rrc9bmhh5dcpyj4rcpjr2lj1q8vjdoxp
> > > > (vote)
> > > > > >>
> > > > > >> Thank you again to everyone who reviewed the AIP and took part in
> > > the
> > > > > >> discussions. A special thank you to Kaxil for the many detailed
> > > > reviews,
> > > > > >> and to TP, Jens, Jarek and Ash for their comments. Thanks as well
> > to
> > > > > >> Daniel, Jedidiah and Amogh for their more recent reviews on the
> > AIP
> > > > > page,
> > > > > >> and to everyone who voted on the naming of the spread primitive.
> > > > > >>
> > > > > >> The vote will remain open for 5 days and will close on Wednesday,
> > 7
> > > > > >> October 2026, at 00:00 GMT.
> > > > > >>
> > > > > >> Everyone is encouraged to vote, although only votes from PMC
> > members
> > > > and
> > > > > >> committers are considered binding.
> > > > > >>
> > > > > >> Please vote accordingly:
> > > > > >>
> > > > > >> [ ] +1 Approve
> > > > > >> [ ] +0 no opinion
> > > > > >> [ ] -1 disapprove with the reason
> > > > > >>
> > > > > >> Please consider this my +1 (binding).
> > > > > >>
> > > > > >> Regards,
> > > > > >> David
> > > > > >>
> > > > > >> General (Internal Property)
> > > > > >>
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [VOTE] AIP-104 Iterable Tasks (IT) and Task Spreading (TS)
Posted by Vikram Koka via dev <de...@airflow.apache.org>.
+1 (binding) on the feature.
I have no opinion on the naming either.
Vikram
On Tue, Oct 6, 2026 at 5:59 AM Dheeraj Turaga <tu...@apache.org> wrote:
> +1 (binding)
>
> On Mon, Oct 5, 2026 at 3:38 PM Kaxil Naik <ka...@gmail.com> wrote:
>
> > +1 on the features. No opinion on naming
> >
> > On Mon, 5 Oct 2026 at 18:08, Zach Gottesman via dev <
> > dev@airflow.apache.org>
> > wrote:
> >
> > > +1 (non-binding). Coincidentally I was just discussing the need for
> > > something similar to this in a meeting earlier today. Would love to see
> > it
> > > as part of the Airflow public interface.
> > >
> > > On Mon, Oct 5, 2026 at 9:36 AM Jens Scheffler via dev <
> > > dev@airflow.apache.org> wrote:
> > >
> > > > +1
> > > >
> > > > On 05.10.26 11:45, Shahar Epstein wrote:
> > > > > +1 (binding), thanks for leading this effort!
> > > > >
> > > > >
> > > > > Shahar
> > > > >
> > > > > On Thu, Oct 1, 2026, 11:00 Blain David <da...@infrabel.be>
> > > wrote:
> > > > >
> > > > >> Hi everyone,
> > > > >>
> > > > >> I would like to restart the vote on AIP-104, now titled Iterable
> > Tasks
> > > > >> (IT) and Task Spreading (TS):
> > > > >>
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
> > > > >>
> > > > >> The previous vote, under the name "Task Iteration and Dynamic Task
> > > > >> Batching", ran during the summer and did not get much engagement:
> > > > >>
> > > > >> https://lists.apache.org/thread/x5bd26zfqkkhp7sx7nb0crjlsctf2z5j
> > > > >>
> > > > >> The AIP has changed since then, so please vote on the current page
> > > > rather
> > > > >> than on what you may remember:
> > > > >>
> > > > >> - Renamed "Task Iteration (TI)" to "Iterable Tasks (IT)", as TI is
> > > > already
> > > > >> widely used for Task Instance (as remarked by Daniel).
> > > > >> - Renamed "Dynamic Task Batching (DTB)" to "Task Spreading (TS)".
> > The
> > > > >> primitive is now .spread(across=N) instead of .batch(size=N), as
> > > "size"
> > > > >> read like a chunk length while it is the number of Task Instances.
> > > Items
> > > > >> are distributed round robin over those N Task Instances. The new
> > name
> > > > was
> > > > >> discussed and voted on in separate threads (linked below).
> > > > >> - Iterable Tasks now checkpoint progress per item through AIP-103
> > Task
> > > > >> State. On a retry, or after a worker crash, items that already
> > > finished
> > > > are
> > > > >> not run again; only the items that failed or were in flight are.
> > > > >> - The page now documents task_concurrency, the setting that
> controls
> > > how
> > > > >> many items run at once inside one Task Instance.
> > > > >> - The page now lists the current limits: deferrable operators,
> > > > >> reschedule-mode sensors and TriggerDagRunOperator are not
> supported
> > > > inside
> > > > >> .iterate() yet (deferrable support is planned in a separate PR);
> > > > >> execution_timeout applies per item only to async operators;
> > > > >> multiple_outputs is ignored; and pools count the Task Instance
> > rather
> > > > than
> > > > >> its items.
> > > > >>
> > > > >> The previous discussion threads can be found here:
> > > > >>
> > > > >> https://lists.apache.org/thread/ztnfsqolow4v1zsv4pkpnxc1fk0hbf2p
> > > > >> https://lists.apache.org/thread/tr3x4sx55k0fk2coq28xs2648m1p65cc
> > > > >>
> > > > >> The naming of the spread primitive:
> > > > >>
> > > > >> https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2
> > > > >> (discussion)
> > > > >> https://lists.apache.org/thread/rrc9bmhh5dcpyj4rcpjr2lj1q8vjdoxp
> > > (vote)
> > > > >>
> > > > >> Thank you again to everyone who reviewed the AIP and took part in
> > the
> > > > >> discussions. A special thank you to Kaxil for the many detailed
> > > reviews,
> > > > >> and to TP, Jens, Jarek and Ash for their comments. Thanks as well
> to
> > > > >> Daniel, Jedidiah and Amogh for their more recent reviews on the
> AIP
> > > > page,
> > > > >> and to everyone who voted on the naming of the spread primitive.
> > > > >>
> > > > >> The vote will remain open for 5 days and will close on Wednesday,
> 7
> > > > >> October 2026, at 00:00 GMT.
> > > > >>
> > > > >> Everyone is encouraged to vote, although only votes from PMC
> members
> > > and
> > > > >> committers are considered binding.
> > > > >>
> > > > >> Please vote accordingly:
> > > > >>
> > > > >> [ ] +1 Approve
> > > > >> [ ] +0 no opinion
> > > > >> [ ] -1 disapprove with the reason
> > > > >>
> > > > >> Please consider this my +1 (binding).
> > > > >>
> > > > >> Regards,
> > > > >> David
> > > > >>
> > > > >> General (Internal Property)
> > > > >>
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > > >
> > >
> >
>
Re: [VOTE] AIP-104 Iterable Tasks (IT) and Task Spreading (TS)
Posted by Dheeraj Turaga <tu...@apache.org>.
+1 (binding)
On Mon, Oct 5, 2026 at 3:38 PM Kaxil Naik <ka...@gmail.com> wrote:
> +1 on the features. No opinion on naming
>
> On Mon, 5 Oct 2026 at 18:08, Zach Gottesman via dev <
> dev@airflow.apache.org>
> wrote:
>
> > +1 (non-binding). Coincidentally I was just discussing the need for
> > something similar to this in a meeting earlier today. Would love to see
> it
> > as part of the Airflow public interface.
> >
> > On Mon, Oct 5, 2026 at 9:36 AM Jens Scheffler via dev <
> > dev@airflow.apache.org> wrote:
> >
> > > +1
> > >
> > > On 05.10.26 11:45, Shahar Epstein wrote:
> > > > +1 (binding), thanks for leading this effort!
> > > >
> > > >
> > > > Shahar
> > > >
> > > > On Thu, Oct 1, 2026, 11:00 Blain David <da...@infrabel.be>
> > wrote:
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> I would like to restart the vote on AIP-104, now titled Iterable
> Tasks
> > > >> (IT) and Task Spreading (TS):
> > > >>
> > > >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
> > > >>
> > > >> The previous vote, under the name "Task Iteration and Dynamic Task
> > > >> Batching", ran during the summer and did not get much engagement:
> > > >>
> > > >> https://lists.apache.org/thread/x5bd26zfqkkhp7sx7nb0crjlsctf2z5j
> > > >>
> > > >> The AIP has changed since then, so please vote on the current page
> > > rather
> > > >> than on what you may remember:
> > > >>
> > > >> - Renamed "Task Iteration (TI)" to "Iterable Tasks (IT)", as TI is
> > > already
> > > >> widely used for Task Instance (as remarked by Daniel).
> > > >> - Renamed "Dynamic Task Batching (DTB)" to "Task Spreading (TS)".
> The
> > > >> primitive is now .spread(across=N) instead of .batch(size=N), as
> > "size"
> > > >> read like a chunk length while it is the number of Task Instances.
> > Items
> > > >> are distributed round robin over those N Task Instances. The new
> name
> > > was
> > > >> discussed and voted on in separate threads (linked below).
> > > >> - Iterable Tasks now checkpoint progress per item through AIP-103
> Task
> > > >> State. On a retry, or after a worker crash, items that already
> > finished
> > > are
> > > >> not run again; only the items that failed or were in flight are.
> > > >> - The page now documents task_concurrency, the setting that controls
> > how
> > > >> many items run at once inside one Task Instance.
> > > >> - The page now lists the current limits: deferrable operators,
> > > >> reschedule-mode sensors and TriggerDagRunOperator are not supported
> > > inside
> > > >> .iterate() yet (deferrable support is planned in a separate PR);
> > > >> execution_timeout applies per item only to async operators;
> > > >> multiple_outputs is ignored; and pools count the Task Instance
> rather
> > > than
> > > >> its items.
> > > >>
> > > >> The previous discussion threads can be found here:
> > > >>
> > > >> https://lists.apache.org/thread/ztnfsqolow4v1zsv4pkpnxc1fk0hbf2p
> > > >> https://lists.apache.org/thread/tr3x4sx55k0fk2coq28xs2648m1p65cc
> > > >>
> > > >> The naming of the spread primitive:
> > > >>
> > > >> https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2
> > > >> (discussion)
> > > >> https://lists.apache.org/thread/rrc9bmhh5dcpyj4rcpjr2lj1q8vjdoxp
> > (vote)
> > > >>
> > > >> Thank you again to everyone who reviewed the AIP and took part in
> the
> > > >> discussions. A special thank you to Kaxil for the many detailed
> > reviews,
> > > >> and to TP, Jens, Jarek and Ash for their comments. Thanks as well to
> > > >> Daniel, Jedidiah and Amogh for their more recent reviews on the AIP
> > > page,
> > > >> and to everyone who voted on the naming of the spread primitive.
> > > >>
> > > >> The vote will remain open for 5 days and will close on Wednesday, 7
> > > >> October 2026, at 00:00 GMT.
> > > >>
> > > >> Everyone is encouraged to vote, although only votes from PMC members
> > and
> > > >> committers are considered binding.
> > > >>
> > > >> Please vote accordingly:
> > > >>
> > > >> [ ] +1 Approve
> > > >> [ ] +0 no opinion
> > > >> [ ] -1 disapprove with the reason
> > > >>
> > > >> Please consider this my +1 (binding).
> > > >>
> > > >> Regards,
> > > >> David
> > > >>
> > > >> General (Internal Property)
> > > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>
Re: [VOTE] AIP-104 Iterable Tasks (IT) and Task Spreading (TS)
Posted by Kaxil Naik <ka...@gmail.com>.
+1 on the features. No opinion on naming
On Mon, 5 Oct 2026 at 18:08, Zach Gottesman via dev <de...@airflow.apache.org>
wrote:
> +1 (non-binding). Coincidentally I was just discussing the need for
> something similar to this in a meeting earlier today. Would love to see it
> as part of the Airflow public interface.
>
> On Mon, Oct 5, 2026 at 9:36 AM Jens Scheffler via dev <
> dev@airflow.apache.org> wrote:
>
> > +1
> >
> > On 05.10.26 11:45, Shahar Epstein wrote:
> > > +1 (binding), thanks for leading this effort!
> > >
> > >
> > > Shahar
> > >
> > > On Thu, Oct 1, 2026, 11:00 Blain David <da...@infrabel.be>
> wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> I would like to restart the vote on AIP-104, now titled Iterable Tasks
> > >> (IT) and Task Spreading (TS):
> > >>
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
> > >>
> > >> The previous vote, under the name "Task Iteration and Dynamic Task
> > >> Batching", ran during the summer and did not get much engagement:
> > >>
> > >> https://lists.apache.org/thread/x5bd26zfqkkhp7sx7nb0crjlsctf2z5j
> > >>
> > >> The AIP has changed since then, so please vote on the current page
> > rather
> > >> than on what you may remember:
> > >>
> > >> - Renamed "Task Iteration (TI)" to "Iterable Tasks (IT)", as TI is
> > already
> > >> widely used for Task Instance (as remarked by Daniel).
> > >> - Renamed "Dynamic Task Batching (DTB)" to "Task Spreading (TS)". The
> > >> primitive is now .spread(across=N) instead of .batch(size=N), as
> "size"
> > >> read like a chunk length while it is the number of Task Instances.
> Items
> > >> are distributed round robin over those N Task Instances. The new name
> > was
> > >> discussed and voted on in separate threads (linked below).
> > >> - Iterable Tasks now checkpoint progress per item through AIP-103 Task
> > >> State. On a retry, or after a worker crash, items that already
> finished
> > are
> > >> not run again; only the items that failed or were in flight are.
> > >> - The page now documents task_concurrency, the setting that controls
> how
> > >> many items run at once inside one Task Instance.
> > >> - The page now lists the current limits: deferrable operators,
> > >> reschedule-mode sensors and TriggerDagRunOperator are not supported
> > inside
> > >> .iterate() yet (deferrable support is planned in a separate PR);
> > >> execution_timeout applies per item only to async operators;
> > >> multiple_outputs is ignored; and pools count the Task Instance rather
> > than
> > >> its items.
> > >>
> > >> The previous discussion threads can be found here:
> > >>
> > >> https://lists.apache.org/thread/ztnfsqolow4v1zsv4pkpnxc1fk0hbf2p
> > >> https://lists.apache.org/thread/tr3x4sx55k0fk2coq28xs2648m1p65cc
> > >>
> > >> The naming of the spread primitive:
> > >>
> > >> https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2
> > >> (discussion)
> > >> https://lists.apache.org/thread/rrc9bmhh5dcpyj4rcpjr2lj1q8vjdoxp
> (vote)
> > >>
> > >> Thank you again to everyone who reviewed the AIP and took part in the
> > >> discussions. A special thank you to Kaxil for the many detailed
> reviews,
> > >> and to TP, Jens, Jarek and Ash for their comments. Thanks as well to
> > >> Daniel, Jedidiah and Amogh for their more recent reviews on the AIP
> > page,
> > >> and to everyone who voted on the naming of the spread primitive.
> > >>
> > >> The vote will remain open for 5 days and will close on Wednesday, 7
> > >> October 2026, at 00:00 GMT.
> > >>
> > >> Everyone is encouraged to vote, although only votes from PMC members
> and
> > >> committers are considered binding.
> > >>
> > >> Please vote accordingly:
> > >>
> > >> [ ] +1 Approve
> > >> [ ] +0 no opinion
> > >> [ ] -1 disapprove with the reason
> > >>
> > >> Please consider this my +1 (binding).
> > >>
> > >> Regards,
> > >> David
> > >>
> > >> General (Internal Property)
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: [VOTE] AIP-104 Iterable Tasks (IT) and Task Spreading (TS)
Posted by Zach Gottesman via dev <de...@airflow.apache.org>.
+1 (non-binding). Coincidentally I was just discussing the need for
something similar to this in a meeting earlier today. Would love to see it
as part of the Airflow public interface.
On Mon, Oct 5, 2026 at 9:36 AM Jens Scheffler via dev <
dev@airflow.apache.org> wrote:
> +1
>
> On 05.10.26 11:45, Shahar Epstein wrote:
> > +1 (binding), thanks for leading this effort!
> >
> >
> > Shahar
> >
> > On Thu, Oct 1, 2026, 11:00 Blain David <da...@infrabel.be> wrote:
> >
> >> Hi everyone,
> >>
> >> I would like to restart the vote on AIP-104, now titled Iterable Tasks
> >> (IT) and Task Spreading (TS):
> >>
> >>
> >>
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
> >>
> >> The previous vote, under the name "Task Iteration and Dynamic Task
> >> Batching", ran during the summer and did not get much engagement:
> >>
> >> https://lists.apache.org/thread/x5bd26zfqkkhp7sx7nb0crjlsctf2z5j
> >>
> >> The AIP has changed since then, so please vote on the current page
> rather
> >> than on what you may remember:
> >>
> >> - Renamed "Task Iteration (TI)" to "Iterable Tasks (IT)", as TI is
> already
> >> widely used for Task Instance (as remarked by Daniel).
> >> - Renamed "Dynamic Task Batching (DTB)" to "Task Spreading (TS)". The
> >> primitive is now .spread(across=N) instead of .batch(size=N), as "size"
> >> read like a chunk length while it is the number of Task Instances. Items
> >> are distributed round robin over those N Task Instances. The new name
> was
> >> discussed and voted on in separate threads (linked below).
> >> - Iterable Tasks now checkpoint progress per item through AIP-103 Task
> >> State. On a retry, or after a worker crash, items that already finished
> are
> >> not run again; only the items that failed or were in flight are.
> >> - The page now documents task_concurrency, the setting that controls how
> >> many items run at once inside one Task Instance.
> >> - The page now lists the current limits: deferrable operators,
> >> reschedule-mode sensors and TriggerDagRunOperator are not supported
> inside
> >> .iterate() yet (deferrable support is planned in a separate PR);
> >> execution_timeout applies per item only to async operators;
> >> multiple_outputs is ignored; and pools count the Task Instance rather
> than
> >> its items.
> >>
> >> The previous discussion threads can be found here:
> >>
> >> https://lists.apache.org/thread/ztnfsqolow4v1zsv4pkpnxc1fk0hbf2p
> >> https://lists.apache.org/thread/tr3x4sx55k0fk2coq28xs2648m1p65cc
> >>
> >> The naming of the spread primitive:
> >>
> >> https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2
> >> (discussion)
> >> https://lists.apache.org/thread/rrc9bmhh5dcpyj4rcpjr2lj1q8vjdoxp (vote)
> >>
> >> Thank you again to everyone who reviewed the AIP and took part in the
> >> discussions. A special thank you to Kaxil for the many detailed reviews,
> >> and to TP, Jens, Jarek and Ash for their comments. Thanks as well to
> >> Daniel, Jedidiah and Amogh for their more recent reviews on the AIP
> page,
> >> and to everyone who voted on the naming of the spread primitive.
> >>
> >> The vote will remain open for 5 days and will close on Wednesday, 7
> >> October 2026, at 00:00 GMT.
> >>
> >> Everyone is encouraged to vote, although only votes from PMC members and
> >> committers are considered binding.
> >>
> >> Please vote accordingly:
> >>
> >> [ ] +1 Approve
> >> [ ] +0 no opinion
> >> [ ] -1 disapprove with the reason
> >>
> >> Please consider this my +1 (binding).
> >>
> >> Regards,
> >> David
> >>
> >> General (Internal Property)
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [VOTE] AIP-104 Iterable Tasks (IT) and Task Spreading (TS)
Posted by Jens Scheffler via dev <de...@airflow.apache.org>.
+1
On 05.10.26 11:45, Shahar Epstein wrote:
> +1 (binding), thanks for leading this effort!
>
>
> Shahar
>
> On Thu, Oct 1, 2026, 11:00 Blain David <da...@infrabel.be> wrote:
>
>> Hi everyone,
>>
>> I would like to restart the vote on AIP-104, now titled Iterable Tasks
>> (IT) and Task Spreading (TS):
>>
>>
>> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
>>
>> The previous vote, under the name "Task Iteration and Dynamic Task
>> Batching", ran during the summer and did not get much engagement:
>>
>> https://lists.apache.org/thread/x5bd26zfqkkhp7sx7nb0crjlsctf2z5j
>>
>> The AIP has changed since then, so please vote on the current page rather
>> than on what you may remember:
>>
>> - Renamed "Task Iteration (TI)" to "Iterable Tasks (IT)", as TI is already
>> widely used for Task Instance (as remarked by Daniel).
>> - Renamed "Dynamic Task Batching (DTB)" to "Task Spreading (TS)". The
>> primitive is now .spread(across=N) instead of .batch(size=N), as "size"
>> read like a chunk length while it is the number of Task Instances. Items
>> are distributed round robin over those N Task Instances. The new name was
>> discussed and voted on in separate threads (linked below).
>> - Iterable Tasks now checkpoint progress per item through AIP-103 Task
>> State. On a retry, or after a worker crash, items that already finished are
>> not run again; only the items that failed or were in flight are.
>> - The page now documents task_concurrency, the setting that controls how
>> many items run at once inside one Task Instance.
>> - The page now lists the current limits: deferrable operators,
>> reschedule-mode sensors and TriggerDagRunOperator are not supported inside
>> .iterate() yet (deferrable support is planned in a separate PR);
>> execution_timeout applies per item only to async operators;
>> multiple_outputs is ignored; and pools count the Task Instance rather than
>> its items.
>>
>> The previous discussion threads can be found here:
>>
>> https://lists.apache.org/thread/ztnfsqolow4v1zsv4pkpnxc1fk0hbf2p
>> https://lists.apache.org/thread/tr3x4sx55k0fk2coq28xs2648m1p65cc
>>
>> The naming of the spread primitive:
>>
>> https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2
>> (discussion)
>> https://lists.apache.org/thread/rrc9bmhh5dcpyj4rcpjr2lj1q8vjdoxp (vote)
>>
>> Thank you again to everyone who reviewed the AIP and took part in the
>> discussions. A special thank you to Kaxil for the many detailed reviews,
>> and to TP, Jens, Jarek and Ash for their comments. Thanks as well to
>> Daniel, Jedidiah and Amogh for their more recent reviews on the AIP page,
>> and to everyone who voted on the naming of the spread primitive.
>>
>> The vote will remain open for 5 days and will close on Wednesday, 7
>> October 2026, at 00:00 GMT.
>>
>> Everyone is encouraged to vote, although only votes from PMC members and
>> committers are considered binding.
>>
>> Please vote accordingly:
>>
>> [ ] +1 Approve
>> [ ] +0 no opinion
>> [ ] -1 disapprove with the reason
>>
>> Please consider this my +1 (binding).
>>
>> Regards,
>> David
>>
>> General (Internal Property)
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [VOTE] AIP-104 Iterable Tasks (IT) and Task Spreading (TS)
Posted by Shahar Epstein <sh...@apache.org>.
+1 (binding), thanks for leading this effort!
Shahar
On Thu, Oct 1, 2026, 11:00 Blain David <da...@infrabel.be> wrote:
> Hi everyone,
>
> I would like to restart the vote on AIP-104, now titled Iterable Tasks
> (IT) and Task Spreading (TS):
>
>
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
>
> The previous vote, under the name "Task Iteration and Dynamic Task
> Batching", ran during the summer and did not get much engagement:
>
> https://lists.apache.org/thread/x5bd26zfqkkhp7sx7nb0crjlsctf2z5j
>
> The AIP has changed since then, so please vote on the current page rather
> than on what you may remember:
>
> - Renamed "Task Iteration (TI)" to "Iterable Tasks (IT)", as TI is already
> widely used for Task Instance (as remarked by Daniel).
> - Renamed "Dynamic Task Batching (DTB)" to "Task Spreading (TS)". The
> primitive is now .spread(across=N) instead of .batch(size=N), as "size"
> read like a chunk length while it is the number of Task Instances. Items
> are distributed round robin over those N Task Instances. The new name was
> discussed and voted on in separate threads (linked below).
> - Iterable Tasks now checkpoint progress per item through AIP-103 Task
> State. On a retry, or after a worker crash, items that already finished are
> not run again; only the items that failed or were in flight are.
> - The page now documents task_concurrency, the setting that controls how
> many items run at once inside one Task Instance.
> - The page now lists the current limits: deferrable operators,
> reschedule-mode sensors and TriggerDagRunOperator are not supported inside
> .iterate() yet (deferrable support is planned in a separate PR);
> execution_timeout applies per item only to async operators;
> multiple_outputs is ignored; and pools count the Task Instance rather than
> its items.
>
> The previous discussion threads can be found here:
>
> https://lists.apache.org/thread/ztnfsqolow4v1zsv4pkpnxc1fk0hbf2p
> https://lists.apache.org/thread/tr3x4sx55k0fk2coq28xs2648m1p65cc
>
> The naming of the spread primitive:
>
> https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2
> (discussion)
> https://lists.apache.org/thread/rrc9bmhh5dcpyj4rcpjr2lj1q8vjdoxp (vote)
>
> Thank you again to everyone who reviewed the AIP and took part in the
> discussions. A special thank you to Kaxil for the many detailed reviews,
> and to TP, Jens, Jarek and Ash for their comments. Thanks as well to
> Daniel, Jedidiah and Amogh for their more recent reviews on the AIP page,
> and to everyone who voted on the naming of the spread primitive.
>
> The vote will remain open for 5 days and will close on Wednesday, 7
> October 2026, at 00:00 GMT.
>
> Everyone is encouraged to vote, although only votes from PMC members and
> committers are considered binding.
>
> Please vote accordingly:
>
> [ ] +1 Approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
> Please consider this my +1 (binding).
>
> Regards,
> David
>
> General (Internal Property)
>