You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Blain David <da...@infrabel.be> on 2026/09/17 09:24:05 UTC
[DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable Tasks and Dynamic Task Batching)
Hi all,
While reviewing the AIP-104 PR (https://github.com/apache/airflow/pull/62922), Amogh raised a naming
concern about the Dynamic Task Batching API that I would like to settle here before the PR merges,
since renaming afterwards would be a breaking change.
## What `.batch(size=N)` does today
`task.batch(size=17).iterate(url=urls)` creates 17 mapped task instances through Dynamic Task
Mapping, and each of them iterates over its share of `urls` in a single task instance using
Iterable Tasks. The items are distributed round robin: item `i` goes to task instance `i % 17`.
So `size` is the number of task instances, not a chunk length, and the task instances do not
receive contiguous slices of the input.
The round robin scheme is deliberate. The number of task instances has to be fixed before the
underlying iterable is consumed, because the scheduler needs the mapping cardinality up front.
With round robin that number is simply `size`. Contiguous chunking, the way `itertools.batched`
works, would need `ceil(len(items) / size)` task instances, which is unknowable until a possibly
unbounded or paginated iterable has been fully drained. That would defeat the purpose of iterating
lazily.
## The concern
The name reads like `itertools.batched(iterable, size)`, where `size` is the chunk length and the
number of chunks is derived from it. Ours is the opposite: the number of task instances is given,
and the per-instance share is derived. The class docstring and the Task SDK docs currently spend a
full paragraph undoing that first impression. A name that carries the meaning directly would not
need it.
## Options
1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but the mismatch with
`itertools.batched` stays and every new reader has to be corrected.
2. Keep the method, rename the parameter: `.batch(count=N)` or `.batch(tasks=N)`. Keeps the
"Dynamic Task Batching" vocabulary from the AIP, and `count` or `tasks` says what the number
is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17 tasks, then iterate".
3. Rename the method as well, to something that describes partitioning rather than chunking, for
example `.partition(count=N)`, `.shard(count=N)` or `.spread(across=N)`. Most descriptive, but
drifts from the AIP's own terminology, and "shard" and "partition" carry data-engineering
connotations that may suggest contiguous ranges just as much as "batch" does.
My preference is option 2 with `count=`. It keeps the API shape and the AIP terminology, fixes
the misleading part, and costs nothing now because the feature is unreleased. `size` would be
kept as a rejected alias raising a clear error rather than silently accepted, so nobody copies
the old spelling from an early draft.
Unless there are objections or a better name comes up, I will treat this as lazy consensus in
72 hours and update the PR, the docs and the AIP page accordingly.
Thanks,
David
General (Internal Property)
Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable Tasks and Dynamic Task Batching)
Posted by Ash Berlin-Taylor <as...@apache.org>.
That’s exactly the point though. This AIP/PR as currently proposed does not operate the same as itertools.batchhed, so naming it the same is confusing.
From the original post:
> Ours is the opposite [to itertools.batched]: the number of task instances is given, and the per-instance share is derived
> On 18 Sep 2026, at 20:48, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> How about we follow Python 3.12 convention:
>
> itertools.batched(it, n)
>
> The ecosystem didn't fully converge on one spelling. more_itertools.batched
> is now just an alias for the stdlib function, but chunked (any iterable →
> lists) and sliced (sequences → real slices, type-preserving) are still
> separate functions with separate semantics, because tuples aren't always
> what you want.
>
> Worth noting for your original question: the stdlib settled on positional n,
> not a keyword. So batched(it, 100), not batched(it, size=100) — and the
> parameter-order question was considered explicitly rather than falling out
> by accident. If you're designing a .batch(), that's the precedent to match
> or consciously break.
>
>
> J.
>
>
> On Fri, Sep 18, 2026 at 7:50 PM Constance Martineau via dev <
> dev@airflow.apache.org> wrote:
>
>> My preference is option 3. Of the names on the table, I'd rank them roughly
>> `spread`, then `batch`, then `shard` with `partition` last. `spread` works
>> best in my opinion because the preposition makes the intended behaviour
>> really obvious. "spread across 17" can't be misread as "17 each", which is
>> the risk with `batch(count=N)`. Agree that `shard` and `partition` has
>> specific connotations, and Partitions are now a feature in Airflow so if we
>> chose that we'd be introducing a name class.
>>
>> If the consensus ends up being option 2, I'd pick `tasks=` over `count=`,
>> since `count` doesn't say count of what and can be misinterpreted as
>> per-instance item count. Either way though, I'd make this keyword-only
>> (`spread(17)` is just as ambigious as `batch(17)`) and agreed that we
>> should raise a clear error if someone passes `size=`.
>>
>> Whichever option wins, I think you still need to make the round-robin
>> distribution very clear in the docs, along with the reasoning, since it
>> might not be immediately obvious to Dag authors why we chose round-robin.
>>
>> Constance
>>
>> On Fri, Sep 18, 2026 at 11:22 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>>
>>> I’d like more time to think about this please.
>>>
>>> -ash
>>>
>>>> On 17 Sep 2026, at 10:24, Blain David <da...@infrabel.be> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> While reviewing the AIP-104 PR (
>>> https://github.com/apache/airflow/pull/62922), Amogh raised a naming
>>>> concern about the Dynamic Task Batching API that I would like to settle
>>> here before the PR merges,
>>>> since renaming afterwards would be a breaking change.
>>>>
>>>> ## What `.batch(size=N)` does today
>>>>
>>>> `task.batch(size=17).iterate(url=urls)` creates 17 mapped task
>> instances
>>> through Dynamic Task
>>>> Mapping, and each of them iterates over its share of `urls` in a single
>>> task instance using
>>>> Iterable Tasks. The items are distributed round robin: item `i` goes to
>>> task instance `i % 17`.
>>>> So `size` is the number of task instances, not a chunk length, and the
>>> task instances do not
>>>> receive contiguous slices of the input.
>>>>
>>>> The round robin scheme is deliberate. The number of task instances has
>>> to be fixed before the
>>>> underlying iterable is consumed, because the scheduler needs the
>> mapping
>>> cardinality up front.
>>>> With round robin that number is simply `size`. Contiguous chunking, the
>>> way `itertools.batched`
>>>> works, would need `ceil(len(items) / size)` task instances, which is
>>> unknowable until a possibly
>>>> unbounded or paginated iterable has been fully drained. That would
>>> defeat the purpose of iterating
>>>> lazily.
>>>>
>>>> ## The concern
>>>>
>>>> The name reads like `itertools.batched(iterable, size)`, where `size`
>> is
>>> the chunk length and the
>>>> number of chunks is derived from it. Ours is the opposite: the number
>> of
>>> task instances is given,
>>>> and the per-instance share is derived. The class docstring and the Task
>>> SDK docs currently spend a
>>>> full paragraph undoing that first impression. A name that carries the
>>> meaning directly would not
>>>> need it.
>>>>
>>>> ## Options
>>>>
>>>> 1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but
>>> the mismatch with
>>>> `itertools.batched` stays and every new reader has to be corrected.
>>>>
>>>> 2. Keep the method, rename the parameter: `.batch(count=N)` or
>>> `.batch(tasks=N)`. Keeps the
>>>> "Dynamic Task Batching" vocabulary from the AIP, and `count` or
>>> `tasks` says what the number
>>>> is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17
>>> tasks, then iterate".
>>>>
>>>> 3. Rename the method as well, to something that describes partitioning
>>> rather than chunking, for
>>>> example `.partition(count=N)`, `.shard(count=N)` or
>>> `.spread(across=N)`. Most descriptive, but
>>>> drifts from the AIP's own terminology, and "shard" and "partition"
>>> carry data-engineering
>>>> connotations that may suggest contiguous ranges just as much as
>>> "batch" does.
>>>>
>>>> My preference is option 2 with `count=`. It keeps the API shape and the
>>> AIP terminology, fixes
>>>> the misleading part, and costs nothing now because the feature is
>>> unreleased. `size` would be
>>>> kept as a rejected alias raising a clear error rather than silently
>>> accepted, so nobody copies
>>>> the old spelling from an early draft.
>>>>
>>>> Unless there are objections or a better name comes up, I will treat
>> this
>>> as lazy consensus in
>>>> 72 hours and update the PR, the docs and the AIP page accordingly.
>>>>
>>>> Thanks,
>>>> David
>>>>
>>>> General (Internal Property)
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable Tasks and Dynamic Task Batching)
Posted by Jarek Potiuk <ja...@potiuk.com>.
How about we follow Python 3.12 convention:
itertools.batched(it, n)
The ecosystem didn't fully converge on one spelling. more_itertools.batched
is now just an alias for the stdlib function, but chunked (any iterable →
lists) and sliced (sequences → real slices, type-preserving) are still
separate functions with separate semantics, because tuples aren't always
what you want.
Worth noting for your original question: the stdlib settled on positional n,
not a keyword. So batched(it, 100), not batched(it, size=100) — and the
parameter-order question was considered explicitly rather than falling out
by accident. If you're designing a .batch(), that's the precedent to match
or consciously break.
J.
On Fri, Sep 18, 2026 at 7:50 PM Constance Martineau via dev <
dev@airflow.apache.org> wrote:
> My preference is option 3. Of the names on the table, I'd rank them roughly
> `spread`, then `batch`, then `shard` with `partition` last. `spread` works
> best in my opinion because the preposition makes the intended behaviour
> really obvious. "spread across 17" can't be misread as "17 each", which is
> the risk with `batch(count=N)`. Agree that `shard` and `partition` has
> specific connotations, and Partitions are now a feature in Airflow so if we
> chose that we'd be introducing a name class.
>
> If the consensus ends up being option 2, I'd pick `tasks=` over `count=`,
> since `count` doesn't say count of what and can be misinterpreted as
> per-instance item count. Either way though, I'd make this keyword-only
> (`spread(17)` is just as ambigious as `batch(17)`) and agreed that we
> should raise a clear error if someone passes `size=`.
>
> Whichever option wins, I think you still need to make the round-robin
> distribution very clear in the docs, along with the reasoning, since it
> might not be immediately obvious to Dag authors why we chose round-robin.
>
> Constance
>
> On Fri, Sep 18, 2026 at 11:22 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > I’d like more time to think about this please.
> >
> > -ash
> >
> > > On 17 Sep 2026, at 10:24, Blain David <da...@infrabel.be> wrote:
> > >
> > > Hi all,
> > >
> > > While reviewing the AIP-104 PR (
> > https://github.com/apache/airflow/pull/62922), Amogh raised a naming
> > > concern about the Dynamic Task Batching API that I would like to settle
> > here before the PR merges,
> > > since renaming afterwards would be a breaking change.
> > >
> > > ## What `.batch(size=N)` does today
> > >
> > > `task.batch(size=17).iterate(url=urls)` creates 17 mapped task
> instances
> > through Dynamic Task
> > > Mapping, and each of them iterates over its share of `urls` in a single
> > task instance using
> > > Iterable Tasks. The items are distributed round robin: item `i` goes to
> > task instance `i % 17`.
> > > So `size` is the number of task instances, not a chunk length, and the
> > task instances do not
> > > receive contiguous slices of the input.
> > >
> > > The round robin scheme is deliberate. The number of task instances has
> > to be fixed before the
> > > underlying iterable is consumed, because the scheduler needs the
> mapping
> > cardinality up front.
> > > With round robin that number is simply `size`. Contiguous chunking, the
> > way `itertools.batched`
> > > works, would need `ceil(len(items) / size)` task instances, which is
> > unknowable until a possibly
> > > unbounded or paginated iterable has been fully drained. That would
> > defeat the purpose of iterating
> > > lazily.
> > >
> > > ## The concern
> > >
> > > The name reads like `itertools.batched(iterable, size)`, where `size`
> is
> > the chunk length and the
> > > number of chunks is derived from it. Ours is the opposite: the number
> of
> > task instances is given,
> > > and the per-instance share is derived. The class docstring and the Task
> > SDK docs currently spend a
> > > full paragraph undoing that first impression. A name that carries the
> > meaning directly would not
> > > need it.
> > >
> > > ## Options
> > >
> > > 1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but
> > the mismatch with
> > > `itertools.batched` stays and every new reader has to be corrected.
> > >
> > > 2. Keep the method, rename the parameter: `.batch(count=N)` or
> > `.batch(tasks=N)`. Keeps the
> > > "Dynamic Task Batching" vocabulary from the AIP, and `count` or
> > `tasks` says what the number
> > > is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17
> > tasks, then iterate".
> > >
> > > 3. Rename the method as well, to something that describes partitioning
> > rather than chunking, for
> > > example `.partition(count=N)`, `.shard(count=N)` or
> > `.spread(across=N)`. Most descriptive, but
> > > drifts from the AIP's own terminology, and "shard" and "partition"
> > carry data-engineering
> > > connotations that may suggest contiguous ranges just as much as
> > "batch" does.
> > >
> > > My preference is option 2 with `count=`. It keeps the API shape and the
> > AIP terminology, fixes
> > > the misleading part, and costs nothing now because the feature is
> > unreleased. `size` would be
> > > kept as a rejected alias raising a clear error rather than silently
> > accepted, so nobody copies
> > > the old spelling from an early draft.
> > >
> > > Unless there are objections or a better name comes up, I will treat
> this
> > as lazy consensus in
> > > 72 hours and update the PR, the docs and the AIP page accordingly.
> > >
> > > Thanks,
> > > David
> > >
> > > General (Internal Property)
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable Tasks and Dynamic Task Batching)
Posted by Constance Martineau via dev <de...@airflow.apache.org>.
My preference is option 3. Of the names on the table, I'd rank them roughly
`spread`, then `batch`, then `shard` with `partition` last. `spread` works
best in my opinion because the preposition makes the intended behaviour
really obvious. "spread across 17" can't be misread as "17 each", which is
the risk with `batch(count=N)`. Agree that `shard` and `partition` has
specific connotations, and Partitions are now a feature in Airflow so if we
chose that we'd be introducing a name class.
If the consensus ends up being option 2, I'd pick `tasks=` over `count=`,
since `count` doesn't say count of what and can be misinterpreted as
per-instance item count. Either way though, I'd make this keyword-only
(`spread(17)` is just as ambigious as `batch(17)`) and agreed that we
should raise a clear error if someone passes `size=`.
Whichever option wins, I think you still need to make the round-robin
distribution very clear in the docs, along with the reasoning, since it
might not be immediately obvious to Dag authors why we chose round-robin.
Constance
On Fri, Sep 18, 2026 at 11:22 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> I’d like more time to think about this please.
>
> -ash
>
> > On 17 Sep 2026, at 10:24, Blain David <da...@infrabel.be> wrote:
> >
> > Hi all,
> >
> > While reviewing the AIP-104 PR (
> https://github.com/apache/airflow/pull/62922), Amogh raised a naming
> > concern about the Dynamic Task Batching API that I would like to settle
> here before the PR merges,
> > since renaming afterwards would be a breaking change.
> >
> > ## What `.batch(size=N)` does today
> >
> > `task.batch(size=17).iterate(url=urls)` creates 17 mapped task instances
> through Dynamic Task
> > Mapping, and each of them iterates over its share of `urls` in a single
> task instance using
> > Iterable Tasks. The items are distributed round robin: item `i` goes to
> task instance `i % 17`.
> > So `size` is the number of task instances, not a chunk length, and the
> task instances do not
> > receive contiguous slices of the input.
> >
> > The round robin scheme is deliberate. The number of task instances has
> to be fixed before the
> > underlying iterable is consumed, because the scheduler needs the mapping
> cardinality up front.
> > With round robin that number is simply `size`. Contiguous chunking, the
> way `itertools.batched`
> > works, would need `ceil(len(items) / size)` task instances, which is
> unknowable until a possibly
> > unbounded or paginated iterable has been fully drained. That would
> defeat the purpose of iterating
> > lazily.
> >
> > ## The concern
> >
> > The name reads like `itertools.batched(iterable, size)`, where `size` is
> the chunk length and the
> > number of chunks is derived from it. Ours is the opposite: the number of
> task instances is given,
> > and the per-instance share is derived. The class docstring and the Task
> SDK docs currently spend a
> > full paragraph undoing that first impression. A name that carries the
> meaning directly would not
> > need it.
> >
> > ## Options
> >
> > 1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but
> the mismatch with
> > `itertools.batched` stays and every new reader has to be corrected.
> >
> > 2. Keep the method, rename the parameter: `.batch(count=N)` or
> `.batch(tasks=N)`. Keeps the
> > "Dynamic Task Batching" vocabulary from the AIP, and `count` or
> `tasks` says what the number
> > is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17
> tasks, then iterate".
> >
> > 3. Rename the method as well, to something that describes partitioning
> rather than chunking, for
> > example `.partition(count=N)`, `.shard(count=N)` or
> `.spread(across=N)`. Most descriptive, but
> > drifts from the AIP's own terminology, and "shard" and "partition"
> carry data-engineering
> > connotations that may suggest contiguous ranges just as much as
> "batch" does.
> >
> > My preference is option 2 with `count=`. It keeps the API shape and the
> AIP terminology, fixes
> > the misleading part, and costs nothing now because the feature is
> unreleased. `size` would be
> > kept as a rejected alias raising a clear error rather than silently
> accepted, so nobody copies
> > the old spelling from an early draft.
> >
> > Unless there are objections or a better name comes up, I will treat this
> as lazy consensus in
> > 72 hours and update the PR, the docs and the AIP page accordingly.
> >
> > Thanks,
> > David
> >
> > General (Internal Property)
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable Tasks and Dynamic Task Batching)
Posted by Ash Berlin-Taylor <as...@apache.org>.
I’d like more time to think about this please.
-ash
> On 17 Sep 2026, at 10:24, Blain David <da...@infrabel.be> wrote:
>
> Hi all,
>
> While reviewing the AIP-104 PR (https://github.com/apache/airflow/pull/62922), Amogh raised a naming
> concern about the Dynamic Task Batching API that I would like to settle here before the PR merges,
> since renaming afterwards would be a breaking change.
>
> ## What `.batch(size=N)` does today
>
> `task.batch(size=17).iterate(url=urls)` creates 17 mapped task instances through Dynamic Task
> Mapping, and each of them iterates over its share of `urls` in a single task instance using
> Iterable Tasks. The items are distributed round robin: item `i` goes to task instance `i % 17`.
> So `size` is the number of task instances, not a chunk length, and the task instances do not
> receive contiguous slices of the input.
>
> The round robin scheme is deliberate. The number of task instances has to be fixed before the
> underlying iterable is consumed, because the scheduler needs the mapping cardinality up front.
> With round robin that number is simply `size`. Contiguous chunking, the way `itertools.batched`
> works, would need `ceil(len(items) / size)` task instances, which is unknowable until a possibly
> unbounded or paginated iterable has been fully drained. That would defeat the purpose of iterating
> lazily.
>
> ## The concern
>
> The name reads like `itertools.batched(iterable, size)`, where `size` is the chunk length and the
> number of chunks is derived from it. Ours is the opposite: the number of task instances is given,
> and the per-instance share is derived. The class docstring and the Task SDK docs currently spend a
> full paragraph undoing that first impression. A name that carries the meaning directly would not
> need it.
>
> ## Options
>
> 1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but the mismatch with
> `itertools.batched` stays and every new reader has to be corrected.
>
> 2. Keep the method, rename the parameter: `.batch(count=N)` or `.batch(tasks=N)`. Keeps the
> "Dynamic Task Batching" vocabulary from the AIP, and `count` or `tasks` says what the number
> is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17 tasks, then iterate".
>
> 3. Rename the method as well, to something that describes partitioning rather than chunking, for
> example `.partition(count=N)`, `.shard(count=N)` or `.spread(across=N)`. Most descriptive, but
> drifts from the AIP's own terminology, and "shard" and "partition" carry data-engineering
> connotations that may suggest contiguous ranges just as much as "batch" does.
>
> My preference is option 2 with `count=`. It keeps the API shape and the AIP terminology, fixes
> the misleading part, and costs nothing now because the feature is unreleased. `size` would be
> kept as a rejected alias raising a clear error rather than silently accepted, so nobody copies
> the old spelling from an early draft.
>
> Unless there are objections or a better name comes up, I will treat this as lazy consensus in
> 72 hours and update the PR, the docs and the AIP page accordingly.
>
> Thanks,
> David
>
> General (Internal Property)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable Tasks and Dynamic Task Batching)
Posted by Blain David <da...@infrabel.be>.
Would .spread(accross=n) then be the name to go with as Shahar and Constance also suggested?
________________________________
From: Shahar Epstein <sh...@apache.org>
Sent: Saturday, September 19, 2026 10:37
To: dev@airflow.apache.org <de...@airflow.apache.org>
Subject: Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable Tasks and Dynamic Task Batching)
EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar abuse@infrabel.be<ma...@infrabel.be>.
I'd go with .spread(...), as both batch(...) and shard(...) are already
semantically overloaded.
Shahar
On Thu, Sep 17, 2026 at 12:26 PM Blain David <da...@infrabel.be>
wrote:
> Hi all,
>
> While reviewing the AIP-104 PR (
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F62922&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cfb61e6c745c34a77080408df1629466b%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639254038688014321%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=S4qPFkVdn2y1F%2FtLRWhlkqwazsOUbNVrPEi%2FYuHoN7w%3D&reserved=0)<https://github.com/apache/airflow/pull/62922>, Amogh raised a naming
> concern about the Dynamic Task Batching API that I would like to settle
> here before the PR merges,
> since renaming afterwards would be a breaking change.
>
> ## What `.batch(size=N)` does today
>
> `task.batch(size=17).iterate(url=urls)` creates 17 mapped task instances
> through Dynamic Task
> Mapping, and each of them iterates over its share of `urls` in a single
> task instance using
> Iterable Tasks. The items are distributed round robin: item `i` goes to
> task instance `i % 17`.
> So `size` is the number of task instances, not a chunk length, and the
> task instances do not
> receive contiguous slices of the input.
>
> The round robin scheme is deliberate. The number of task instances has to
> be fixed before the
> underlying iterable is consumed, because the scheduler needs the mapping
> cardinality up front.
> With round robin that number is simply `size`. Contiguous chunking, the
> way `itertools.batched`
> works, would need `ceil(len(items) / size)` task instances, which is
> unknowable until a possibly
> unbounded or paginated iterable has been fully drained. That would defeat
> the purpose of iterating
> lazily.
>
> ## The concern
>
> The name reads like `itertools.batched(iterable, size)`, where `size` is
> the chunk length and the
> number of chunks is derived from it. Ours is the opposite: the number of
> task instances is given,
> and the per-instance share is derived. The class docstring and the Task
> SDK docs currently spend a
> full paragraph undoing that first impression. A name that carries the
> meaning directly would not
> need it.
>
> ## Options
>
> 1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but the
> mismatch with
> `itertools.batched` stays and every new reader has to be corrected.
>
> 2. Keep the method, rename the parameter: `.batch(count=N)` or
> `.batch(tasks=N)`. Keeps the
> "Dynamic Task Batching" vocabulary from the AIP, and `count` or `tasks`
> says what the number
> is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17
> tasks, then iterate".
>
> 3. Rename the method as well, to something that describes partitioning
> rather than chunking, for
> example `.partition(count=N)`, `.shard(count=N)` or
> `.spread(across=N)`. Most descriptive, but
> drifts from the AIP's own terminology, and "shard" and "partition"
> carry data-engineering
> connotations that may suggest contiguous ranges just as much as "batch"
> does.
>
> My preference is option 2 with `count=`. It keeps the API shape and the
> AIP terminology, fixes
> the misleading part, and costs nothing now because the feature is
> unreleased. `size` would be
> kept as a rejected alias raising a clear error rather than silently
> accepted, so nobody copies
> the old spelling from an early draft.
>
> Unless there are objections or a better name comes up, I will treat this
> as lazy consensus in
> 72 hours and update the PR, the docs and the AIP page accordingly.
>
> Thanks,
> David
>
> General (Internal Property)
>
General (Internal Property)
Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable Tasks and Dynamic Task Batching)
Posted by Jarek Potiuk <ja...@potiuk.com>.
Yeah. After the discussion, spread is a clear winner for me.
On Thu, Sep 24, 2026 at 3:13 PM constance.astronomer.io via dev <
dev@airflow.apache.org> wrote:
> +1 to keeping the feature name as is.
>
> For the function name, my vote is still some_task.spread(across=N), but
> between parallelize and distribute it’s just a matter of personal
> preference. It makes the behaviour really obvious, sounds more natural and
> is more readable (to me) and doesn’t imply that all mapped tasks will
> literally run in parallel at the exact same time.
>
> -1 to _(across_n_tasks=N). It’s unnecessary with the verbs in the running.
>
> > On Sep 24, 2026, at 6:20 AM, Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> > Sooo I was throwing some ideas around with David in a DM, and here are
> the ones I came up with
> >
> > some_task.batch(num_tasks=5)?
> > some_task.batch(across=5) ?
> > some_task.spread(across=5) ?
> > some_task.parallelise(across=5)?
> > some_task.distribute(across=5)?
> > some_task.distribute(across_n_tasks=5)?
> >
> > I think I like the `.parallelise(across=n)` the most? It’s not perfect I
> admit.
> >
> > Also (perhaps oddly) I think even with that I think the feature should
> be called "Batched Mapped Task” even if the fn is .parallelise or
> .distribute — but I don’t think "Task Spreading” is very clear from a
> user-reading-the-docs PoV.
> >
> > -a
> >
> >> On 24 Sep 2026, at 09:51, Amogh Desai <am...@apache.org> wrote:
> >>
> >> Late to the party, but thanks, David.
> >>
> >> Out of all the options, I vote +1 for `.spread(across=N)`.
> >>
> >> -1 to unfold because, to me, it means generating a sequence from
> something,
> >> which describes
> >> creating items rather than distributing them.
> >>
> >> And regardless of the name, the round-robin distribution needs to be
> >> explicit in the SDK docs with the reasoning,
> >> as Constance said. The name fixes the "how many" confusion, not the
> "which
> >> items go where" one.
> >>
> >> Thanks & Regards,
> >> Amogh Desai
> >>
> >>
> >>> On Thu, Sep 24, 2026 at 1:48 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>
> >>> How about "scatter" or "unfold" :)?
> >>>
> >>> (just had a little conversation with Claude about it).
> >>>
> >>> On Wed, Sep 23, 2026 at 12:20 PM Blain David <da...@infrabel.be>
> >>> wrote:
> >>>
> >>>> I've updated the AIP-104 confluence page and choose the spread option
> and
> >>>> thus the Task Spreading instead of Batched Task Mapping.
> >>>>
> >>>> Does everyone agree with this change?
> >>>>
> >>>>
> >>>>
> >>>
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
> >>>>
> >>>> If so, then I will push those changes in the PR as well:
> >>>>
> >>>> https://github.com/apache/airflow/pull/62922
> >>>>
> >>>> General (Internal Property)
> >>>> ________________________________
> >>>> From: Shahar Epstein <sh...@apache.org>
> >>>> Sent: Saturday, September 19, 2026 10:37
> >>>> To: dev@airflow.apache.org <de...@airflow.apache.org>
> >>>> Subject: Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable
> >>>> Tasks and Dynamic Task Batching)
> >>>>
> >>>> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze
> >>>> niet vertrouwt, klik niet op een link of open geen bijlages. Bij
> twijfel,
> >>>> stuur deze e-mail als bijlage naar abuse@infrabel.be<mailto:
> >>>> abuse@infrabel.be>.
> >>>>
> >>>> I'd go with .spread(...), as both batch(...) and shard(...) are
> already
> >>>> semantically overloaded.
> >>>>
> >>>>
> >>>> Shahar
> >>>>
> >>>> On Thu, Sep 17, 2026 at 12:26 PM Blain David <david.blain@infrabel.be
> >
> >>>> wrote:
> >>>>
> >>>>> Hi all,
> >>>>>
> >>>>> While reviewing the AIP-104 PR (
> >>>>>
> >>>>
> >>>
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F62922&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cfb61e6c745c34a77080408df1629466b%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639254038688014321%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=S4qPFkVdn2y1F%2FtLRWhlkqwazsOUbNVrPEi%2FYuHoN7w%3D&reserved=0
> >>>> )<https://github.com/apache/airflow/pull/62922>, Amogh raised a
> naming
> >>>>> concern about the Dynamic Task Batching API that I would like to
> settle
> >>>>> here before the PR merges,
> >>>>> since renaming afterwards would be a breaking change.
> >>>>>
> >>>>> ## What `.batch(size=N)` does today
> >>>>>
> >>>>> `task.batch(size=17).iterate(url=urls)` creates 17 mapped task
> >>> instances
> >>>>> through Dynamic Task
> >>>>> Mapping, and each of them iterates over its share of `urls` in a
> single
> >>>>> task instance using
> >>>>> Iterable Tasks. The items are distributed round robin: item `i` goes
> to
> >>>>> task instance `i % 17`.
> >>>>> So `size` is the number of task instances, not a chunk length, and
> the
> >>>>> task instances do not
> >>>>> receive contiguous slices of the input.
> >>>>>
> >>>>> The round robin scheme is deliberate. The number of task instances
> has
> >>> to
> >>>>> be fixed before the
> >>>>> underlying iterable is consumed, because the scheduler needs the
> >>> mapping
> >>>>> cardinality up front.
> >>>>> With round robin that number is simply `size`. Contiguous chunking,
> the
> >>>>> way `itertools.batched`
> >>>>> works, would need `ceil(len(items) / size)` task instances, which is
> >>>>> unknowable until a possibly
> >>>>> unbounded or paginated iterable has been fully drained. That would
> >>> defeat
> >>>>> the purpose of iterating
> >>>>> lazily.
> >>>>>
> >>>>> ## The concern
> >>>>>
> >>>>> The name reads like `itertools.batched(iterable, size)`, where `size`
> >>> is
> >>>>> the chunk length and the
> >>>>> number of chunks is derived from it. Ours is the opposite: the number
> >>> of
> >>>>> task instances is given,
> >>>>> and the per-instance share is derived. The class docstring and the
> Task
> >>>>> SDK docs currently spend a
> >>>>> full paragraph undoing that first impression. A name that carries the
> >>>>> meaning directly would not
> >>>>> need it.
> >>>>>
> >>>>> ## Options
> >>>>>
> >>>>> 1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but
> >>> the
> >>>>> mismatch with
> >>>>> `itertools.batched` stays and every new reader has to be corrected.
> >>>>>
> >>>>> 2. Keep the method, rename the parameter: `.batch(count=N)` or
> >>>>> `.batch(tasks=N)`. Keeps the
> >>>>> "Dynamic Task Batching" vocabulary from the AIP, and `count` or
> >>>> `tasks`
> >>>>> says what the number
> >>>>> is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17
> >>>>> tasks, then iterate".
> >>>>>
> >>>>> 3. Rename the method as well, to something that describes
> partitioning
> >>>>> rather than chunking, for
> >>>>> example `.partition(count=N)`, `.shard(count=N)` or
> >>>>> `.spread(across=N)`. Most descriptive, but
> >>>>> drifts from the AIP's own terminology, and "shard" and "partition"
> >>>>> carry data-engineering
> >>>>> connotations that may suggest contiguous ranges just as much as
> >>>> "batch"
> >>>>> does.
> >>>>>
> >>>>> My preference is option 2 with `count=`. It keeps the API shape and
> the
> >>>>> AIP terminology, fixes
> >>>>> the misleading part, and costs nothing now because the feature is
> >>>>> unreleased. `size` would be
> >>>>> kept as a rejected alias raising a clear error rather than silently
> >>>>> accepted, so nobody copies
> >>>>> the old spelling from an early draft.
> >>>>>
> >>>>> Unless there are objections or a better name comes up, I will treat
> >>> this
> >>>>> as lazy consensus in
> >>>>> 72 hours and update the PR, the docs and the AIP page accordingly.
> >>>>>
> >>>>> Thanks,
> >>>>> David
> >>>>>
> >>>>> General (Internal Property)
> >>>>>
> >>>>
> >>>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable Tasks and Dynamic Task Batching)
Posted by "constance.astronomer.io via dev" <de...@airflow.apache.org>.
+1 to keeping the feature name as is.
For the function name, my vote is still some_task.spread(across=N), but between parallelize and distribute it’s just a matter of personal preference. It makes the behaviour really obvious, sounds more natural and is more readable (to me) and doesn’t imply that all mapped tasks will literally run in parallel at the exact same time.
-1 to _(across_n_tasks=N). It’s unnecessary with the verbs in the running.
> On Sep 24, 2026, at 6:20 AM, Ash Berlin-Taylor <as...@apache.org> wrote:
>
> Sooo I was throwing some ideas around with David in a DM, and here are the ones I came up with
>
> some_task.batch(num_tasks=5)?
> some_task.batch(across=5) ?
> some_task.spread(across=5) ?
> some_task.parallelise(across=5)?
> some_task.distribute(across=5)?
> some_task.distribute(across_n_tasks=5)?
>
> I think I like the `.parallelise(across=n)` the most? It’s not perfect I admit.
>
> Also (perhaps oddly) I think even with that I think the feature should be called "Batched Mapped Task” even if the fn is .parallelise or .distribute — but I don’t think "Task Spreading” is very clear from a user-reading-the-docs PoV.
>
> -a
>
>> On 24 Sep 2026, at 09:51, Amogh Desai <am...@apache.org> wrote:
>>
>> Late to the party, but thanks, David.
>>
>> Out of all the options, I vote +1 for `.spread(across=N)`.
>>
>> -1 to unfold because, to me, it means generating a sequence from something,
>> which describes
>> creating items rather than distributing them.
>>
>> And regardless of the name, the round-robin distribution needs to be
>> explicit in the SDK docs with the reasoning,
>> as Constance said. The name fixes the "how many" confusion, not the "which
>> items go where" one.
>>
>> Thanks & Regards,
>> Amogh Desai
>>
>>
>>> On Thu, Sep 24, 2026 at 1:48 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>> How about "scatter" or "unfold" :)?
>>>
>>> (just had a little conversation with Claude about it).
>>>
>>> On Wed, Sep 23, 2026 at 12:20 PM Blain David <da...@infrabel.be>
>>> wrote:
>>>
>>>> I've updated the AIP-104 confluence page and choose the spread option and
>>>> thus the Task Spreading instead of Batched Task Mapping.
>>>>
>>>> Does everyone agree with this change?
>>>>
>>>>
>>>>
>>> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
>>>>
>>>> If so, then I will push those changes in the PR as well:
>>>>
>>>> https://github.com/apache/airflow/pull/62922
>>>>
>>>> General (Internal Property)
>>>> ________________________________
>>>> From: Shahar Epstein <sh...@apache.org>
>>>> Sent: Saturday, September 19, 2026 10:37
>>>> To: dev@airflow.apache.org <de...@airflow.apache.org>
>>>> Subject: Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable
>>>> Tasks and Dynamic Task Batching)
>>>>
>>>> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze
>>>> niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel,
>>>> stuur deze e-mail als bijlage naar abuse@infrabel.be<mailto:
>>>> abuse@infrabel.be>.
>>>>
>>>> I'd go with .spread(...), as both batch(...) and shard(...) are already
>>>> semantically overloaded.
>>>>
>>>>
>>>> Shahar
>>>>
>>>> On Thu, Sep 17, 2026 at 12:26 PM Blain David <da...@infrabel.be>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> While reviewing the AIP-104 PR (
>>>>>
>>>>
>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F62922&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cfb61e6c745c34a77080408df1629466b%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639254038688014321%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=S4qPFkVdn2y1F%2FtLRWhlkqwazsOUbNVrPEi%2FYuHoN7w%3D&reserved=0
>>>> )<https://github.com/apache/airflow/pull/62922>, Amogh raised a naming
>>>>> concern about the Dynamic Task Batching API that I would like to settle
>>>>> here before the PR merges,
>>>>> since renaming afterwards would be a breaking change.
>>>>>
>>>>> ## What `.batch(size=N)` does today
>>>>>
>>>>> `task.batch(size=17).iterate(url=urls)` creates 17 mapped task
>>> instances
>>>>> through Dynamic Task
>>>>> Mapping, and each of them iterates over its share of `urls` in a single
>>>>> task instance using
>>>>> Iterable Tasks. The items are distributed round robin: item `i` goes to
>>>>> task instance `i % 17`.
>>>>> So `size` is the number of task instances, not a chunk length, and the
>>>>> task instances do not
>>>>> receive contiguous slices of the input.
>>>>>
>>>>> The round robin scheme is deliberate. The number of task instances has
>>> to
>>>>> be fixed before the
>>>>> underlying iterable is consumed, because the scheduler needs the
>>> mapping
>>>>> cardinality up front.
>>>>> With round robin that number is simply `size`. Contiguous chunking, the
>>>>> way `itertools.batched`
>>>>> works, would need `ceil(len(items) / size)` task instances, which is
>>>>> unknowable until a possibly
>>>>> unbounded or paginated iterable has been fully drained. That would
>>> defeat
>>>>> the purpose of iterating
>>>>> lazily.
>>>>>
>>>>> ## The concern
>>>>>
>>>>> The name reads like `itertools.batched(iterable, size)`, where `size`
>>> is
>>>>> the chunk length and the
>>>>> number of chunks is derived from it. Ours is the opposite: the number
>>> of
>>>>> task instances is given,
>>>>> and the per-instance share is derived. The class docstring and the Task
>>>>> SDK docs currently spend a
>>>>> full paragraph undoing that first impression. A name that carries the
>>>>> meaning directly would not
>>>>> need it.
>>>>>
>>>>> ## Options
>>>>>
>>>>> 1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but
>>> the
>>>>> mismatch with
>>>>> `itertools.batched` stays and every new reader has to be corrected.
>>>>>
>>>>> 2. Keep the method, rename the parameter: `.batch(count=N)` or
>>>>> `.batch(tasks=N)`. Keeps the
>>>>> "Dynamic Task Batching" vocabulary from the AIP, and `count` or
>>>> `tasks`
>>>>> says what the number
>>>>> is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17
>>>>> tasks, then iterate".
>>>>>
>>>>> 3. Rename the method as well, to something that describes partitioning
>>>>> rather than chunking, for
>>>>> example `.partition(count=N)`, `.shard(count=N)` or
>>>>> `.spread(across=N)`. Most descriptive, but
>>>>> drifts from the AIP's own terminology, and "shard" and "partition"
>>>>> carry data-engineering
>>>>> connotations that may suggest contiguous ranges just as much as
>>>> "batch"
>>>>> does.
>>>>>
>>>>> My preference is option 2 with `count=`. It keeps the API shape and the
>>>>> AIP terminology, fixes
>>>>> the misleading part, and costs nothing now because the feature is
>>>>> unreleased. `size` would be
>>>>> kept as a rejected alias raising a clear error rather than silently
>>>>> accepted, so nobody copies
>>>>> the old spelling from an early draft.
>>>>>
>>>>> Unless there are objections or a better name comes up, I will treat
>>> this
>>>>> as lazy consensus in
>>>>> 72 hours and update the PR, the docs and the AIP page accordingly.
>>>>>
>>>>> Thanks,
>>>>> David
>>>>>
>>>>> General (Internal Property)
>>>>>
>>>>
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable Tasks and Dynamic Task Batching)
Posted by Ash Berlin-Taylor <as...@apache.org>.
Sooo I was throwing some ideas around with David in a DM, and here are the ones I came up with
some_task.batch(num_tasks=5)?
some_task.batch(across=5) ?
some_task.spread(across=5) ?
some_task.parallelise(across=5)?
some_task.distribute(across=5)?
some_task.distribute(across_n_tasks=5)?
I think I like the `.parallelise(across=n)` the most? It’s not perfect I admit.
Also (perhaps oddly) I think even with that I think the feature should be called "Batched Mapped Task” even if the fn is .parallelise or .distribute — but I don’t think "Task Spreading” is very clear from a user-reading-the-docs PoV.
-a
> On 24 Sep 2026, at 09:51, Amogh Desai <am...@apache.org> wrote:
>
> Late to the party, but thanks, David.
>
> Out of all the options, I vote +1 for `.spread(across=N)`.
>
> -1 to unfold because, to me, it means generating a sequence from something,
> which describes
> creating items rather than distributing them.
>
> And regardless of the name, the round-robin distribution needs to be
> explicit in the SDK docs with the reasoning,
> as Constance said. The name fixes the "how many" confusion, not the "which
> items go where" one.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Thu, Sep 24, 2026 at 1:48 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> How about "scatter" or "unfold" :)?
>>
>> (just had a little conversation with Claude about it).
>>
>> On Wed, Sep 23, 2026 at 12:20 PM Blain David <da...@infrabel.be>
>> wrote:
>>
>>> I've updated the AIP-104 confluence page and choose the spread option and
>>> thus the Task Spreading instead of Batched Task Mapping.
>>>
>>> Does everyone agree with this change?
>>>
>>>
>>>
>> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
>>>
>>> If so, then I will push those changes in the PR as well:
>>>
>>> https://github.com/apache/airflow/pull/62922
>>>
>>> General (Internal Property)
>>> ________________________________
>>> From: Shahar Epstein <sh...@apache.org>
>>> Sent: Saturday, September 19, 2026 10:37
>>> To: dev@airflow.apache.org <de...@airflow.apache.org>
>>> Subject: Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable
>>> Tasks and Dynamic Task Batching)
>>>
>>> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze
>>> niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel,
>>> stuur deze e-mail als bijlage naar abuse@infrabel.be<mailto:
>>> abuse@infrabel.be>.
>>>
>>> I'd go with .spread(...), as both batch(...) and shard(...) are already
>>> semantically overloaded.
>>>
>>>
>>> Shahar
>>>
>>> On Thu, Sep 17, 2026 at 12:26 PM Blain David <da...@infrabel.be>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> While reviewing the AIP-104 PR (
>>>>
>>>
>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F62922&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cfb61e6c745c34a77080408df1629466b%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639254038688014321%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=S4qPFkVdn2y1F%2FtLRWhlkqwazsOUbNVrPEi%2FYuHoN7w%3D&reserved=0
>>> )<https://github.com/apache/airflow/pull/62922>, Amogh raised a naming
>>>> concern about the Dynamic Task Batching API that I would like to settle
>>>> here before the PR merges,
>>>> since renaming afterwards would be a breaking change.
>>>>
>>>> ## What `.batch(size=N)` does today
>>>>
>>>> `task.batch(size=17).iterate(url=urls)` creates 17 mapped task
>> instances
>>>> through Dynamic Task
>>>> Mapping, and each of them iterates over its share of `urls` in a single
>>>> task instance using
>>>> Iterable Tasks. The items are distributed round robin: item `i` goes to
>>>> task instance `i % 17`.
>>>> So `size` is the number of task instances, not a chunk length, and the
>>>> task instances do not
>>>> receive contiguous slices of the input.
>>>>
>>>> The round robin scheme is deliberate. The number of task instances has
>> to
>>>> be fixed before the
>>>> underlying iterable is consumed, because the scheduler needs the
>> mapping
>>>> cardinality up front.
>>>> With round robin that number is simply `size`. Contiguous chunking, the
>>>> way `itertools.batched`
>>>> works, would need `ceil(len(items) / size)` task instances, which is
>>>> unknowable until a possibly
>>>> unbounded or paginated iterable has been fully drained. That would
>> defeat
>>>> the purpose of iterating
>>>> lazily.
>>>>
>>>> ## The concern
>>>>
>>>> The name reads like `itertools.batched(iterable, size)`, where `size`
>> is
>>>> the chunk length and the
>>>> number of chunks is derived from it. Ours is the opposite: the number
>> of
>>>> task instances is given,
>>>> and the per-instance share is derived. The class docstring and the Task
>>>> SDK docs currently spend a
>>>> full paragraph undoing that first impression. A name that carries the
>>>> meaning directly would not
>>>> need it.
>>>>
>>>> ## Options
>>>>
>>>> 1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but
>> the
>>>> mismatch with
>>>> `itertools.batched` stays and every new reader has to be corrected.
>>>>
>>>> 2. Keep the method, rename the parameter: `.batch(count=N)` or
>>>> `.batch(tasks=N)`. Keeps the
>>>> "Dynamic Task Batching" vocabulary from the AIP, and `count` or
>>> `tasks`
>>>> says what the number
>>>> is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17
>>>> tasks, then iterate".
>>>>
>>>> 3. Rename the method as well, to something that describes partitioning
>>>> rather than chunking, for
>>>> example `.partition(count=N)`, `.shard(count=N)` or
>>>> `.spread(across=N)`. Most descriptive, but
>>>> drifts from the AIP's own terminology, and "shard" and "partition"
>>>> carry data-engineering
>>>> connotations that may suggest contiguous ranges just as much as
>>> "batch"
>>>> does.
>>>>
>>>> My preference is option 2 with `count=`. It keeps the API shape and the
>>>> AIP terminology, fixes
>>>> the misleading part, and costs nothing now because the feature is
>>>> unreleased. `size` would be
>>>> kept as a rejected alias raising a clear error rather than silently
>>>> accepted, so nobody copies
>>>> the old spelling from an early draft.
>>>>
>>>> Unless there are objections or a better name comes up, I will treat
>> this
>>>> as lazy consensus in
>>>> 72 hours and update the PR, the docs and the AIP page accordingly.
>>>>
>>>> Thanks,
>>>> David
>>>>
>>>> General (Internal Property)
>>>>
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable Tasks and Dynamic Task Batching)
Posted by Amogh Desai <am...@apache.org>.
Late to the party, but thanks, David.
Out of all the options, I vote +1 for `.spread(across=N)`.
-1 to unfold because, to me, it means generating a sequence from something,
which describes
creating items rather than distributing them.
And regardless of the name, the round-robin distribution needs to be
explicit in the SDK docs with the reasoning,
as Constance said. The name fixes the "how many" confusion, not the "which
items go where" one.
Thanks & Regards,
Amogh Desai
On Thu, Sep 24, 2026 at 1:48 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> How about "scatter" or "unfold" :)?
>
> (just had a little conversation with Claude about it).
>
> On Wed, Sep 23, 2026 at 12:20 PM Blain David <da...@infrabel.be>
> wrote:
>
> > I've updated the AIP-104 confluence page and choose the spread option and
> > thus the Task Spreading instead of Batched Task Mapping.
> >
> > Does everyone agree with this change?
> >
> >
> >
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
> >
> > If so, then I will push those changes in the PR as well:
> >
> > https://github.com/apache/airflow/pull/62922
> >
> > General (Internal Property)
> > ________________________________
> > From: Shahar Epstein <sh...@apache.org>
> > Sent: Saturday, September 19, 2026 10:37
> > To: dev@airflow.apache.org <de...@airflow.apache.org>
> > Subject: Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable
> > Tasks and Dynamic Task Batching)
> >
> > EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze
> > niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel,
> > stuur deze e-mail als bijlage naar abuse@infrabel.be<mailto:
> > abuse@infrabel.be>.
> >
> > I'd go with .spread(...), as both batch(...) and shard(...) are already
> > semantically overloaded.
> >
> >
> > Shahar
> >
> > On Thu, Sep 17, 2026 at 12:26 PM Blain David <da...@infrabel.be>
> > wrote:
> >
> > > Hi all,
> > >
> > > While reviewing the AIP-104 PR (
> > >
> >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F62922&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cfb61e6c745c34a77080408df1629466b%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639254038688014321%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=S4qPFkVdn2y1F%2FtLRWhlkqwazsOUbNVrPEi%2FYuHoN7w%3D&reserved=0
> > )<https://github.com/apache/airflow/pull/62922>, Amogh raised a naming
> > > concern about the Dynamic Task Batching API that I would like to settle
> > > here before the PR merges,
> > > since renaming afterwards would be a breaking change.
> > >
> > > ## What `.batch(size=N)` does today
> > >
> > > `task.batch(size=17).iterate(url=urls)` creates 17 mapped task
> instances
> > > through Dynamic Task
> > > Mapping, and each of them iterates over its share of `urls` in a single
> > > task instance using
> > > Iterable Tasks. The items are distributed round robin: item `i` goes to
> > > task instance `i % 17`.
> > > So `size` is the number of task instances, not a chunk length, and the
> > > task instances do not
> > > receive contiguous slices of the input.
> > >
> > > The round robin scheme is deliberate. The number of task instances has
> to
> > > be fixed before the
> > > underlying iterable is consumed, because the scheduler needs the
> mapping
> > > cardinality up front.
> > > With round robin that number is simply `size`. Contiguous chunking, the
> > > way `itertools.batched`
> > > works, would need `ceil(len(items) / size)` task instances, which is
> > > unknowable until a possibly
> > > unbounded or paginated iterable has been fully drained. That would
> defeat
> > > the purpose of iterating
> > > lazily.
> > >
> > > ## The concern
> > >
> > > The name reads like `itertools.batched(iterable, size)`, where `size`
> is
> > > the chunk length and the
> > > number of chunks is derived from it. Ours is the opposite: the number
> of
> > > task instances is given,
> > > and the per-instance share is derived. The class docstring and the Task
> > > SDK docs currently spend a
> > > full paragraph undoing that first impression. A name that carries the
> > > meaning directly would not
> > > need it.
> > >
> > > ## Options
> > >
> > > 1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but
> the
> > > mismatch with
> > > `itertools.batched` stays and every new reader has to be corrected.
> > >
> > > 2. Keep the method, rename the parameter: `.batch(count=N)` or
> > > `.batch(tasks=N)`. Keeps the
> > > "Dynamic Task Batching" vocabulary from the AIP, and `count` or
> > `tasks`
> > > says what the number
> > > is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17
> > > tasks, then iterate".
> > >
> > > 3. Rename the method as well, to something that describes partitioning
> > > rather than chunking, for
> > > example `.partition(count=N)`, `.shard(count=N)` or
> > > `.spread(across=N)`. Most descriptive, but
> > > drifts from the AIP's own terminology, and "shard" and "partition"
> > > carry data-engineering
> > > connotations that may suggest contiguous ranges just as much as
> > "batch"
> > > does.
> > >
> > > My preference is option 2 with `count=`. It keeps the API shape and the
> > > AIP terminology, fixes
> > > the misleading part, and costs nothing now because the feature is
> > > unreleased. `size` would be
> > > kept as a rejected alias raising a clear error rather than silently
> > > accepted, so nobody copies
> > > the old spelling from an early draft.
> > >
> > > Unless there are objections or a better name comes up, I will treat
> this
> > > as lazy consensus in
> > > 72 hours and update the PR, the docs and the AIP page accordingly.
> > >
> > > Thanks,
> > > David
> > >
> > > General (Internal Property)
> > >
> >
>
Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable Tasks and Dynamic Task Batching)
Posted by Jarek Potiuk <ja...@potiuk.com>.
How about "scatter" or "unfold" :)?
(just had a little conversation with Claude about it).
On Wed, Sep 23, 2026 at 12:20 PM Blain David <da...@infrabel.be>
wrote:
> I've updated the AIP-104 confluence page and choose the spread option and
> thus the Task Spreading instead of Batched Task Mapping.
>
> Does everyone agree with this change?
>
>
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
>
> If so, then I will push those changes in the PR as well:
>
> https://github.com/apache/airflow/pull/62922
>
> General (Internal Property)
> ________________________________
> From: Shahar Epstein <sh...@apache.org>
> Sent: Saturday, September 19, 2026 10:37
> To: dev@airflow.apache.org <de...@airflow.apache.org>
> Subject: Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable
> Tasks and Dynamic Task Batching)
>
> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze
> niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel,
> stuur deze e-mail als bijlage naar abuse@infrabel.be<mailto:
> abuse@infrabel.be>.
>
> I'd go with .spread(...), as both batch(...) and shard(...) are already
> semantically overloaded.
>
>
> Shahar
>
> On Thu, Sep 17, 2026 at 12:26 PM Blain David <da...@infrabel.be>
> wrote:
>
> > Hi all,
> >
> > While reviewing the AIP-104 PR (
> >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F62922&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cfb61e6c745c34a77080408df1629466b%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639254038688014321%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=S4qPFkVdn2y1F%2FtLRWhlkqwazsOUbNVrPEi%2FYuHoN7w%3D&reserved=0
> )<https://github.com/apache/airflow/pull/62922>, Amogh raised a naming
> > concern about the Dynamic Task Batching API that I would like to settle
> > here before the PR merges,
> > since renaming afterwards would be a breaking change.
> >
> > ## What `.batch(size=N)` does today
> >
> > `task.batch(size=17).iterate(url=urls)` creates 17 mapped task instances
> > through Dynamic Task
> > Mapping, and each of them iterates over its share of `urls` in a single
> > task instance using
> > Iterable Tasks. The items are distributed round robin: item `i` goes to
> > task instance `i % 17`.
> > So `size` is the number of task instances, not a chunk length, and the
> > task instances do not
> > receive contiguous slices of the input.
> >
> > The round robin scheme is deliberate. The number of task instances has to
> > be fixed before the
> > underlying iterable is consumed, because the scheduler needs the mapping
> > cardinality up front.
> > With round robin that number is simply `size`. Contiguous chunking, the
> > way `itertools.batched`
> > works, would need `ceil(len(items) / size)` task instances, which is
> > unknowable until a possibly
> > unbounded or paginated iterable has been fully drained. That would defeat
> > the purpose of iterating
> > lazily.
> >
> > ## The concern
> >
> > The name reads like `itertools.batched(iterable, size)`, where `size` is
> > the chunk length and the
> > number of chunks is derived from it. Ours is the opposite: the number of
> > task instances is given,
> > and the per-instance share is derived. The class docstring and the Task
> > SDK docs currently spend a
> > full paragraph undoing that first impression. A name that carries the
> > meaning directly would not
> > need it.
> >
> > ## Options
> >
> > 1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but the
> > mismatch with
> > `itertools.batched` stays and every new reader has to be corrected.
> >
> > 2. Keep the method, rename the parameter: `.batch(count=N)` or
> > `.batch(tasks=N)`. Keeps the
> > "Dynamic Task Batching" vocabulary from the AIP, and `count` or
> `tasks`
> > says what the number
> > is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17
> > tasks, then iterate".
> >
> > 3. Rename the method as well, to something that describes partitioning
> > rather than chunking, for
> > example `.partition(count=N)`, `.shard(count=N)` or
> > `.spread(across=N)`. Most descriptive, but
> > drifts from the AIP's own terminology, and "shard" and "partition"
> > carry data-engineering
> > connotations that may suggest contiguous ranges just as much as
> "batch"
> > does.
> >
> > My preference is option 2 with `count=`. It keeps the API shape and the
> > AIP terminology, fixes
> > the misleading part, and costs nothing now because the feature is
> > unreleased. `size` would be
> > kept as a rejected alias raising a clear error rather than silently
> > accepted, so nobody copies
> > the old spelling from an early draft.
> >
> > Unless there are objections or a better name comes up, I will treat this
> > as lazy consensus in
> > 72 hours and update the PR, the docs and the AIP page accordingly.
> >
> > Thanks,
> > David
> >
> > General (Internal Property)
> >
>
Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable Tasks and Dynamic Task Batching)
Posted by Blain David <da...@infrabel.be>.
I've updated the AIP-104 confluence page and choose the spread option and thus the Task Spreading instead of Batched Task Mapping.
Does everyone agree with this change?
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
If so, then I will push those changes in the PR as well:
https://github.com/apache/airflow/pull/62922
General (Internal Property)
________________________________
From: Shahar Epstein <sh...@apache.org>
Sent: Saturday, September 19, 2026 10:37
To: dev@airflow.apache.org <de...@airflow.apache.org>
Subject: Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable Tasks and Dynamic Task Batching)
EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar abuse@infrabel.be<ma...@infrabel.be>.
I'd go with .spread(...), as both batch(...) and shard(...) are already
semantically overloaded.
Shahar
On Thu, Sep 17, 2026 at 12:26 PM Blain David <da...@infrabel.be>
wrote:
> Hi all,
>
> While reviewing the AIP-104 PR (
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F62922&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cfb61e6c745c34a77080408df1629466b%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639254038688014321%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=S4qPFkVdn2y1F%2FtLRWhlkqwazsOUbNVrPEi%2FYuHoN7w%3D&reserved=0)<https://github.com/apache/airflow/pull/62922>, Amogh raised a naming
> concern about the Dynamic Task Batching API that I would like to settle
> here before the PR merges,
> since renaming afterwards would be a breaking change.
>
> ## What `.batch(size=N)` does today
>
> `task.batch(size=17).iterate(url=urls)` creates 17 mapped task instances
> through Dynamic Task
> Mapping, and each of them iterates over its share of `urls` in a single
> task instance using
> Iterable Tasks. The items are distributed round robin: item `i` goes to
> task instance `i % 17`.
> So `size` is the number of task instances, not a chunk length, and the
> task instances do not
> receive contiguous slices of the input.
>
> The round robin scheme is deliberate. The number of task instances has to
> be fixed before the
> underlying iterable is consumed, because the scheduler needs the mapping
> cardinality up front.
> With round robin that number is simply `size`. Contiguous chunking, the
> way `itertools.batched`
> works, would need `ceil(len(items) / size)` task instances, which is
> unknowable until a possibly
> unbounded or paginated iterable has been fully drained. That would defeat
> the purpose of iterating
> lazily.
>
> ## The concern
>
> The name reads like `itertools.batched(iterable, size)`, where `size` is
> the chunk length and the
> number of chunks is derived from it. Ours is the opposite: the number of
> task instances is given,
> and the per-instance share is derived. The class docstring and the Task
> SDK docs currently spend a
> full paragraph undoing that first impression. A name that carries the
> meaning directly would not
> need it.
>
> ## Options
>
> 1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but the
> mismatch with
> `itertools.batched` stays and every new reader has to be corrected.
>
> 2. Keep the method, rename the parameter: `.batch(count=N)` or
> `.batch(tasks=N)`. Keeps the
> "Dynamic Task Batching" vocabulary from the AIP, and `count` or `tasks`
> says what the number
> is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17
> tasks, then iterate".
>
> 3. Rename the method as well, to something that describes partitioning
> rather than chunking, for
> example `.partition(count=N)`, `.shard(count=N)` or
> `.spread(across=N)`. Most descriptive, but
> drifts from the AIP's own terminology, and "shard" and "partition"
> carry data-engineering
> connotations that may suggest contiguous ranges just as much as "batch"
> does.
>
> My preference is option 2 with `count=`. It keeps the API shape and the
> AIP terminology, fixes
> the misleading part, and costs nothing now because the feature is
> unreleased. `size` would be
> kept as a rejected alias raising a clear error rather than silently
> accepted, so nobody copies
> the old spelling from an early draft.
>
> Unless there are objections or a better name comes up, I will treat this
> as lazy consensus in
> 72 hours and update the PR, the docs and the AIP page accordingly.
>
> Thanks,
> David
>
> General (Internal Property)
>
Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable Tasks and Dynamic Task Batching)
Posted by Shahar Epstein <sh...@apache.org>.
I'd go with .spread(...), as both batch(...) and shard(...) are already
semantically overloaded.
Shahar
On Thu, Sep 17, 2026 at 12:26 PM Blain David <da...@infrabel.be>
wrote:
> Hi all,
>
> While reviewing the AIP-104 PR (
> https://github.com/apache/airflow/pull/62922), Amogh raised a naming
> concern about the Dynamic Task Batching API that I would like to settle
> here before the PR merges,
> since renaming afterwards would be a breaking change.
>
> ## What `.batch(size=N)` does today
>
> `task.batch(size=17).iterate(url=urls)` creates 17 mapped task instances
> through Dynamic Task
> Mapping, and each of them iterates over its share of `urls` in a single
> task instance using
> Iterable Tasks. The items are distributed round robin: item `i` goes to
> task instance `i % 17`.
> So `size` is the number of task instances, not a chunk length, and the
> task instances do not
> receive contiguous slices of the input.
>
> The round robin scheme is deliberate. The number of task instances has to
> be fixed before the
> underlying iterable is consumed, because the scheduler needs the mapping
> cardinality up front.
> With round robin that number is simply `size`. Contiguous chunking, the
> way `itertools.batched`
> works, would need `ceil(len(items) / size)` task instances, which is
> unknowable until a possibly
> unbounded or paginated iterable has been fully drained. That would defeat
> the purpose of iterating
> lazily.
>
> ## The concern
>
> The name reads like `itertools.batched(iterable, size)`, where `size` is
> the chunk length and the
> number of chunks is derived from it. Ours is the opposite: the number of
> task instances is given,
> and the per-instance share is derived. The class docstring and the Task
> SDK docs currently spend a
> full paragraph undoing that first impression. A name that carries the
> meaning directly would not
> need it.
>
> ## Options
>
> 1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but the
> mismatch with
> `itertools.batched` stays and every new reader has to be corrected.
>
> 2. Keep the method, rename the parameter: `.batch(count=N)` or
> `.batch(tasks=N)`. Keeps the
> "Dynamic Task Batching" vocabulary from the AIP, and `count` or `tasks`
> says what the number
> is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17
> tasks, then iterate".
>
> 3. Rename the method as well, to something that describes partitioning
> rather than chunking, for
> example `.partition(count=N)`, `.shard(count=N)` or
> `.spread(across=N)`. Most descriptive, but
> drifts from the AIP's own terminology, and "shard" and "partition"
> carry data-engineering
> connotations that may suggest contiguous ranges just as much as "batch"
> does.
>
> My preference is option 2 with `count=`. It keeps the API shape and the
> AIP terminology, fixes
> the misleading part, and costs nothing now because the feature is
> unreleased. `size` would be
> kept as a rejected alias raising a clear error rather than silently
> accepted, so nobody copies
> the old spelling from an early draft.
>
> Unless there are objections or a better name comes up, I will treat this
> as lazy consensus in
> 72 hours and update the PR, the docs and the AIP page accordingly.
>
> Thanks,
> David
>
> General (Internal Property)
>