You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Tarun Annapareddy via dev <de...@beam.apache.org> on 2026/10/21 03:01:51 UTC

[PROPOSAL] Environment-Sensitive Staging in Beam Python

Hi everyone,

Recently, we have encountered Dataflow Multi-Environment pipelines
crash-looping due to integrity check failures. However, while debugging we
realized it's a long-standing issue in Beam Python dependency staging. This
fix is small, but a bit convoluted, so I have created a doc and attached
the PR.

Doc:
https://docs.google.com/document/d/1gNfjbIYNzbv15-RjCj0ad4MHenVPoSjUEcH0Ok3iNyU/edit?usp=sharing

PR: go/beam-pr/40699

Thank you,
Tarun.

Re: [PROPOSAL] Environment-Sensitive Staging in Beam Python

Posted by Valentyn Tymofieiev via dev <de...@beam.apache.org>.
I looked closer into the tentative fix and and the crux of the issue. The
issue pertains to a contract between the runner and the SDK regarding the
--semi_persist_dir that a runner provisions and provides to the SDK
(fallback is /tmp). Currently, in XLang Dataflow pipelines two different
SDK containers can recieve from the runner a semi_persist_dir path that is
backed by the same physical location.

The question is then, what is responsible to resolve this conflict:

A) Runner is responsible to provide a unique --semi_persist_dir path to
every SDK.

B) SDK is responsible to write to a unique path, namespaced by SDK ID
assigned by the runner, to avoid conflicts.

Option A requires added implementation work by runners. Option B makes it
harder to know the exact path to staged files on the worker, since the SDK
ID could change based on runtime configuration.

Approach B seems to be a simpler solution since we could address it within
the SDK, but  can cause a breaking change for users who currently rely
on --files_to_stage
option and make an assumption on the local path in Dataflow (or other
runners).




On Wed, Oct 21, 2026 at 11:14 AM Valentyn Tymofieiev <va...@google.com>
wrote:

> +1, thank you!
>
> On Wed, Oct 21, 2026 at 10:56 AM Chamikara Jayalath via dev <
> dev@beam.apache.org> wrote:
>
>> +1. Thanks a lot for looking into this and providing a fix Tarun.
>>
>> - Cham
>>
>> On Wed, Oct 21, 2026 at 6:41 AM Kenneth Knowles <ke...@apache.org> wrote:
>>
>>> Nice catch, and good write-up. Thanks for the clear summary in the doc,
>>> too.
>>>
>>> Kenn
>>>
>>> On Tue, Oct 20, 2026 at 11:02 PM Tarun Annapareddy via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> Recently, we have encountered Dataflow Multi-Environment pipelines
>>>> crash-looping due to integrity check failures. However, while debugging we
>>>> realized it's a long-standing issue in Beam Python dependency staging. This
>>>> fix is small, but a bit convoluted, so I have created a doc and attached
>>>> the PR.
>>>>
>>>> Doc:
>>>> https://docs.google.com/document/d/1gNfjbIYNzbv15-RjCj0ad4MHenVPoSjUEcH0Ok3iNyU/edit?usp=sharing
>>>>
>>>> PR: go/beam-pr/40699 <https://goto.google.com/beam-pr/40699>
>>>>
>>>> Thank you,
>>>> Tarun.
>>>>
>>>

Re: [PROPOSAL] Environment-Sensitive Staging in Beam Python

Posted by Valentyn Tymofieiev via dev <de...@beam.apache.org>.
Thanks, i think this can work. in this case it becomes a Dataflow runner
particuliarity that  "$semi_persist_dir/staged" is not shared between
environments but other portions of $semi_persist_dir may still be shared".
The advantage is that we avoid a breaking change for Python users, and it
is still possible to write a file into a location shared across all envs if
necessary.

>

Re: [PROPOSAL] Environment-Sensitive Staging in Beam Python

Posted by Tarun Annapareddy via dev <de...@beam.apache.org>.
Thanks everyone, for the suggestions!

I updated the doc to accommodate the new design we discussed on the PR
go/beam-pr/40699 <https://goto.google.com/beam-pr/40699>

https://docs.google.com/document/d/1gNfjbIYNzbv15-RjCj0ad4MHenVPoSjUEcH0Ok3iNyU/edit?usp=sharing

Thank you,
Tarun.

On Thu, Oct 29, 2026 at 6:09 PM Valentyn Tymofieiev <va...@google.com>
wrote:

> It looks like there is an experiment that can be used to temporarily
> override the sha checks while service side mitigation is in progress. This
> is not ideal, and we still have a race condition during artifact
> materialization in the latest SDK release. But this race is a preexisting
> problem. Therefore my preference would be not to incur technical debt in
> the SDK with a rushed workaround if we can avoid it, so I am voting -0 on
> implementing option 1 in the ongoing release.
>
> I think the existing workaround (opt-out experiment for hash validations)
> can be acceptable until the next release since the experiment can be
> user-controlled, and this is probably safer than landing a change to
> artifact staging path done right before the release (bigger blast radius is
> something goes wrong). If we find new information/opinion/evidence that
> resolving staging conflict must be done in SDK rather than runner, then we
> can create a better mechanism (e.g. option 1 or variation thereof) for the
> next release.
>
> Thanks,
> Valentyn
>
> On Thu, Oct 29, 2026 at 3:22 PM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>>
>>
>> On Thu, Oct 29, 2026 at 10:09 AM Tarun Annapareddy <
>> tannapareddy@google.com> wrote:
>>
>>> Unless there are objections. I am proposing the option-1 short-term fix
>>> to unblock the 2.78 release cut. I will update the design doc for option-3,
>>> once 2.78 is done, it also needs alignment from Runner.
>>>
>>
>> Using option 1 to unblock the release sounds good to me. We can look into
>> options 2 or 3 as the long term fix subsequently.
>>
>>
>>>
>>> Thank you,
>>> Tarun.
>>>
>>> On Wed, Oct 28, 2026 at 4:45 PM Chamikara Jayalath <ch...@google.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Wed, Oct 28, 2026 at 4:35 PM Tarun Annapareddy <
>>>> tannapareddy@google.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> ` I understand there is some urgency to have a quick fix as multi-env
>>>>> pipelines might be blocked. If those pipelines are blocked already, do we
>>>>> have a workaround for affected users, for example to use a custom container
>>>>> instead of passing --requirements_file that collides in Python vs YAML
>>>>> environment?`
>>>>>
>>>>> > I do not have a good workaround for these customers. Multi-env YAML
>>>>> pipelines will fail. I am surprised there is no report, perhaps because the
>>>>> regression started only in 2.76 and multi-env is not a regular use case.
>>>>>
>>>>
>>>> This will include all Beam YAML users that use Python providers.
>>>>
>>>> https://beam.apache.org/documentation/sdks/yaml-providers/#python
>>>>
>>>>
>>>>>
>>>>> Thank you,
>>>>> Tarun.
>>>>>
>>>>> On Wed, Oct 28, 2026 at 4:27 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> Thanks, Tarun.
>>>>>>
>>>>>> In addition to boot.go, SDK is also looking for some portion of the
>>>>>> staged files, like pickled_main_session
>>>>>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405> (somehow
>>>>>> we are still using it after cloudpickle migration for new purposes), in
>>>>>> #1/#2 SDK would need to be aware that there is an assigned worker ID (and
>>>>>> it is not the same ID anymore as the ID within boot.go, due to sibling
>>>>>> SDK protocol
>>>>>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278>
>>>>>> ).
>>>>>>
>>>>>> We could do a targeted fix just for requirements.txt, which is even
>>>>>> more narrow and reversible.  Though I think the proper way to solve
>>>>>> this is on the runner. I understand there is some urgency to have a
>>>>>> quick fix as multi-env pipelines might be blocked. If those pipelines are
>>>>>> blocked already, do we have a workaround for affected users, for example to
>>>>>> use a custom container instead of passing --requirements_file that collides
>>>>>> in Python vs YAML  environment?
>>>>>>
>>>>>> If we have a workaround, I'd suggest to try to prototype a runner
>>>>>> side fix before rushing in the SDK change.
>>>>>>
>>>>>> I think fixing the contract on the runner is a cleaner direction (see
>>>>>> #A vs #B in my earlier email), though others are welcome to weigh in here.
>>>>>>
>>>>>> If there are reasons to believe #B is the way to go (SDK must
>>>>>> deduplicate), then I'd explore if we can strengthen some guarantees for
>>>>>> files supplied via --files_to_stage (perhaps via a proto change), so that
>>>>>> we determine the location for these files in a more deterministic manner,
>>>>>> possilbly irrespective of the runner.
>>>>>>
>>>>>

Re: [PROPOSAL] Environment-Sensitive Staging in Beam Python

Posted by Valentyn Tymofieiev via dev <de...@beam.apache.org>.
It looks like there is an experiment that can be used to temporarily
override the sha checks while service side mitigation is in progress. This
is not ideal, and we still have a race condition during artifact
materialization in the latest SDK release. But this race is a preexisting
problem. Therefore my preference would be not to incur technical debt in
the SDK with a rushed workaround if we can avoid it, so I am voting -0 on
implementing option 1 in the ongoing release.

I think the existing workaround (opt-out experiment for hash validations)
can be acceptable until the next release since the experiment can be
user-controlled, and this is probably safer than landing a change to
artifact staging path done right before the release (bigger blast radius is
something goes wrong). If we find new information/opinion/evidence that
resolving staging conflict must be done in SDK rather than runner, then we
can create a better mechanism (e.g. option 1 or variation thereof) for the
next release.

Thanks,
Valentyn

On Thu, Oct 29, 2026 at 3:22 PM Chamikara Jayalath <ch...@google.com>
wrote:

>
>
> On Thu, Oct 29, 2026 at 10:09 AM Tarun Annapareddy <
> tannapareddy@google.com> wrote:
>
>> Unless there are objections. I am proposing the option-1 short-term fix
>> to unblock the 2.78 release cut. I will update the design doc for option-3,
>> once 2.78 is done, it also needs alignment from Runner.
>>
>
> Using option 1 to unblock the release sounds good to me. We can look into
> options 2 or 3 as the long term fix subsequently.
>
>
>>
>> Thank you,
>> Tarun.
>>
>> On Wed, Oct 28, 2026 at 4:45 PM Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>>
>>>
>>> On Wed, Oct 28, 2026 at 4:35 PM Tarun Annapareddy <
>>> tannapareddy@google.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> ` I understand there is some urgency to have a quick fix as multi-env
>>>> pipelines might be blocked. If those pipelines are blocked already, do we
>>>> have a workaround for affected users, for example to use a custom container
>>>> instead of passing --requirements_file that collides in Python vs YAML
>>>> environment?`
>>>>
>>>> > I do not have a good workaround for these customers. Multi-env YAML
>>>> pipelines will fail. I am surprised there is no report, perhaps because the
>>>> regression started only in 2.76 and multi-env is not a regular use case.
>>>>
>>>
>>> This will include all Beam YAML users that use Python providers.
>>>
>>> https://beam.apache.org/documentation/sdks/yaml-providers/#python
>>>
>>>
>>>>
>>>> Thank you,
>>>> Tarun.
>>>>
>>>> On Wed, Oct 28, 2026 at 4:27 PM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>>
>>>>> Thanks, Tarun.
>>>>>
>>>>> In addition to boot.go, SDK is also looking for some portion of the
>>>>> staged files, like pickled_main_session
>>>>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405> (somehow
>>>>> we are still using it after cloudpickle migration for new purposes), in
>>>>> #1/#2 SDK would need to be aware that there is an assigned worker ID (and
>>>>> it is not the same ID anymore as the ID within boot.go, due to sibling
>>>>> SDK protocol
>>>>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278>
>>>>> ).
>>>>>
>>>>> We could do a targeted fix just for requirements.txt, which is even
>>>>> more narrow and reversible.  Though I think the proper way to solve
>>>>> this is on the runner. I understand there is some urgency to have a
>>>>> quick fix as multi-env pipelines might be blocked. If those pipelines are
>>>>> blocked already, do we have a workaround for affected users, for example to
>>>>> use a custom container instead of passing --requirements_file that collides
>>>>> in Python vs YAML  environment?
>>>>>
>>>>> If we have a workaround, I'd suggest to try to prototype a runner side
>>>>> fix before rushing in the SDK change.
>>>>>
>>>>> I think fixing the contract on the runner is a cleaner direction (see
>>>>> #A vs #B in my earlier email), though others are welcome to weigh in here.
>>>>>
>>>>> If there are reasons to believe #B is the way to go (SDK must
>>>>> deduplicate), then I'd explore if we can strengthen some guarantees for
>>>>> files supplied via --files_to_stage (perhaps via a proto change), so that
>>>>> we determine the location for these files in a more deterministic manner,
>>>>> possilbly irrespective of the runner.
>>>>>
>>>>

Re: [PROPOSAL] Environment-Sensitive Staging in Beam Python

Posted by Chamikara Jayalath via dev <de...@beam.apache.org>.
On Thu, Oct 29, 2026 at 10:09 AM Tarun Annapareddy <ta...@google.com>
wrote:

> Unless there are objections. I am proposing the option-1 short-term fix to
> unblock the 2.78 release cut. I will update the design doc for option-3,
> once 2.78 is done, it also needs alignment from Runner.
>

Using option 1 to unblock the release sounds good to me. We can look into
options 2 or 3 as the long term fix subsequently.


>
> Thank you,
> Tarun.
>
> On Wed, Oct 28, 2026 at 4:45 PM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>>
>>
>> On Wed, Oct 28, 2026 at 4:35 PM Tarun Annapareddy <
>> tannapareddy@google.com> wrote:
>>
>>> Hi,
>>>
>>> ` I understand there is some urgency to have a quick fix as multi-env
>>> pipelines might be blocked. If those pipelines are blocked already, do we
>>> have a workaround for affected users, for example to use a custom container
>>> instead of passing --requirements_file that collides in Python vs YAML
>>> environment?`
>>>
>>> > I do not have a good workaround for these customers. Multi-env YAML
>>> pipelines will fail. I am surprised there is no report, perhaps because the
>>> regression started only in 2.76 and multi-env is not a regular use case.
>>>
>>
>> This will include all Beam YAML users that use Python providers.
>>
>> https://beam.apache.org/documentation/sdks/yaml-providers/#python
>>
>>
>>>
>>> Thank you,
>>> Tarun.
>>>
>>> On Wed, Oct 28, 2026 at 4:27 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> Thanks, Tarun.
>>>>
>>>> In addition to boot.go, SDK is also looking for some portion of the
>>>> staged files, like pickled_main_session
>>>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405> (somehow
>>>> we are still using it after cloudpickle migration for new purposes), in
>>>> #1/#2 SDK would need to be aware that there is an assigned worker ID (and
>>>> it is not the same ID anymore as the ID within boot.go, due to sibling
>>>> SDK protocol
>>>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278>
>>>> ).
>>>>
>>>> We could do a targeted fix just for requirements.txt, which is even
>>>> more narrow and reversible.  Though I think the proper way to solve
>>>> this is on the runner. I understand there is some urgency to have a
>>>> quick fix as multi-env pipelines might be blocked. If those pipelines are
>>>> blocked already, do we have a workaround for affected users, for example to
>>>> use a custom container instead of passing --requirements_file that collides
>>>> in Python vs YAML  environment?
>>>>
>>>> If we have a workaround, I'd suggest to try to prototype a runner side
>>>> fix before rushing in the SDK change.
>>>>
>>>> I think fixing the contract on the runner is a cleaner direction (see
>>>> #A vs #B in my earlier email), though others are welcome to weigh in here.
>>>>
>>>> If there are reasons to believe #B is the way to go (SDK must
>>>> deduplicate), then I'd explore if we can strengthen some guarantees for
>>>> files supplied via --files_to_stage (perhaps via a proto change), so that
>>>> we determine the location for these files in a more deterministic manner,
>>>> possilbly irrespective of the runner.
>>>>
>>>

Re: [PROPOSAL] Environment-Sensitive Staging in Beam Python

Posted by Tarun Annapareddy via dev <de...@beam.apache.org>.
Unless there are objections. I am proposing the option-1 short-term fix to
unblock the 2.78 release cut. I will update the design doc for option-3,
once 2.78 is done, it also needs alignment from Runner.

Thank you,
Tarun.

On Wed, Oct 28, 2026 at 4:45 PM Chamikara Jayalath <ch...@google.com>
wrote:

>
>
> On Wed, Oct 28, 2026 at 4:35 PM Tarun Annapareddy <ta...@google.com>
> wrote:
>
>> Hi,
>>
>> ` I understand there is some urgency to have a quick fix as multi-env
>> pipelines might be blocked. If those pipelines are blocked already, do we
>> have a workaround for affected users, for example to use a custom container
>> instead of passing --requirements_file that collides in Python vs YAML
>> environment?`
>>
>> > I do not have a good workaround for these customers. Multi-env YAML
>> pipelines will fail. I am surprised there is no report, perhaps because the
>> regression started only in 2.76 and multi-env is not a regular use case.
>>
>
> This will include all Beam YAML users that use Python providers.
>
> https://beam.apache.org/documentation/sdks/yaml-providers/#python
>
>
>>
>> Thank you,
>> Tarun.
>>
>> On Wed, Oct 28, 2026 at 4:27 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> Thanks, Tarun.
>>>
>>> In addition to boot.go, SDK is also looking for some portion of the
>>> staged files, like pickled_main_session
>>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405> (somehow
>>> we are still using it after cloudpickle migration for new purposes), in
>>> #1/#2 SDK would need to be aware that there is an assigned worker ID (and
>>> it is not the same ID anymore as the ID within boot.go, due to sibling
>>> SDK protocol
>>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278>
>>> ).
>>>
>>> We could do a targeted fix just for requirements.txt, which is even more
>>> narrow and reversible.  Though I think the proper way to solve this is
>>> on the runner. I understand there is some urgency to have a quick fix
>>> as multi-env pipelines might be blocked. If those pipelines are blocked
>>> already, do we have a workaround for affected users, for example to use a
>>> custom container instead of passing --requirements_file that collides in
>>> Python vs YAML  environment?
>>>
>>> If we have a workaround, I'd suggest to try to prototype a runner side
>>> fix before rushing in the SDK change.
>>>
>>> I think fixing the contract on the runner is a cleaner direction (see #A
>>> vs #B in my earlier email), though others are welcome to weigh in here.
>>>
>>> If there are reasons to believe #B is the way to go (SDK must
>>> deduplicate), then I'd explore if we can strengthen some guarantees for
>>> files supplied via --files_to_stage (perhaps via a proto change), so that
>>> we determine the location for these files in a more deterministic manner,
>>> possilbly irrespective of the runner.
>>>
>>

Re: [PROPOSAL] Environment-Sensitive Staging in Beam Python

Posted by Chamikara Jayalath via dev <de...@beam.apache.org>.
On Wed, Oct 28, 2026 at 4:35 PM Tarun Annapareddy <ta...@google.com>
wrote:

> Hi,
>
> ` I understand there is some urgency to have a quick fix as multi-env
> pipelines might be blocked. If those pipelines are blocked already, do we
> have a workaround for affected users, for example to use a custom container
> instead of passing --requirements_file that collides in Python vs YAML
> environment?`
>
> > I do not have a good workaround for these customers. Multi-env YAML
> pipelines will fail. I am surprised there is no report, perhaps because the
> regression started only in 2.76 and multi-env is not a regular use case.
>

This will include all Beam YAML users that use Python providers.

https://beam.apache.org/documentation/sdks/yaml-providers/#python


>
> Thank you,
> Tarun.
>
> On Wed, Oct 28, 2026 at 4:27 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Thanks, Tarun.
>>
>> In addition to boot.go, SDK is also looking for some portion of the
>> staged files, like pickled_main_session
>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405> (somehow
>> we are still using it after cloudpickle migration for new purposes), in
>> #1/#2 SDK would need to be aware that there is an assigned worker ID (and
>> it is not the same ID anymore as the ID within boot.go, due to sibling
>> SDK protocol
>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278>
>> ).
>>
>> We could do a targeted fix just for requirements.txt, which is even more
>> narrow and reversible.  Though I think the proper way to solve this is
>> on the runner. I understand there is some urgency to have a quick fix as
>> multi-env pipelines might be blocked. If those pipelines are blocked
>> already, do we have a workaround for affected users, for example to use a
>> custom container instead of passing --requirements_file that collides in
>> Python vs YAML  environment?
>>
>> If we have a workaround, I'd suggest to try to prototype a runner side
>> fix before rushing in the SDK change.
>>
>> I think fixing the contract on the runner is a cleaner direction (see #A
>> vs #B in my earlier email), though others are welcome to weigh in here.
>>
>> If there are reasons to believe #B is the way to go (SDK must
>> deduplicate), then I'd explore if we can strengthen some guarantees for
>> files supplied via --files_to_stage (perhaps via a proto change), so that
>> we determine the location for these files in a more deterministic manner,
>> possilbly irrespective of the runner.
>>
>

Re: [PROPOSAL] Environment-Sensitive Staging in Beam Python

Posted by Tarun Annapareddy via dev <de...@beam.apache.org>.
Hi,

` I understand there is some urgency to have a quick fix as multi-env
pipelines might be blocked. If those pipelines are blocked already, do we
have a workaround for affected users, for example to use a custom container
instead of passing --requirements_file that collides in Python vs YAML
environment?`

> I do not have a good workaround for these customers. Multi-env YAML
pipelines will fail. I am surprised there is no report, perhaps because the
regression started only in 2.76 and multi-env is not a regular use case.

Thank you,
Tarun.

On Wed, Oct 28, 2026 at 4:27 PM Valentyn Tymofieiev <va...@google.com>
wrote:

> Thanks, Tarun.
>
> In addition to boot.go, SDK is also looking for some portion of the staged
> files, like pickled_main_session
> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405> (somehow
> we are still using it after cloudpickle migration for new purposes), in
> #1/#2 SDK would need to be aware that there is an assigned worker ID (and
> it is not the same ID anymore as the ID within boot.go, due to sibling
> SDK protocol
> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278>
> ).
>
> We could do a targeted fix just for requirements.txt, which is even more
> narrow and reversible.  Though I think the proper way to solve this is on
> the runner. I understand there is some urgency to have a quick fix as
> multi-env pipelines might be blocked. If those pipelines are blocked
> already, do we have a workaround for affected users, for example to use a
> custom container instead of passing --requirements_file that collides in
> Python vs YAML  environment?
>
> If we have a workaround, I'd suggest to try to prototype a runner side fix
> before rushing in the SDK change.
>
> I think fixing the contract on the runner is a cleaner direction (see #A
> vs #B in my earlier email), though others are welcome to weigh in here.
>
> If there are reasons to believe #B is the way to go (SDK must
> deduplicate), then I'd explore if we can strengthen some guarantees for
> files supplied via --files_to_stage (perhaps via a proto change), so that
> we determine the location for these files in a more deterministic manner,
> possilbly irrespective of the runner.
>

Re: [PROPOSAL] Environment-Sensitive Staging in Beam Python

Posted by Tarun Annapareddy via dev <de...@beam.apache.org>.
+1 I agree Runner fix look like a good longer term fix. It addresses all
SDK's(Java/Python/Go)

On Wed, Oct 28, 2026 at 4:34 PM Valentyn Tymofieiev <va...@google.com>
wrote:

> >  but it will increase worker memory, as each container will now download
> and store dependencies separately
>
> I think with your original proposal to download files into
> semi_persist_dir/worker_id, we will still consume extra disk space since
> each SDK container will be downloading the dependencies separately.
>
> If we intentionally make multiple environments share staged files on the
> host VM, we need to fix a race that can happen between writes and reads in
> different environments: https://github.com/apache/beam/issues/28605 (was
> autoclosed, but it may be relevant again for Python + YAML / XLANG
> environments). This is another reason to consider different host
> directories for different containers.
>
> On Wed, Oct 28, 2026 at 4:26 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Thanks, Tarun.
>>
>> In addition to boot.go, SDK is also looking for some portion of the
>> staged files, like pickled_main_session
>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405> (somehow
>> we are still using it after cloudpickle migration for new purposes), in
>> #1/#2 SDK would need to be aware that there is an assigned worker ID (and
>> it is not the same ID anymore as the ID within boot.go, due to sibling
>> SDK protocol
>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278>
>> ).
>>
>> We could do a targeted fix just for requirements.txt, which is even more
>> narrow and reversible.  Though I think the proper way to solve this is
>> on the runner. I understand there is some urgency to have a quick fix as
>> multi-env pipelines might be blocked. If those pipelines are blocked
>> already, do we have a workaround for affected users, for example to use a
>> custom container instead of passing --requirements_file that collides in
>> Python vs YAML  environment?
>>
>> If we have a workaround, I'd suggest to try to prototype a runner side
>> fix before rushing in the SDK change.
>>
>> I think fixing the contract on the runner is a cleaner direction (see #A
>> vs #B in my earlier email), though others are welcome to weigh in here.
>>
>> If there are reasons to believe #B is the way to go (SDK must
>> deduplicate), then I'd explore if we can strengthen some guarantees for
>> files supplied via --files_to_stage (perhaps via a proto change), so that
>> we determine the location for these files in a more deterministic manner,
>> possilbly irrespective of the runner.
>>
>

Re: [PROPOSAL] Environment-Sensitive Staging in Beam Python

Posted by Valentyn Tymofieiev via dev <de...@beam.apache.org>.
>  but it will increase worker memory, as each container will now download
and store dependencies separately

I think with your original proposal to download files into
semi_persist_dir/worker_id, we will still consume extra disk space since
each SDK container will be downloading the dependencies separately.

If we intentionally make multiple environments share staged files on the
host VM, we need to fix a race that can happen between writes and reads in
different environments: https://github.com/apache/beam/issues/28605 (was
autoclosed, but it may be relevant again for Python + YAML / XLANG
environments). This is another reason to consider different host
directories for different containers.

On Wed, Oct 28, 2026 at 4:26 PM Valentyn Tymofieiev <va...@google.com>
wrote:

> Thanks, Tarun.
>
> In addition to boot.go, SDK is also looking for some portion of the staged
> files, like pickled_main_session
> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405> (somehow
> we are still using it after cloudpickle migration for new purposes), in
> #1/#2 SDK would need to be aware that there is an assigned worker ID (and
> it is not the same ID anymore as the ID within boot.go, due to sibling
> SDK protocol
> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278>
> ).
>
> We could do a targeted fix just for requirements.txt, which is even more
> narrow and reversible.  Though I think the proper way to solve this is on
> the runner. I understand there is some urgency to have a quick fix as
> multi-env pipelines might be blocked. If those pipelines are blocked
> already, do we have a workaround for affected users, for example to use a
> custom container instead of passing --requirements_file that collides in
> Python vs YAML  environment?
>
> If we have a workaround, I'd suggest to try to prototype a runner side fix
> before rushing in the SDK change.
>
> I think fixing the contract on the runner is a cleaner direction (see #A
> vs #B in my earlier email), though others are welcome to weigh in here.
>
> If there are reasons to believe #B is the way to go (SDK must
> deduplicate), then I'd explore if we can strengthen some guarantees for
> files supplied via --files_to_stage (perhaps via a proto change), so that
> we determine the location for these files in a more deterministic manner,
> possilbly irrespective of the runner.
>

Re: [PROPOSAL] Environment-Sensitive Staging in Beam Python

Posted by Valentyn Tymofieiev via dev <de...@beam.apache.org>.
Thanks, Tarun.

In addition to boot.go, SDK is also looking for some portion of the staged
files, like pickled_main_session
<https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405>
(somehow
we are still using it after cloudpickle migration for new purposes), in
#1/#2 SDK would need to be aware that there is an assigned worker ID (and
it is not the same ID anymore as the ID within boot.go, due to sibling SDK
protocol
<https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278>
).

We could do a targeted fix just for requirements.txt, which is even more
narrow and reversible.  Though I think the proper way to solve this is on
the runner. I understand there is some urgency to have a quick fix as
multi-env pipelines might be blocked. If those pipelines are blocked
already, do we have a workaround for affected users, for example to use a
custom container instead of passing --requirements_file that collides in
Python vs YAML  environment?

If we have a workaround, I'd suggest to try to prototype a runner side fix
before rushing in the SDK change.

I think fixing the contract on the runner is a cleaner direction (see #A vs
#B in my earlier email), though others are welcome to weigh in here.

If there are reasons to believe #B is the way to go (SDK must deduplicate),
then I'd explore if we can strengthen some guarantees for files supplied
via --files_to_stage (perhaps via a proto change), so that we determine the
location for these files in a more deterministic manner, possilbly
irrespective of the runner.

Re: [PROPOSAL] Environment-Sensitive Staging in Beam Python

Posted by Chamikara Jayalath via dev <de...@beam.apache.org>.
On Wed, Oct 28, 2026 at 2:13 PM Tarun Annapareddy <ta...@google.com>
wrote:

> Hi everyone,
>
> New findings and discussions are available at go/beam-pr/40699
> <https://goto.google.com/beam-pr/40699>. While the high level idea is to
> align Python dependency management similar to Java(previous work
> <https://github.com/apache/beam/issues/20009>). We realized that some
> Python use cases assumed a fixed path existed. For example --
> files_to_stage
> <https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/>,
> which is used by run_inference
> <https://github.com/apache/beam/blob/master/sdks/java/extensions/python/src/test/java/org/apache/beam/sdk/extensions/python/transforms/RunInferenceTransformTest.java#L116>.
> With this proposal the SEMI_PERSISTENT_DIRECTORY is no longer a static
> path, known during pipeline submission. So, this would break their
> pipelines; they should apply a similar fix (go/beam-pr/40795
> <https://goto.google.com/beam-pr/40795>).
>
> Here are a few options:
>
> 1. [Quick Fix]: Narrow down the scope of problem. Instead of moving all
> the dependencies into separate environment-based paths. We narrow
> down(hardcode) specific files(example
> <https://github.com/apache/beam/blob/master/sdks/python/container/boot.go#L71>,
> which we already do) This will allow us to solve this issue in the latest
> release cut (planned today).
> 2. [SDK fix]: Instead of listing all these files manually in the boot.go.
> Stager marks these files with a separate marker in the pipeline proto. The
> boot process downloads the files into separate directories (shared vs.
> environment-based)
> 3.[Runner fix]: Runner can host different folder, mounted at the same path
> in each container(host /var/opt/google/<container> mounted at
> /var/opt/google) So, there is no conflict between the containers. May be
> with this, we can remove environment-based downloads for Java as well. It
> looks clean, but it will increase worker memory, as each container will now
> download and store dependencies separately.
>

I would choose option 3 to unblock the release.

For the long term fix, option 3 probably has the least disruption for
existing customers who might be using the hard coded staging path for
runners like Dataflow but this probably also the riskiest (changes the way
containers are setup for all SDKs that use the runner). If we choose option
2 we can do a more limited fix just to Python SDK (which is the only SDK
affected) but in that case we should provide a clear util for users to
easily find staged files locally on the workers.

Thanks,
Cham


>
> With all these options, customers using the run_inference
> <https://github.com/apache/beam/blob/master/sdks/java/extensions/python/src/test/java/org/apache/beam/sdk/extensions/python/transforms/RunInferenceTransformTest.java#L116> pattern
> are still un-impacted. I recommend option-1, to unblock multi-env YAML
> pipelines, option-3 needs agreement on the memory trade-off.
>
> Thank you,
> Tarun.
>
>
> On Wed, Oct 21, 2026 at 11:14 AM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> +1, thank you!
>>
>> On Wed, Oct 21, 2026 at 10:56 AM Chamikara Jayalath via dev <
>> dev@beam.apache.org> wrote:
>>
>>> +1. Thanks a lot for looking into this and providing a fix Tarun.
>>>
>>> - Cham
>>>
>>> On Wed, Oct 21, 2026 at 6:41 AM Kenneth Knowles <ke...@apache.org> wrote:
>>>
>>>> Nice catch, and good write-up. Thanks for the clear summary in the doc,
>>>> too.
>>>>
>>>> Kenn
>>>>
>>>> On Tue, Oct 20, 2026 at 11:02 PM Tarun Annapareddy via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Recently, we have encountered Dataflow Multi-Environment pipelines
>>>>> crash-looping due to integrity check failures. However, while debugging we
>>>>> realized it's a long-standing issue in Beam Python dependency staging. This
>>>>> fix is small, but a bit convoluted, so I have created a doc and attached
>>>>> the PR.
>>>>>
>>>>> Doc:
>>>>> https://docs.google.com/document/d/1gNfjbIYNzbv15-RjCj0ad4MHenVPoSjUEcH0Ok3iNyU/edit?usp=sharing
>>>>>
>>>>> PR: go/beam-pr/40699 <https://goto.google.com/beam-pr/40699>
>>>>>
>>>>> Thank you,
>>>>> Tarun.
>>>>>
>>>>

Re: [PROPOSAL] Environment-Sensitive Staging in Beam Python

Posted by Tarun Annapareddy via dev <de...@beam.apache.org>.
Hi everyone,

New findings and discussions are available at go/beam-pr/40699
<https://goto.google.com/beam-pr/40699>. While the high level idea is to
align Python dependency management similar to Java(previous work
<https://github.com/apache/beam/issues/20009>). We realized that some
Python use cases assumed a fixed path existed. For example --files_to_stage
<https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/>,
which is used by run_inference
<https://github.com/apache/beam/blob/master/sdks/java/extensions/python/src/test/java/org/apache/beam/sdk/extensions/python/transforms/RunInferenceTransformTest.java#L116>.
With this proposal the SEMI_PERSISTENT_DIRECTORY is no longer a static
path, known during pipeline submission. So, this would break their
pipelines; they should apply a similar fix (go/beam-pr/40795
<https://goto.google.com/beam-pr/40795>).

Here are a few options:

1. [Quick Fix]: Narrow down the scope of problem. Instead of moving all the
dependencies into separate environment-based paths. We narrow
down(hardcode) specific files(example
<https://github.com/apache/beam/blob/master/sdks/python/container/boot.go#L71>,
which we already do) This will allow us to solve this issue in the latest
release cut (planned today).
2. [SDK fix]: Instead of listing all these files manually in the boot.go.
Stager marks these files with a separate marker in the pipeline proto. The
boot process downloads the files into separate directories (shared vs.
environment-based)
3.[Runner fix]: Runner can host different folder, mounted at the same path
in each container(host /var/opt/google/<container> mounted at
/var/opt/google) So, there is no conflict between the containers. May be
with this, we can remove environment-based downloads for Java as well. It
looks clean, but it will increase worker memory, as each container will now
download and store dependencies separately.

With all these options, customers using the run_inference
<https://github.com/apache/beam/blob/master/sdks/java/extensions/python/src/test/java/org/apache/beam/sdk/extensions/python/transforms/RunInferenceTransformTest.java#L116>
pattern
are still un-impacted. I recommend option-1, to unblock multi-env YAML
pipelines, option-3 needs agreement on the memory trade-off.

Thank you,
Tarun.


On Wed, Oct 21, 2026 at 11:14 AM Valentyn Tymofieiev <va...@google.com>
wrote:

> +1, thank you!
>
> On Wed, Oct 21, 2026 at 10:56 AM Chamikara Jayalath via dev <
> dev@beam.apache.org> wrote:
>
>> +1. Thanks a lot for looking into this and providing a fix Tarun.
>>
>> - Cham
>>
>> On Wed, Oct 21, 2026 at 6:41 AM Kenneth Knowles <ke...@apache.org> wrote:
>>
>>> Nice catch, and good write-up. Thanks for the clear summary in the doc,
>>> too.
>>>
>>> Kenn
>>>
>>> On Tue, Oct 20, 2026 at 11:02 PM Tarun Annapareddy via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> Recently, we have encountered Dataflow Multi-Environment pipelines
>>>> crash-looping due to integrity check failures. However, while debugging we
>>>> realized it's a long-standing issue in Beam Python dependency staging. This
>>>> fix is small, but a bit convoluted, so I have created a doc and attached
>>>> the PR.
>>>>
>>>> Doc:
>>>> https://docs.google.com/document/d/1gNfjbIYNzbv15-RjCj0ad4MHenVPoSjUEcH0Ok3iNyU/edit?usp=sharing
>>>>
>>>> PR: go/beam-pr/40699 <https://goto.google.com/beam-pr/40699>
>>>>
>>>> Thank you,
>>>> Tarun.
>>>>
>>>

Re: [PROPOSAL] Environment-Sensitive Staging in Beam Python

Posted by Valentyn Tymofieiev via dev <de...@beam.apache.org>.
+1, thank you!

On Wed, Oct 21, 2026 at 10:56 AM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:

> +1. Thanks a lot for looking into this and providing a fix Tarun.
>
> - Cham
>
> On Wed, Oct 21, 2026 at 6:41 AM Kenneth Knowles <ke...@apache.org> wrote:
>
>> Nice catch, and good write-up. Thanks for the clear summary in the doc,
>> too.
>>
>> Kenn
>>
>> On Tue, Oct 20, 2026 at 11:02 PM Tarun Annapareddy via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Hi everyone,
>>>
>>> Recently, we have encountered Dataflow Multi-Environment pipelines
>>> crash-looping due to integrity check failures. However, while debugging we
>>> realized it's a long-standing issue in Beam Python dependency staging. This
>>> fix is small, but a bit convoluted, so I have created a doc and attached
>>> the PR.
>>>
>>> Doc:
>>> https://docs.google.com/document/d/1gNfjbIYNzbv15-RjCj0ad4MHenVPoSjUEcH0Ok3iNyU/edit?usp=sharing
>>>
>>> PR: go/beam-pr/40699 <https://goto.google.com/beam-pr/40699>
>>>
>>> Thank you,
>>> Tarun.
>>>
>>

Re: [PROPOSAL] Environment-Sensitive Staging in Beam Python

Posted by Chamikara Jayalath via dev <de...@beam.apache.org>.
+1. Thanks a lot for looking into this and providing a fix Tarun.

- Cham

On Wed, Oct 21, 2026 at 6:41 AM Kenneth Knowles <ke...@apache.org> wrote:

> Nice catch, and good write-up. Thanks for the clear summary in the doc,
> too.
>
> Kenn
>
> On Tue, Oct 20, 2026 at 11:02 PM Tarun Annapareddy via dev <
> dev@beam.apache.org> wrote:
>
>> Hi everyone,
>>
>> Recently, we have encountered Dataflow Multi-Environment pipelines
>> crash-looping due to integrity check failures. However, while debugging we
>> realized it's a long-standing issue in Beam Python dependency staging. This
>> fix is small, but a bit convoluted, so I have created a doc and attached
>> the PR.
>>
>> Doc:
>> https://docs.google.com/document/d/1gNfjbIYNzbv15-RjCj0ad4MHenVPoSjUEcH0Ok3iNyU/edit?usp=sharing
>>
>> PR: go/beam-pr/40699 <https://goto.google.com/beam-pr/40699>
>>
>> Thank you,
>> Tarun.
>>
>

Re: [PROPOSAL] Environment-Sensitive Staging in Beam Python

Posted by Kenneth Knowles <ke...@apache.org>.
Nice catch, and good write-up. Thanks for the clear summary in the doc, too.

Kenn

On Tue, Oct 20, 2026 at 11:02 PM Tarun Annapareddy via dev <
dev@beam.apache.org> wrote:

> Hi everyone,
>
> Recently, we have encountered Dataflow Multi-Environment pipelines
> crash-looping due to integrity check failures. However, while debugging we
> realized it's a long-standing issue in Beam Python dependency staging. This
> fix is small, but a bit convoluted, so I have created a doc and attached
> the PR.
>
> Doc:
> https://docs.google.com/document/d/1gNfjbIYNzbv15-RjCj0ad4MHenVPoSjUEcH0Ok3iNyU/edit?usp=sharing
>
> PR: go/beam-pr/40699 <https://goto.google.com/beam-pr/40699>
>
> Thank you,
> Tarun.
>