/opt/userowned/ folder

[Eila Oriel Research <ei...@orielresearch.org>]

Hello,

We used /opt/userowned/ drive on the workers to copy files and installation
vi setup.py

We have noticed that /opt/userowned/ is not available anymore. The only
folder under opt is google.

Please let me know if there is any folder "dedicated" for installation /
external files.

Thanks,
-- 
Eila Arich

Founder, CEO

Oriel Research Therapeutics

https://www.orielresearch.com/blog
eila@orielresearch.com
www.orielresearch.com
Cambridge, MA
[image: twitter] <https://twitter.com/eilalan1>
[image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
[image: instagram] <https://www.instagram.com/eilalan/>

Re: /opt/userowned/ folder

[Eila Oriel Research <ei...@orielresearch.org>]

Thank you for the clarification.
We will follow the folder creation recommendation.
All the best,
Eila

On Fri, Mar 25, 2022 at 1:49 PM Luke Cwik <lc...@google.com> wrote:

> /opt/userowned was slightly more as it gave access to the VMs filesystem
> outside of the container. This was done due to space limits within
> Dataflows containers preventing arbitrarily large files. This was done
> since custom containers were not supported for Dataflow runner v1. Now with
> custom containers this is no longer an issue and yes if the users files fit
> within the container they could create the /op/userowned directory
> themselves.
>
> On Fri, Mar 25, 2022 at 10:43 AM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> There's no reason we couldn't create a /opt/userowned in the runner v2
>> (aka portable) container as well for backwards compatibility (though in
>> some sense, the user owns the full container rather than just a
>> subdirectory in this case). Any reason not to do that? Should we try to get
>> this into the next release?
>>
>> A short term workaround would be to simply create /opt/userowned in your
>> script.
>>
>>
>> On Fri, Mar 25, 2022 at 10:03 AM Luke Cwik <lc...@google.com> wrote:
>>
>>> /opt/userowned was for Dataflow runner v1 to store extra data and a
>>> short term solution which was replaced with custom containers that is
>>> now GA for Python for runner v2 and it looks like your pipeline(s) have
>>> been opted into runner v2.
>>>
>>> You can restore the existing behavior by opting out of using runner v2
>>> by passing in the experiment:
>>> --experiments=disable_runner_v2
>>>
>>> It would be worthwhile to swap to use custom containers instead of using
>>> /opt/userowned because eventually runner v1 will be unsupported.
>>>
>>> 1: https://cloud.google.com/dataflow/docs/guides/using-custom-containers
>>>
>>> On Fri, Mar 25, 2022 at 5:19 AM Eila Oriel Research <
>>> eila@orielresearch.org> wrote:
>>>
>>>> Thank you. I am trying to figure it out too. most of my pipelines are
>>>> relying on setup.py work on the workers
>>>> Eila
>>>>
>>>> On Thu, Mar 24, 2022 at 10:31 PM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>>> Adding people who might have knowledge: @Reza Rokni <re...@google.com> @Valentyn
>>>>> Tymofieiev <va...@google.com>
>>>>>
>>>>> On Thu, Mar 24, 2022 at 7:08 PM Eila Oriel Research <
>>>>> eila@orielresearch.org> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> We used /opt/userowned/ drive on the workers to copy files and
>>>>>> installation vi setup.py
>>>>>>
>>>>>> We have noticed that /opt/userowned/ is not available anymore. The
>>>>>> only folder under opt is google.
>>>>>>
>>>>>> Please let me know if there is any folder "dedicated" for
>>>>>> installation / external files.
>>>>>>
>>>>>> Thanks,
>>>>>> --
>>>>>> Eila Arich
>>>>>>
>>>>>> Founder, CEO
>>>>>>
>>>>>> Oriel Research Therapeutics
>>>>>>
>>>>>> https://www.orielresearch.com/blog
>>>>>> eila@orielresearch.com
>>>>>> www.orielresearch.com
>>>>>> Cambridge, MA
>>>>>> [image: twitter] <https://twitter.com/eilalan1>
>>>>>> [image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
>>>>>> [image: instagram] <https://www.instagram.com/eilalan/>
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Eila Arich
>>>>
>>>> Founder, CEO
>>>>
>>>> Oriel Research Therapeutics
>>>>
>>>> https://www.orielresearch.com/blog
>>>> eila@orielresearch.com
>>>> www.orielresearch.com
>>>> Cambridge, MA
>>>> [image: twitter] <https://twitter.com/eilalan1>
>>>> [image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
>>>> [image: instagram] <https://www.instagram.com/eilalan/>
>>>>
>>>>

-- 
Eila Arich

Founder, CEO

Oriel Research Therapeutics

https://www.orielresearch.com/blog
eila@orielresearch.com
www.orielresearch.com
Cambridge, MA
[image: twitter] <https://twitter.com/eilalan1>
[image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
[image: instagram] <https://www.instagram.com/eilalan/>

Re: /opt/userowned/ folder

[Luke Cwik <lc...@google.com>]

/opt/userowned was slightly more as it gave access to the VMs filesystem
outside of the container. This was done due to space limits within
Dataflows containers preventing arbitrarily large files. This was done
since custom containers were not supported for Dataflow runner v1. Now with
custom containers this is no longer an issue and yes if the users files fit
within the container they could create the /op/userowned directory
themselves.

On Fri, Mar 25, 2022 at 10:43 AM Robert Bradshaw <ro...@google.com>
wrote:

> There's no reason we couldn't create a /opt/userowned in the runner v2
> (aka portable) container as well for backwards compatibility (though in
> some sense, the user owns the full container rather than just a
> subdirectory in this case). Any reason not to do that? Should we try to get
> this into the next release?
>
> A short term workaround would be to simply create /opt/userowned in your
> script.
>
>
> On Fri, Mar 25, 2022 at 10:03 AM Luke Cwik <lc...@google.com> wrote:
>
>> /opt/userowned was for Dataflow runner v1 to store extra data and a short
>> term solution which was replaced with custom containers that is now GA for
>> Python for runner v2 and it looks like your pipeline(s) have been opted
>> into runner v2.
>>
>> You can restore the existing behavior by opting out of using runner v2 by
>> passing in the experiment:
>> --experiments=disable_runner_v2
>>
>> It would be worthwhile to swap to use custom containers instead of using
>> /opt/userowned because eventually runner v1 will be unsupported.
>>
>> 1: https://cloud.google.com/dataflow/docs/guides/using-custom-containers
>>
>> On Fri, Mar 25, 2022 at 5:19 AM Eila Oriel Research <
>> eila@orielresearch.org> wrote:
>>
>>> Thank you. I am trying to figure it out too. most of my pipelines are
>>> relying on setup.py work on the workers
>>> Eila
>>>
>>> On Thu, Mar 24, 2022 at 10:31 PM Ahmet Altay <al...@google.com> wrote:
>>>
>>>> Adding people who might have knowledge: @Reza Rokni <re...@google.com> @Valentyn
>>>> Tymofieiev <va...@google.com>
>>>>
>>>> On Thu, Mar 24, 2022 at 7:08 PM Eila Oriel Research <
>>>> eila@orielresearch.org> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> We used /opt/userowned/ drive on the workers to copy files and
>>>>> installation vi setup.py
>>>>>
>>>>> We have noticed that /opt/userowned/ is not available anymore. The
>>>>> only folder under opt is google.
>>>>>
>>>>> Please let me know if there is any folder "dedicated" for installation
>>>>> / external files.
>>>>>
>>>>> Thanks,
>>>>> --
>>>>> Eila Arich
>>>>>
>>>>> Founder, CEO
>>>>>
>>>>> Oriel Research Therapeutics
>>>>>
>>>>> https://www.orielresearch.com/blog
>>>>> eila@orielresearch.com
>>>>> www.orielresearch.com
>>>>> Cambridge, MA
>>>>> [image: twitter] <https://twitter.com/eilalan1>
>>>>> [image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
>>>>> [image: instagram] <https://www.instagram.com/eilalan/>
>>>>>
>>>>>
>>>
>>> --
>>> Eila Arich
>>>
>>> Founder, CEO
>>>
>>> Oriel Research Therapeutics
>>>
>>> https://www.orielresearch.com/blog
>>> eila@orielresearch.com
>>> www.orielresearch.com
>>> Cambridge, MA
>>> [image: twitter] <https://twitter.com/eilalan1>
>>> [image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
>>> [image: instagram] <https://www.instagram.com/eilalan/>
>>>
>>>

Re: /opt/userowned/ folder

[Robert Bradshaw <ro...@google.com>]

There's no reason we couldn't create a /opt/userowned in the runner v2 (aka
portable) container as well for backwards compatibility (though in some
sense, the user owns the full container rather than just a subdirectory in
this case). Any reason not to do that? Should we try to get this into the
next release?

A short term workaround would be to simply create /opt/userowned in your
script.


On Fri, Mar 25, 2022 at 10:03 AM Luke Cwik <lc...@google.com> wrote:

> /opt/userowned was for Dataflow runner v1 to store extra data and a short
> term solution which was replaced with custom containers that is now GA for
> Python for runner v2 and it looks like your pipeline(s) have been opted
> into runner v2.
>
> You can restore the existing behavior by opting out of using runner v2 by
> passing in the experiment:
> --experiments=disable_runner_v2
>
> It would be worthwhile to swap to use custom containers instead of using
> /opt/userowned because eventually runner v1 will be unsupported.
>
> 1: https://cloud.google.com/dataflow/docs/guides/using-custom-containers
>
> On Fri, Mar 25, 2022 at 5:19 AM Eila Oriel Research <
> eila@orielresearch.org> wrote:
>
>> Thank you. I am trying to figure it out too. most of my pipelines are
>> relying on setup.py work on the workers
>> Eila
>>
>> On Thu, Mar 24, 2022 at 10:31 PM Ahmet Altay <al...@google.com> wrote:
>>
>>> Adding people who might have knowledge: @Reza Rokni <re...@google.com> @Valentyn
>>> Tymofieiev <va...@google.com>
>>>
>>> On Thu, Mar 24, 2022 at 7:08 PM Eila Oriel Research <
>>> eila@orielresearch.org> wrote:
>>>
>>>> Hello,
>>>>
>>>> We used /opt/userowned/ drive on the workers to copy files and
>>>> installation vi setup.py
>>>>
>>>> We have noticed that /opt/userowned/ is not available anymore. The only
>>>> folder under opt is google.
>>>>
>>>> Please let me know if there is any folder "dedicated" for installation
>>>> / external files.
>>>>
>>>> Thanks,
>>>> --
>>>> Eila Arich
>>>>
>>>> Founder, CEO
>>>>
>>>> Oriel Research Therapeutics
>>>>
>>>> https://www.orielresearch.com/blog
>>>> eila@orielresearch.com
>>>> www.orielresearch.com
>>>> Cambridge, MA
>>>> [image: twitter] <https://twitter.com/eilalan1>
>>>> [image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
>>>> [image: instagram] <https://www.instagram.com/eilalan/>
>>>>
>>>>
>>
>> --
>> Eila Arich
>>
>> Founder, CEO
>>
>> Oriel Research Therapeutics
>>
>> https://www.orielresearch.com/blog
>> eila@orielresearch.com
>> www.orielresearch.com
>> Cambridge, MA
>> [image: twitter] <https://twitter.com/eilalan1>
>> [image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
>> [image: instagram] <https://www.instagram.com/eilalan/>
>>
>>

Re: /opt/userowned/ folder

[Luke Cwik <lc...@google.com>]

/opt/userowned was for Dataflow runner v1 to store extra data and a short
term solution which was replaced with custom containers that is now GA for
Python for runner v2 and it looks like your pipeline(s) have been opted
into runner v2.

You can restore the existing behavior by opting out of using runner v2 by
passing in the experiment:
--experiments=disable_runner_v2

It would be worthwhile to swap to use custom containers instead of using
/opt/userowned because eventually runner v1 will be unsupported.

1: https://cloud.google.com/dataflow/docs/guides/using-custom-containers

On Fri, Mar 25, 2022 at 5:19 AM Eila Oriel Research <ei...@orielresearch.org>
wrote:

> Thank you. I am trying to figure it out too. most of my pipelines are
> relying on setup.py work on the workers
> Eila
>
> On Thu, Mar 24, 2022 at 10:31 PM Ahmet Altay <al...@google.com> wrote:
>
>> Adding people who might have knowledge: @Reza Rokni <re...@google.com> @Valentyn
>> Tymofieiev <va...@google.com>
>>
>> On Thu, Mar 24, 2022 at 7:08 PM Eila Oriel Research <
>> eila@orielresearch.org> wrote:
>>
>>> Hello,
>>>
>>> We used /opt/userowned/ drive on the workers to copy files and
>>> installation vi setup.py
>>>
>>> We have noticed that /opt/userowned/ is not available anymore. The only
>>> folder under opt is google.
>>>
>>> Please let me know if there is any folder "dedicated" for installation /
>>> external files.
>>>
>>> Thanks,
>>> --
>>> Eila Arich
>>>
>>> Founder, CEO
>>>
>>> Oriel Research Therapeutics
>>>
>>> https://www.orielresearch.com/blog
>>> eila@orielresearch.com
>>> www.orielresearch.com
>>> Cambridge, MA
>>> [image: twitter] <https://twitter.com/eilalan1>
>>> [image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
>>> [image: instagram] <https://www.instagram.com/eilalan/>
>>>
>>>
>
> --
> Eila Arich
>
> Founder, CEO
>
> Oriel Research Therapeutics
>
> https://www.orielresearch.com/blog
> eila@orielresearch.com
> www.orielresearch.com
> Cambridge, MA
> [image: twitter] <https://twitter.com/eilalan1>
> [image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
> [image: instagram] <https://www.instagram.com/eilalan/>
>
>

Re: /opt/userowned/ folder

[Eila Oriel Research <ei...@orielresearch.org>]

Thank you. I am trying to figure it out too. most of my pipelines are
relying on setup.py work on the workers
Eila

On Thu, Mar 24, 2022 at 10:31 PM Ahmet Altay <al...@google.com> wrote:

> Adding people who might have knowledge: @Reza Rokni <re...@google.com> @Valentyn
> Tymofieiev <va...@google.com>
>
> On Thu, Mar 24, 2022 at 7:08 PM Eila Oriel Research <
> eila@orielresearch.org> wrote:
>
>> Hello,
>>
>> We used /opt/userowned/ drive on the workers to copy files and
>> installation vi setup.py
>>
>> We have noticed that /opt/userowned/ is not available anymore. The only
>> folder under opt is google.
>>
>> Please let me know if there is any folder "dedicated" for installation /
>> external files.
>>
>> Thanks,
>> --
>> Eila Arich
>>
>> Founder, CEO
>>
>> Oriel Research Therapeutics
>>
>> https://www.orielresearch.com/blog
>> eila@orielresearch.com
>> www.orielresearch.com
>> Cambridge, MA
>> [image: twitter] <https://twitter.com/eilalan1>
>> [image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
>> [image: instagram] <https://www.instagram.com/eilalan/>
>>
>>

-- 
Eila Arich

Founder, CEO

Oriel Research Therapeutics

https://www.orielresearch.com/blog
eila@orielresearch.com
www.orielresearch.com
Cambridge, MA
[image: twitter] <https://twitter.com/eilalan1>
[image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
[image: instagram] <https://www.instagram.com/eilalan/>

Re: /opt/userowned/ folder

[Ahmet Altay <al...@google.com>]

Adding people who might have knowledge: @Reza Rokni <re...@google.com> @Valentyn
Tymofieiev <va...@google.com>

On Thu, Mar 24, 2022 at 7:08 PM Eila Oriel Research <ei...@orielresearch.org>
wrote:

> Hello,
>
> We used /opt/userowned/ drive on the workers to copy files and
> installation vi setup.py
>
> We have noticed that /opt/userowned/ is not available anymore. The only
> folder under opt is google.
>
> Please let me know if there is any folder "dedicated" for installation /
> external files.
>
> Thanks,
> --
> Eila Arich
>
> Founder, CEO
>
> Oriel Research Therapeutics
>
> https://www.orielresearch.com/blog
> eila@orielresearch.com
> www.orielresearch.com
> Cambridge, MA
> [image: twitter] <https://twitter.com/eilalan1>
> [image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
> [image: instagram] <https://www.instagram.com/eilalan/>
>
>