You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@beam.apache.org by Anand Inguva <an...@google.com> on 2022/01/19 01:33:41 UTC

Stage wheels of the pipeline dependencies for default container images

Hi all,

We are planning to download and stage wheels instead of source
distributions when the user provides a requirements.txt and uses apache
beam default container image.

I would like to have a discussion as this would bring a change in the
behavior of how we stage the dependencies.

Please review the [1]. I would like to get some feedback and also any other
ideas/solutions for this problem.


Jira ticket linked to the issue:
https://issues.apache.org/jira/browse/BEAM-4032
Prior discussion:
https://lists.apache.org/thread/3gn4d9kxo6o7qmlknc0ylb14xkpylzr8

Thank you,
Anand


[1]
https://docs.google.com/document/d/1htXDZwg7WcCB-ZyNuy91qWuccaQ6cjX4lY4l4Cqb8P0/edit?usp=sharing

Re: Stage wheels of the pipeline dependencies for default container images

Posted by Anand Inguva <an...@google.com>.

Hi,
We added more context on the problem in the doc[1] along with possible
solutions. Feel free to add any comments or suggestions.

Thanks,
Anand

[1]
https://docs.google.com/document/d/1htXDZwg7WcCB-ZyNuy91qWuccaQ6cjX4lY4l4Cqb8P0/edit?usp=sharing

On Thu, Jan 20, 2022 at 5:22 PM Pablo Estrada <pa...@google.com> wrote:

> Added ac couple comments. Thanks Anand: )
>
> On Tue, Jan 18, 2022 at 9:02 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Thanks Anand!
>>
>> Taking a look at the document and will add some additional context there.
>>
>> On Tue, Jan 18, 2022 at 5:34 PM Anand Inguva <an...@google.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> We are planning to download and stage wheels instead of source
>>> distributions when the user provides a requirements.txt and uses apache
>>> beam default container image.
>>>
>>> I would like to have a discussion as this would bring a change in the
>>> behavior of how we stage the dependencies.
>>>
>>> Please review the [1]. I would like to get some feedback and also any
>>> other ideas/solutions for this problem.
>>>
>>>
>>> Jira ticket linked to the issue:
>>> https://issues.apache.org/jira/browse/BEAM-4032
>>> Prior discussion:
>>> https://lists.apache.org/thread/3gn4d9kxo6o7qmlknc0ylb14xkpylzr8
>>>
>>> Thank you,
>>> Anand
>>>
>>>
>>> [1]
>>> https://docs.google.com/document/d/1htXDZwg7WcCB-ZyNuy91qWuccaQ6cjX4lY4l4Cqb8P0/edit?usp=sharing
>>>
>>

Re: Stage wheels of the pipeline dependencies for default container images

Posted by Pablo Estrada <pa...@google.com>.

Added ac couple comments. Thanks Anand: )

On Tue, Jan 18, 2022 at 9:02 PM Valentyn Tymofieiev <va...@google.com>
wrote:

> Thanks Anand!
>
> Taking a look at the document and will add some additional context there.
>
> On Tue, Jan 18, 2022 at 5:34 PM Anand Inguva <an...@google.com>
> wrote:
>
>> Hi all,
>>
>> We are planning to download and stage wheels instead of source
>> distributions when the user provides a requirements.txt and uses apache
>> beam default container image.
>>
>> I would like to have a discussion as this would bring a change in the
>> behavior of how we stage the dependencies.
>>
>> Please review the [1]. I would like to get some feedback and also any
>> other ideas/solutions for this problem.
>>
>>
>> Jira ticket linked to the issue:
>> https://issues.apache.org/jira/browse/BEAM-4032
>> Prior discussion:
>> https://lists.apache.org/thread/3gn4d9kxo6o7qmlknc0ylb14xkpylzr8
>>
>> Thank you,
>> Anand
>>
>>
>> [1]
>> https://docs.google.com/document/d/1htXDZwg7WcCB-ZyNuy91qWuccaQ6cjX4lY4l4Cqb8P0/edit?usp=sharing
>>
>

Re: Stage wheels of the pipeline dependencies for default container images

Posted by Valentyn Tymofieiev <va...@google.com>.

Thanks Anand!

Taking a look at the document and will add some additional context there.

On Tue, Jan 18, 2022 at 5:34 PM Anand Inguva <an...@google.com> wrote:

> Hi all,
>
> We are planning to download and stage wheels instead of source
> distributions when the user provides a requirements.txt and uses apache
> beam default container image.
>
> I would like to have a discussion as this would bring a change in the
> behavior of how we stage the dependencies.
>
> Please review the [1]. I would like to get some feedback and also any
> other ideas/solutions for this problem.
>
>
> Jira ticket linked to the issue:
> https://issues.apache.org/jira/browse/BEAM-4032
> Prior discussion:
> https://lists.apache.org/thread/3gn4d9kxo6o7qmlknc0ylb14xkpylzr8
>
> Thank you,
> Anand
>
>
> [1]
> https://docs.google.com/document/d/1htXDZwg7WcCB-ZyNuy91qWuccaQ6cjX4lY4l4Cqb8P0/edit?usp=sharing
>