You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Ke Wu <ke...@gmail.com> on 2021/02/17 19:24:47 UTC

Implication on PTransform name

Hello everyone,

Is there any documentation on the implication of name param when applying a PTransform on PCollection? Some questions I have on mind is

Is the name required to be unique across the pipeline?
For stateful PTransforms such as stateful ParDo, Combine and Window, if no name is provided, is state preserved during upgrades?
If a stable name is provided,  is state preserved during upgrades?
If a stable name is provided and changed intentionally during upgrade, is state expected to be clearer?

Thanks,
Ke

Re: Implication on PTransform name

Posted by Ke Wu <ke...@gmail.com>.
Thank you Reuven for the answers. I see that the answers are mostly Runner dependent, does it mean, BEAM as SDK, does not specify the intended behavior and it delegates to each runner to interpret the expected behavior?

Thanks,
Ke

> On Feb 17, 2021, at 11:48 AM, Reuven Lax <re...@google.com> wrote:
> 
> 
> 
> On Wed, Feb 17, 2021 at 11:25 AM Ke Wu <ke.wu.cs@gmail.com <ma...@gmail.com>> wrote:
> Hello everyone,
> 
> Is there any documentation on the implication of name param when applying a PTransform on PCollection? Some questions I have on mind is
> 
> Is the name required to be unique across the pipeline?
> No - only needs to be unique within the current scope. 
> For stateful PTransforms such as stateful ParDo, Combine and Window, if no name is provided, is state preserved during upgrades?
> Runner dependent, but generally the name is required to perform upgrades.
>  
> If a stable name is provided,  is state preserved during upgrades?
> Runner dependent, but often yes. 
> If a stable name is provided and changed intentionally during upgrade, is state expected to be clearer?
> Again runner dependent, but for the dataflow runner you currently can pass in a name mapping that maps the old name to the new one. 
> 
> Thanks,
> Ke
> 
>

Re: Implication on PTransform name

Posted by Reuven Lax <re...@google.com>.
On Wed, Feb 17, 2021 at 11:25 AM Ke Wu <ke...@gmail.com> wrote:

> Hello everyone,
>
> Is there any documentation on the implication of name param when applying
> a PTransform on PCollection? Some questions I have on mind is
>
>
>    - Is the name required to be unique across the pipeline?
>
> No - only needs to be unique within the current scope.

>
>    - For stateful PTransforms such as stateful ParDo, Combine and Window,
>    if no name is provided, is state preserved during upgrades?
>
> Runner dependent, but generally the name is required to perform upgrades.


>
>    - If a stable name is provided,  is state preserved during upgrades?
>
> Runner dependent, but often yes.

>
>    - If a stable name is provided and changed intentionally during
>    upgrade, is state expected to be clearer?
>
> Again runner dependent, but for the dataflow runner you currently can pass
in a name mapping that maps the old name to the new one.

>
> Thanks,
> Ke
>
>
>