You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@beam.apache.org by Jeff Kinard <je...@thekinards.com> on 2024/03/19 18:51:10 UTC

Design proposal for Beam YAML templates

Hi all,

I have another quick design doc discussing the syntax for Beam YAML
templates. This feature would allow a user to create a template pipelines
similar to what is done in
https://github.com/GoogleCloudPlatform/DataflowTemplates
that are configurable and therefore more reusable than static YAML
pipelines.

Design doc:
https://docs.google.com/document/d/1AUt4NEoQCBrOEVhadVozyTU2-CqC-pw7kuK6rzIA6Jc/edit?resourcekey=0-d-aVmsKZ89SzSiVwt8n5pw

Please take a look and feel free to leave any comments.

Thanks,
- Jeff

Re: Design proposal for Beam YAML templates

Posted by Robert Bradshaw via dev <de...@beam.apache.org>.

It looks like we're converging on an optional jinja preprocessing phase to
handle this. I'm in favor of this solution.

On Wed, Mar 20, 2024 at 9:23 AM Robert Bradshaw <ro...@google.com> wrote:

> Thanks. I think this will be a very powerful feature. Left some comments
> on the doc.
>
> On Tue, Mar 19, 2024 at 11:53 AM Jeff Kinard <je...@thekinards.com> wrote:
>
>> Hi all,
>>
>> I have another quick design doc discussing the syntax for Beam YAML
>> templates. This feature would allow a user to create a template pipelines
>> similar to what is done in
>> https://github.com/GoogleCloudPlatform/DataflowTemplates
>> that are configurable and therefore more reusable than static YAML
>> pipelines.
>>
>> Design doc:
>> https://docs.google.com/document/d/1AUt4NEoQCBrOEVhadVozyTU2-CqC-pw7kuK6rzIA6Jc/edit?resourcekey=0-d-aVmsKZ89SzSiVwt8n5pw
>>
>> Please take a look and feel free to leave any comments.
>>
>> Thanks,
>> - Jeff
>>
>

Re: Design proposal for Beam YAML templates

Posted by Robert Bradshaw via dev <de...@beam.apache.org>.

Thanks. I think this will be a very powerful feature. Left some comments on
the doc.

On Tue, Mar 19, 2024 at 11:53 AM Jeff Kinard <je...@thekinards.com> wrote:

> Hi all,
>
> I have another quick design doc discussing the syntax for Beam YAML
> templates. This feature would allow a user to create a template pipelines
> similar to what is done in
> https://github.com/GoogleCloudPlatform/DataflowTemplates
> that are configurable and therefore more reusable than static YAML
> pipelines.
>
> Design doc:
> https://docs.google.com/document/d/1AUt4NEoQCBrOEVhadVozyTU2-CqC-pw7kuK6rzIA6Jc/edit?resourcekey=0-d-aVmsKZ89SzSiVwt8n5pw
>
> Please take a look and feel free to leave any comments.
>
> Thanks,
> - Jeff
>