You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Danny McCormick <da...@google.com> on 2022/02/24 15:42:08 UTC

[RFC][Design] Bundle Finalization in the Go Sdk

Hey everyone, I put together a design doc for adding Bundle Finalization in
the Go Sdk and would appreciate any thoughts you have!

Bundle finalization enables a DoFn to perform side effects after a runner
has acknowledged that it has durably persisted the output. Right now, Java
and Python support bundle finalization by allowing a user to register a
callback function which is invoked when the runner acknowledges that it has
persisted all output, but go does not have any such support. This proposes
an approach for adding bundle finalization in the Go Sdk by allowing users
to register callbacks during their normal DoFn execution. It generally
mirrors the behavior adopted by other languages and allows users to
dynamically specify their callbacks at runtime.

Please share any feedback or support here:
https://docs.google.com/document/d/1dLylt36oFhsWfyBaqPayYXqYHCICNrSZ6jmr51eqZ4k/edit?usp=sharing

Thanks,
Danny

Re: [RFC][Design] Bundle Finalization in the Go Sdk

Posted by Jack McCluskey <jr...@google.com>.
Hey Danny,

Great doc! Don't have much to add beyond the current questions, looking
forward to seeing bundle finalization added!

On Thu, Feb 24, 2022 at 11:30 AM Robert Burke <ro...@frantil.com> wrote:

> Thanks Danny!
>
> I did a first pass on comments, but do like the approach. Needs some
> justifications on why this path should be chosen over alternative
> implementations.
>
>
> On Thu, Feb 24, 2022, 7:42 AM Danny McCormick <da...@google.com>
> wrote:
>
>> Hey everyone, I put together a design doc for adding Bundle Finalization
>> in the Go Sdk and would appreciate any thoughts you have!
>>
>> Bundle finalization enables a DoFn to perform side effects after a runner
>> has acknowledged that it has durably persisted the output. Right now, Java
>> and Python support bundle finalization by allowing a user to register a
>> callback function which is invoked when the runner acknowledges that it has
>> persisted all output, but go does not have any such support. This proposes
>> an approach for adding bundle finalization in the Go Sdk by allowing users
>> to register callbacks during their normal DoFn execution. It generally
>> mirrors the behavior adopted by other languages and allows users to
>> dynamically specify their callbacks at runtime.
>>
>> Please share any feedback or support here:
>> https://docs.google.com/document/d/1dLylt36oFhsWfyBaqPayYXqYHCICNrSZ6jmr51eqZ4k/edit?usp=sharing
>>
>> Thanks,
>> Danny
>>
>

Re: [RFC][Design] Bundle Finalization in the Go Sdk

Posted by Robert Burke <ro...@frantil.com>.
Thanks Danny!

I did a first pass on comments, but do like the approach. Needs some
justifications on why this path should be chosen over alternative
implementations.


On Thu, Feb 24, 2022, 7:42 AM Danny McCormick <da...@google.com>
wrote:

> Hey everyone, I put together a design doc for adding Bundle Finalization
> in the Go Sdk and would appreciate any thoughts you have!
>
> Bundle finalization enables a DoFn to perform side effects after a runner
> has acknowledged that it has durably persisted the output. Right now, Java
> and Python support bundle finalization by allowing a user to register a
> callback function which is invoked when the runner acknowledges that it has
> persisted all output, but go does not have any such support. This proposes
> an approach for adding bundle finalization in the Go Sdk by allowing users
> to register callbacks during their normal DoFn execution. It generally
> mirrors the behavior adopted by other languages and allows users to
> dynamically specify their callbacks at runtime.
>
> Please share any feedback or support here:
> https://docs.google.com/document/d/1dLylt36oFhsWfyBaqPayYXqYHCICNrSZ6jmr51eqZ4k/edit?usp=sharing
>
> Thanks,
> Danny
>