You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Fernando Morales Martinez <fe...@wizeline.com> on 2021/03/16 19:41:28 UTC
SDK improvements (BEAM-3304)
*Hi team,*
We’re working on WI https://issues.apache.org/jira/browse/BEAM-3304 and
have a couple of questions.
As we understand it, the Proto compiler based on file beam_runner_api.proto
creates the whole RunnerApi for Java (RunnerApi.class) and for Go
(beam_runner_api.pb.go)
TriggerTranslation.java uses several methods from the RunnerApi to
translate from Trigger to Proto and viceversa which then get used for
windowing strategies.
Now, if we’re on the right path, what needs to be done as part of the
*BEAM-3304* WI is
- implement, for example, the AfterPane (called trigger ElementCount in
Go) and
- implement the TriggerTranslation code in Go along with the necessary
plumbing, in order to expose that trigger to the user.
- Implement WindowingStrategy in Go
(sdks/go/pkg/beam/core/graph/window/strategy.go)
Is this correct? Is there some important piece I’m missing?
Thanks for the help!
--
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*
Re: SDK improvements (BEAM-3304)
Posted by Fernando Morales Martinez <fe...@wizeline.com>.
Thanks again Robert!
We will start with the implementation of AfterPane and the WindowingStrategy
(core/graph/window/strategy.go)
- Fernando Morales
On Tue, Mar 16, 2021 at 1:02 PM Robert Burke <ro...@frantil.com> wrote:
> Hello Fernando!
>
> That should be correct.
>
> The protos are used to represent the user pipeline in a portable way, and
> handle the serialization of such. However the only time the SDKs actually
> need deal with the protos is the "last mile", but this is a per SDK choice.
>
> The Go SDK is mostly set up to avoid direct coupling of the exec
> constructs and construction constructs and the resulting protos, in the
> event that the Beam protos change dramatically in a breaking version.
>
> Translating construction time constructs to the protos is in
> graphx/translate.go . If a worker needs to execute something as a result,
> the reverse translation to execution time constructs is in
> exec/translate.go. Not everything has both construction time and runtime
> constructions, hence the split translation approach.
>
> Let me know if you have any Go SDK implementation side questions. I'm
> happy to help.
>
> Robert Burke (@lostluck)
>
>
> On Tue, Mar 16, 2021, 12:41 PM Fernando Morales Martinez <
> fernando.morales@wizeline.com> wrote:
>
>> *Hi team,*
>>
>>
>> We’re working on WI https://issues.apache.org/jira/browse/BEAM-3304 and
>> have a couple of questions.
>>
>> As we understand it, the Proto compiler based on file
>> beam_runner_api.proto creates the whole RunnerApi for Java
>> (RunnerApi.class) and for Go (beam_runner_api.pb.go)
>>
>>
>> TriggerTranslation.java uses several methods from the RunnerApi to
>> translate from Trigger to Proto and viceversa which then get used for
>> windowing strategies.
>>
>> Now, if we’re on the right path, what needs to be done as part of the
>> *BEAM-3304* WI is
>>
>>
>>
>> - implement, for example, the AfterPane (called trigger ElementCount
>> in Go) and
>> - implement the TriggerTranslation code in Go along with the
>> necessary plumbing, in order to expose that trigger to the user.
>> - Implement WindowingStrategy in Go
>> (sdks/go/pkg/beam/core/graph/window/strategy.go)
>>
>>
>> Is this correct? Is there some important piece I’m missing?
>>
>>
>> Thanks for the help!
>>
>>
>>
>>
>>
>>
>>
>>
>> *This email and its contents (including any attachments) are being sent
>> toyou on the condition of confidentiality and may be protected by
>> legalprivilege. Access to this email by anyone other than the intended
>> recipientis unauthorized. If you are not the intended recipient, please
>> immediatelynotify the sender by replying to this message and delete the
>> materialimmediately from your system. Any further use, dissemination,
>> distributionor reproduction of this email is strictly prohibited. Further,
>> norepresentation is made with respect to any content contained in this
>> email.*
>
>
--
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*
Re: SDK improvements (BEAM-3304)
Posted by Robert Burke <ro...@frantil.com>.
Hello Fernando!
That should be correct.
The protos are used to represent the user pipeline in a portable way, and
handle the serialization of such. However the only time the SDKs actually
need deal with the protos is the "last mile", but this is a per SDK choice.
The Go SDK is mostly set up to avoid direct coupling of the exec constructs
and construction constructs and the resulting protos, in the event that the
Beam protos change dramatically in a breaking version.
Translating construction time constructs to the protos is in
graphx/translate.go . If a worker needs to execute something as a result,
the reverse translation to execution time constructs is in
exec/translate.go. Not everything has both construction time and runtime
constructions, hence the split translation approach.
Let me know if you have any Go SDK implementation side questions. I'm happy
to help.
Robert Burke (@lostluck)
On Tue, Mar 16, 2021, 12:41 PM Fernando Morales Martinez <
fernando.morales@wizeline.com> wrote:
> *Hi team,*
>
>
> We’re working on WI https://issues.apache.org/jira/browse/BEAM-3304 and
> have a couple of questions.
>
> As we understand it, the Proto compiler based on file
> beam_runner_api.proto creates the whole RunnerApi for Java
> (RunnerApi.class) and for Go (beam_runner_api.pb.go)
>
>
> TriggerTranslation.java uses several methods from the RunnerApi to
> translate from Trigger to Proto and viceversa which then get used for
> windowing strategies.
>
> Now, if we’re on the right path, what needs to be done as part of the
> *BEAM-3304* WI is
>
>
>
> - implement, for example, the AfterPane (called trigger ElementCount
> in Go) and
> - implement the TriggerTranslation code in Go along with the necessary
> plumbing, in order to expose that trigger to the user.
> - Implement WindowingStrategy in Go
> (sdks/go/pkg/beam/core/graph/window/strategy.go)
>
>
> Is this correct? Is there some important piece I’m missing?
>
>
> Thanks for the help!
>
>
>
>
>
>
>
>
> *This email and its contents (including any attachments) are being sent
> toyou on the condition of confidentiality and may be protected by
> legalprivilege. Access to this email by anyone other than the intended
> recipientis unauthorized. If you are not the intended recipient, please
> immediatelynotify the sender by replying to this message and delete the
> materialimmediately from your system. Any further use, dissemination,
> distributionor reproduction of this email is strictly prohibited. Further,
> norepresentation is made with respect to any content contained in this
> email.*