[GitHub] [beam] lostluck commented on issue #23106: [Feature Request][Go SDK]: Support Slowly Changing Side Input Pattern

[GitBox <gi...@apache.org>]

lostluck commented on issue #23106:
URL: https://github.com/apache/beam/issues/23106#issuecomment-1317932566

   OK, I love that we can do charts in the markdown of an issue. Spectacular!
   
   It's a matter of implementing the URN in the switch statement in
   exec/translate.go:, decoding the appropriate payload proto, and using that
   information to configure an exec node that exists on the exec side of the
   pipeline, in this case, taking an timestamp, and producing the appropriate
   window for it, based on the provided windowing strategy (I assume).
   Specific payload (likely a specific proto message) should be listed in the
   proto next to the URN enum in beam_fn_api.proto.
   
   It's not particularly difficult, just a small pain to test, since I don't
   know how well the various semantics are implemented for portable use,
   outside of Dataflow.  Flink should be able to do it, and I recommend that
   for local iteration. Portable Python has issues with streaming semantics.
   Go Direct runner, is right out at present (and is in the process of being
   replaced with a portable implementation because Direct Runners have
   problems.)
   
   So, my recommendation would be to add an integration test to the primitives
   package (
   https://github.com/apache/beam/blob/master/sdks/go/test/integration/primitives/windowinto.go).
   The simple pipeline you have there is a great base and the implementer can
   follow the patterns with the existing integration tests, assuming it can be
   done in a batch mode. But then it can be run against a few of the runners
   at once *and then* the changes implement the changes in the SDK side.  The
   test can then be filtered out from non-compliant runners in
   https://github.com/apache/beam/blob/master/sdks/go/test/integration/integration.go#L63,
   as determined by "what fails after it's working well on Dataflow and Flink".
   
   That slows down iteration time a bit (outside of simple unit testing), but
   ends up with something that will be tested and work on all the runners,
   without spending money on Dataflow, which shouldn't be a pre-req of Beam
   Development....
   
   On Wed, 16 Nov 2022 at 12:30, camphillips22 ***@***.***>
   wrote:
   
   > How hard would it be to just add the map_windows urn support? I wrote a
   > pipeline with a side input that isn't exactly the "slow changing side
   > input" pattern, but the pipeline failed because map_windows isn't
   > implemented.
   >
   > For context, the pipeline was shaped approximately like:
   >
   > flowchart TD
   >
   > A[WindowInto] --> |"KV#lt;string, data#gt;"| B["Distinct(DropValue(col))"];
   > B --> C[Lookup configuration for keys];
   > C --> D["Map to KV#lt;string, config#gt;"];
   > D --> E
   > A --> |"KV#lt;string, data#gt;"| E["Emit new data based on config<br>ParDo(col, beam.SideInput(KV#lt;string, config#gt;)"];
   >
   > —
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/beam/issues/23106#issuecomment-1317629945>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/ADKDOFKGXFX7TIWZ2QUD25DWIU76PANCNFSM6AAAAAAQIAHRAE>
   > .
   > You are receiving this because you authored the thread.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org