GO SDK - help on trigger

[Lakstsap <la...@gmail.com>]

Hi
I am reading large files and emit to a windowing function ( window duration for a minute ) Then using grouping ParDo transform I am grouping by key and collection.

Because of the file size is large it is stuck at the group by key pipeline step. As per the documentation looks like I need to set a trigger to emit the results from the group by key step because of windowing.
Looks like go sdk does not support trigger. 
Could you please tell me alternate way for this or when do you plan to release a code for trigger concept like in java 

Thanks
laks 

Re: GO SDK - help on trigger

[Robert Burke <ro...@frantil.com>]

Correct, triggers are not yet implemented in the Go SDK. This is one of the
laundry list of things keeping the SDK experimental.

No one is working on them either to my knowledge. My understanding of
triggers is limited, but there seems to be a preference for implementing
timers+state which cover many overlapping cases more cleanly. However no
one is working on them either. I would welcome designs & PRs to discuss.

SplittableDoFns would help with this too, and are being worked on but
aren't quite there yet. We should have the first working prototype branch
for folks to try it out in the next few months.

Otherwise, there's no way to get early/parallel processing of elements
within a file, outside of waiting for the full read/emit/reshard-shuffle
cycle ends where most runners can then parallelize processing.

Cheers,
Robert Burke, resident Go SDK expert.

On Sat, Nov 16, 2019, 11:32 AM Lakstsap <la...@gmail.com> wrote:

> Hi
> I am reading large files and emit to a windowing function ( window
> duration for a minute ) Then using grouping ParDo transform I am grouping
> by key and collection.
>
> Because of the file size is large it is stuck at the group by key pipeline
> step. As per the documentation looks like I need to set a trigger to emit
> the results from the group by key step because of windowing.
> Looks like go sdk does not support trigger.
> Could you please tell me alternate way for this or when do you plan to
> release a code for trigger concept like in java
>
> Thanks
> laks