Apply Wait.on() pattern after AvroIO, KinesisIO writes

["Sunny, Mani Kolbe" <Su...@DNB.com>]

Hello,

I am looking to implement Wait.on() pattern to do something after writes for each window are done. There are two outputs - one writing using AvroIO and other using KinesisIO.   The problem is writes don't return PCollection which is required to construct Wait.On(). Is there a way around?

Regards,
Mani

Re: Apply Wait.on() pattern after AvroIO, KinesisIO writes

[Eugene Kirpichov <ki...@google.com>]

If a transform doesn't return something waitable, there is no way to wait
on it.
However:
* AvroIO.write is waitable - if not through AvroIO.write() (I don't
remember off the top of my head), then at least through
FileIO.write().via(AvroIO.sink()).
* KinesisIO.write is *very* easy to change to be waitable - as you can see
here
<https://github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java#L705>,
it simply applies a ParDo. Merely returning the result of that, instead of
returning PDone.in(), would make it waitable.


On Sat, Jun 27, 2020 at 3:39 AM Mani Kolbe <ma...@gmail.com> wrote:

> I am using sparkRunner on streaming mode. So stateful DoFn is not
> supported.
>
> On Sat, 27 Jun, 2020, 7:48 AM Sunny, Mani Kolbe, <Su...@dnb.com> wrote:
>
>> Hello,
>>
>>
>>
>> I am looking to implement Wait.on() pattern to do something after writes
>> for each window are done. There are two outputs - one writing using AvroIO
>> and other using KinesisIO.   The problem is writes don’t return PCollection
>> which is required to construct Wait.On(). Is there a way around?
>>
>>
>>
>> Regards,
>>
>> Mani
>>
>

Re: Apply Wait.on() pattern after AvroIO, KinesisIO writes

[Mani Kolbe <ma...@gmail.com>]

I am using sparkRunner on streaming mode. So stateful DoFn is not
supported.

On Sat, 27 Jun, 2020, 7:48 AM Sunny, Mani Kolbe, <Su...@dnb.com> wrote:

> Hello,
>
>
>
> I am looking to implement Wait.on() pattern to do something after writes
> for each window are done. There are two outputs - one writing using AvroIO
> and other using KinesisIO.   The problem is writes don’t return PCollection
> which is required to construct Wait.On(). Is there a way around?
>
>
>
> Regards,
>
> Mani
>