You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by jitendra sharma <ji...@gmail.com> on 2019/10/22 08:42:37 UTC
How side input will work for streaming application in apache beam.
Hi All,
We have written a streaming application that will consume the Kafka event
and will call the rest API for some validation information and passed as
side input for validation for each event.
I am just wondering how many times side input will be called for streaming
applications as side input value is dynamic in nature(every 10 seconds)? I
don't want to make calls to Rest API for each event.
I am looking for at least every 10-second side input that would be
refreshed.
Any ideas/suggestions are appreciated.
Regards,
Jitendra Sharma
Re: How side input will work for streaming application in apache beam.
Posted by jitendra sharma <ji...@gmail.com>.
Thank you. Link is helpful.
On Wed, Oct 23, 2019 at 2:56 PM Reza Rokni <re...@google.com> wrote:
> Do you have a event that would initiate the creation of the Side Input? If
> not you may want to look at this pattern as a way to trigger the API call
> in process time :
>
> https://beam.apache.org/documentation/patterns/overview/
>
> On Wed, 23 Oct 2019, 00:27 Maximilian Michels, <mx...@apache.org> wrote:
>
>> Hi Jitendra,
>>
>> Side inputs are materialized based on their windowing. If you assign a
>> 10 second window to the side inputs, they can be renewed every 10
>> seconds. Whenever you access the side input, the newest instance of the
>> side input will be retrieved.
>>
>> Cheers,
>> Max
>>
>> On 22.10.19 10:42, jitendra sharma wrote:
>> > Hi All,
>> >
>> > We have written a streaming application that will consume the Kafka
>> > event and will call the rest API for some validation information and
>> > passed as side input for validation for each event.
>> >
>> > I am just wondering how many times side input will be called for
>> > streaming applications as side input value is dynamic in nature(every
>> 10
>> > seconds)? I don't want to make calls to Rest API for each event.
>> >
>> > I am looking for at least every 10-second side input that would be
>> > refreshed.
>> >
>> > Any ideas/suggestions are appreciated.
>> >
>> > Regards,
>> > Jitendra Sharma
>>
>
--
Jitendra Sharma
Re: How side input will work for streaming application in apache beam.
Posted by Reza Rokni <re...@google.com>.
Do you have a event that would initiate the creation of the Side Input? If
not you may want to look at this pattern as a way to trigger the API call
in process time :
https://beam.apache.org/documentation/patterns/overview/
On Wed, 23 Oct 2019, 00:27 Maximilian Michels, <mx...@apache.org> wrote:
> Hi Jitendra,
>
> Side inputs are materialized based on their windowing. If you assign a
> 10 second window to the side inputs, they can be renewed every 10
> seconds. Whenever you access the side input, the newest instance of the
> side input will be retrieved.
>
> Cheers,
> Max
>
> On 22.10.19 10:42, jitendra sharma wrote:
> > Hi All,
> >
> > We have written a streaming application that will consume the Kafka
> > event and will call the rest API for some validation information and
> > passed as side input for validation for each event.
> >
> > I am just wondering how many times side input will be called for
> > streaming applications as side input value is dynamic in nature(every 10
> > seconds)? I don't want to make calls to Rest API for each event.
> >
> > I am looking for at least every 10-second side input that would be
> > refreshed.
> >
> > Any ideas/suggestions are appreciated.
> >
> > Regards,
> > Jitendra Sharma
>
Re: How side input will work for streaming application in apache
beam.
Posted by Maximilian Michels <mx...@apache.org>.
Hi Jitendra,
Side inputs are materialized based on their windowing. If you assign a
10 second window to the side inputs, they can be renewed every 10
seconds. Whenever you access the side input, the newest instance of the
side input will be retrieved.
Cheers,
Max
On 22.10.19 10:42, jitendra sharma wrote:
> Hi All,
>
> We have written a streaming application that will consume the Kafka
> event and will call the rest API for some validation information and
> passed as side input for validation for each event.
>
> I am just wondering how many times side input will be called for
> streaming applications as side input value is dynamic in nature(every 10
> seconds)? I don't want to make calls to Rest API for each event.
>
> I am looking for at least every 10-second side input that would be
> refreshed.
>
> Any ideas/suggestions are appreciated.
>
> Regards,
> Jitendra Sharma