Making RPCs in Beam

[Praveen K Viswanathan <ha...@gmail.com>]

Hello Everyone,

In my pipeline I have to make a *single RPC call* as well as a *Batched RPC
call* to fetch data for enrichment. I could not find any reference on how
to make these call within your pipeline. I am still covering my grounds in
Apache Beam and would appreciate if anyone has done this and could share a
sample code or details on how to do this.
-- 
Thanks,
Praveen K Viswanathan

Re: Making RPCs in Beam

[Praveen K Viswanathan <ha...@gmail.com>]

Thanks for the intro to "GroupIntoBatches" Luke. It looks very interesting
and easy for someone to understand when they read the pipeline. I will give
it a try for my batched RPC use case.

Regards,
Praveen Kumar Viswanathan

On Fri, Jun 19, 2020 at 9:24 AM Luke Cwik <lc...@google.com> wrote:

> I would suggest looking at the GroupIntoBatches[1] transform if you want
> to support batching of elements instead of writing it yourself. Single RPC
> calls should be invoked from the DoFn like you would from a regular program.
>
> 1:
> https://beam.apache.org/documentation/transforms/java/aggregation/groupintobatches/
>
> On Fri, Jun 19, 2020 at 8:20 AM Brian Hulette <bh...@google.com> wrote:
>
>> Kenn wrote a blog post showing how to do batched RPCs with the state and
>> timer APIs: https://beam.apache.org/blog/timely-processing/
>>
>> Is that helpful?
>>
>> Brian
>>
>> On Thu, Jun 18, 2020 at 5:29 PM Praveen K Viswanathan <
>> harish.praveen@gmail.com> wrote:
>>
>>> Hello Everyone,
>>>
>>> In my pipeline I have to make a *single RPC call* as well as a *Batched
>>> RPC call* to fetch data for enrichment. I could not find any reference
>>> on how to make these call within your pipeline. I am still covering my
>>> grounds in Apache Beam and would appreciate if anyone has done this and
>>> could share a sample code or details on how to do this.
>>> --
>>> Thanks,
>>> Praveen K Viswanathan
>>>
>>

-- 
Thanks,
Praveen K Viswanathan

Re: Making RPCs in Beam

[Luke Cwik <lc...@google.com>]

I would suggest looking at the GroupIntoBatches[1] transform if you want to
support batching of elements instead of writing it yourself. Single RPC
calls should be invoked from the DoFn like you would from a regular program.

1:
https://beam.apache.org/documentation/transforms/java/aggregation/groupintobatches/

On Fri, Jun 19, 2020 at 8:20 AM Brian Hulette <bh...@google.com> wrote:

> Kenn wrote a blog post showing how to do batched RPCs with the state and
> timer APIs: https://beam.apache.org/blog/timely-processing/
>
> Is that helpful?
>
> Brian
>
> On Thu, Jun 18, 2020 at 5:29 PM Praveen K Viswanathan <
> harish.praveen@gmail.com> wrote:
>
>> Hello Everyone,
>>
>> In my pipeline I have to make a *single RPC call* as well as a *Batched
>> RPC call* to fetch data for enrichment. I could not find any reference
>> on how to make these call within your pipeline. I am still covering my
>> grounds in Apache Beam and would appreciate if anyone has done this and
>> could share a sample code or details on how to do this.
>> --
>> Thanks,
>> Praveen K Viswanathan
>>
>

Re: Making RPCs in Beam

[Praveen K Viswanathan <ha...@gmail.com>]

Hi Brian - Thanks for responding to my post. Yes, Kenn's blog helped in
getting an high-level idea but I am more interested in the mechanics of how
to fit in a HTTP call in Beam. To be specific, the *enrichEvents* method in
his blog which making real call to HTTP URLs. Unfortunately, that
information was abstracted in the context of Batched RPC.

Thanks,
Praveen Kumar Viswanathan

On Fri, Jun 19, 2020 at 8:20 AM Brian Hulette <bh...@google.com> wrote:

> Kenn wrote a blog post showing how to do batched RPCs with the state and
> timer APIs: https://beam.apache.org/blog/timely-processing/
>
> Is that helpful?
>
> Brian
>
> On Thu, Jun 18, 2020 at 5:29 PM Praveen K Viswanathan <
> harish.praveen@gmail.com> wrote:
>
>> Hello Everyone,
>>
>> In my pipeline I have to make a *single RPC call* as well as a *Batched
>> RPC call* to fetch data for enrichment. I could not find any reference
>> on how to make these call within your pipeline. I am still covering my
>> grounds in Apache Beam and would appreciate if anyone has done this and
>> could share a sample code or details on how to do this.
>> --
>> Thanks,
>> Praveen K Viswanathan
>>
>

-- 
Thanks,
Praveen K Viswanathan

Re: Making RPCs in Beam

[Brian Hulette <bh...@google.com>]

Kenn wrote a blog post showing how to do batched RPCs with the state and
timer APIs: https://beam.apache.org/blog/timely-processing/

Is that helpful?

Brian

On Thu, Jun 18, 2020 at 5:29 PM Praveen K Viswanathan <
harish.praveen@gmail.com> wrote:

> Hello Everyone,
>
> In my pipeline I have to make a *single RPC call* as well as a *Batched
> RPC call* to fetch data for enrichment. I could not find any reference on
> how to make these call within your pipeline. I am still covering my grounds
> in Apache Beam and would appreciate if anyone has done this and could share
> a sample code or details on how to do this.
> --
> Thanks,
> Praveen K Viswanathan
>