Calling rest api at the end of the pipeline

["Sam, David" <da...@optum.com>]

Hi ,
I have a use case where I would like to post the results to an API , I was thinking of using Sink , then its documented to not use that , what is the recommended way of doing this .

My flow is
Read from a flow--> do transformation / validation --> post the results to rest API and also post errors to rest API .

Thank you
David

This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.

Re: Calling rest api at the end of the pipeline

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi David,

you can use a simple ParDo(DoFn) at the end of your pipeline.

For instance, you can mimic what we do in the ElasticsearchIO Write:

https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java#L789

I also have an RestIO to call a service at any point of the pipeline. I can
share it with you (it's a WIP).

Regards
JB

On 03/08/2018 03:55 PM, Sam, David wrote:
> Hi ,
> 
> I have a use case where I would like to post the results to an API , I was
> thinking of using Sink , then its documented to not use that , what is the
> recommended way of doing this .
> 
>  
> 
> My flow is
> 
> Read from a flowà do transformation / validation à post the results to rest API
> and also post errors to rest API .
> 
>  
> 
> Thank you
> 
> David
> 
> 
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authorized agent, the reader is hereby notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify the
> sender by replying to this message and delete this e-mail immediately.
> 

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com