Enrichment of record data with a REST API

[Mike Thomsen <mi...@gmail.com>]

Does anyone know a good pattern using the Record API to enrich a data set
record by record with a REST API?

Thanks,

Mike

Re: Enrichment of record data with a REST API

[Mark Payne <ma...@hotmail.com>]

Mike,

To really do a good job with Enrichment from an http endpoint, we need two transformations really. We need the ability to transform the input Record into what the web service wants/needs. And we also need the ability to take the response from that web service and join that response together with the input Record.

Ideally there would be script-based ways to do each of these, as well as other ways, such as using SQL to query for certain fields from the input record and certain fields from the lookup service response. I have some ideas floating around in my head but I’ve not really laid anything out very clearly. There are really two different approaches that I think could work here.

The first approach would be to have a “request transformation” property that would allow for taking an input record and transforming it into a request for the web service. There would then be another “response transformation” property that would merge the input and the response into a single Record. This is, in a way, simple, because it it means that the LookupRecord processor is all that is needed. But it would need a Controller Service for the Record Reader, one for the Record Writer, and a third for the Lookup Service. It would likely also require a Controller Service for the “request transformation” and another for the “response transformation.” So it ends up being a lot more complex to configure than it first seems. But perhaps this is okay...

The second approach would be to create a couple of different processors that would allow for the implementation of the Claim Check Pattern [1]. Here, we would have a “CheckFlowFile” processor, which could be used in conjunction with any existing processor for transforming the incoming Record into the needed request. The response would then be obtained, and then would need to be merged back together with some sort of “Merge Records” processor. This approach is probably cleaner in terms of flow design and easier to understand. It also results in better code re-use. Unfortunately, though, it does come with the complication of figuring out how to properly correlate the appropriate input Record with the appropriate “enrichment response” Record….

Just some thoughts :)

[1] https://www.enterpriseintegrationpatterns.com/patterns/messaging/StoreInLibrary.html


On Jun 29, 2020, at 8:22 PM, Mike Thomsen <mi...@gmail.com>> wrote:

Matt,

Yeah, I was thinking about that, but there are a lot of variables that have come up since I wrote that service. One of the big ones is how to take partial responses and merge them? There's no transformer API for that service. Nothing like a Groovy script, JOLT, etc.

What do you think? I think something JOLT-based similar to JoltTransformRecord could be a starting point.

On Mon, Jun 29, 2020 at 5:14 PM Matt Burgess <ma...@apache.org>> wrote:
Mike,

I think you can use LookupRecord with a RestLookupService to do this.
If it's missing features or it otherwise doesn't work for your use
case, please let us know and/or write up whatever Jiras you feel are
appropriate.

Regards,
Matt

On Mon, Jun 29, 2020 at 4:56 PM Mike Thomsen <mi...@gmail.com>> wrote:
>
> Does anyone know a good pattern using the Record API to enrich a data set record by record with a REST API?
>
> Thanks,
>
> Mike

Re: Enrichment of record data with a REST API

[Mike Thomsen <mi...@gmail.com>]

Matt,

Yeah, I was thinking about that, but there are a lot of variables that have
come up since I wrote that service. One of the big ones is how to take
partial responses and merge them? There's no transformer API for that
service. Nothing like a Groovy script, JOLT, etc.

What do you think? I think something JOLT-based similar to
JoltTransformRecord could be a starting point.

On Mon, Jun 29, 2020 at 5:14 PM Matt Burgess <ma...@apache.org> wrote:

> Mike,
>
> I think you can use LookupRecord with a RestLookupService to do this.
> If it's missing features or it otherwise doesn't work for your use
> case, please let us know and/or write up whatever Jiras you feel are
> appropriate.
>
> Regards,
> Matt
>
> On Mon, Jun 29, 2020 at 4:56 PM Mike Thomsen <mi...@gmail.com>
> wrote:
> >
> > Does anyone know a good pattern using the Record API to enrich a data
> set record by record with a REST API?
> >
> > Thanks,
> >
> > Mike
>

Re: Enrichment of record data with a REST API

[Matt Burgess <ma...@apache.org>]

Mike,

I think you can use LookupRecord with a RestLookupService to do this.
If it's missing features or it otherwise doesn't work for your use
case, please let us know and/or write up whatever Jiras you feel are
appropriate.

Regards,
Matt

On Mon, Jun 29, 2020 at 4:56 PM Mike Thomsen <mi...@gmail.com> wrote:
>
> Does anyone know a good pattern using the Record API to enrich a data set record by record with a REST API?
>
> Thanks,
>
> Mike