You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "matthewmichala@northwesternmutual.com" <ma...@northwesternmutual.com> on 2019/03/11 22:51:21 UTC
Help with NiFi LookupRecord & MergeRecord Processors
NiFi Developers,
I'm a senior engineer at Northwestern Mutual in Milwaukee, and was just wondering if anyone can help us with the NiFi LookupRecord, or MergeRecord Processors?
We keep getting this error when trying to use any Lookup Service (NiFi or 3rd party) to append additional attributes to our initial DB2 Query:
Failed to lookup coordinates {LEGAL_ENTITY_ID_NUM=13366} in Lookup Service
Has anyone had any success merging data from 2 different sql queries based on a common Unique ID from both queries? We've tried LookupRecord which uses the Lookup Service, and also tried MergeRecord, and MergeContent - nothing seems to work??
Bottom line is - we have an initial main query (ID & Status) - then want to append an additional attribute (Name) from a 2nd query (which just contains ID & Name) - onto each record of the main query - after joining on ID.
Whenever we try using LookupRecord, and pass the ID as a parameter via ? marks, we keep getting the above error. We've gotten nowhere with MergeRecord too. We have learned how to MergeContent, but that is not really what we want (is just a workaround).
Thanks & Regards,
Matt Michala
Senior Engineer
Digital Workplace & Corp. Solutions
Northwestern Mutual (T07-150)
414-661-4668
Thanks & Regards,
Matt Michala
Senior Engineer
Digital Workplace & Corp. Solutions
Northwestern Mutual (T07-150)
414-661-4668
This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.
Re: Help with NiFi LookupRecord & MergeRecord Processors
Posted by Mike Thomsen <mi...@gmail.com>.
If you're not a Java developer and/or don't have access to any on your
team, your best work around would be to look at massaging your lookup data
into HBase, Mongo or ElasticSearch and using those. If you are a Java
developer or have one on your team I can point you to some sample projects
you can hack up to quickly spin up a custom component that is tailored to
your needs.
On Mon, Mar 11, 2019 at 6:57 PM matthewmichala@northwesternmutual.com <
matthewmichala@northwesternmutual.com> wrote:
> FYI, I did get the following responses from the Stack Overflow Nifi Tag
> Site - so until the DB Lookup Service is available, does anyone have any
> other workaround or custom built solutions:
>
> Stack Overflow Reply 1:
>
> When you use LookupRecord, what LookupService are you using? There
> currently is not a released implementation of a LookupService that works
> against a relatonial DB, although there is active work to implement one
>
> Stack Overflow Reply 2:
>
> MergeRecord understands records and can merge them together as proper
> JSON. In both cases though, merging is not joining, merging is writing
> content or records one after another to create larger flow files.
>
>
>
> -----Original Message-----
> From: matthewmichala@northwesternmutual.com <
> matthewmichala@northwesternmutual.com>
> Sent: Monday, March 11, 2019 5:51 PM
> To: dev@nifi.apache.org
> Subject: [EXTERNAL] Help with NiFi LookupRecord & MergeRecord Processors
>
> NiFi Developers,
>
> I'm a senior engineer at Northwestern Mutual in Milwaukee, and was just
> wondering if anyone can help us with the NiFi LookupRecord, or MergeRecord
> Processors?
>
> We keep getting this error when trying to use any Lookup Service (NiFi or
> 3rd party) to append additional attributes to our initial DB2 Query:
>
> Failed to lookup coordinates {LEGAL_ENTITY_ID_NUM=13366} in Lookup
> Service
>
> Has anyone had any success merging data from 2 different sql queries based
> on a common Unique ID from both queries? We've tried LookupRecord which
> uses the Lookup Service, and also tried MergeRecord, and MergeContent -
> nothing seems to work??
>
> Bottom line is - we have an initial main query (ID & Status) - then want
> to append an additional attribute (Name) from a 2nd query (which just
> contains ID & Name) - onto each record of the main query - after joining on
> ID.
>
> Whenever we try using LookupRecord, and pass the ID as a parameter via ?
> marks, we keep getting the above error. We've gotten nowhere with
> MergeRecord too. We have learned how to MergeContent, but that is not
> really what we want (is just a workaround).
> Thanks & Regards,
>
> Matt Michala
> Senior Engineer
> Digital Workplace & Corp. Solutions
> Northwestern Mutual (T07-150)
> 414-661-4668
>
>
> Thanks & Regards,
>
> Matt Michala
> Senior Engineer
> Digital Workplace & Corp. Solutions
> Northwestern Mutual (T07-150)
> 414-661-4668
>
>
>
> This e-mail and any attachments may contain confidential information of
> Northwestern Mutual. If you are not the intended recipient of this message,
> be aware that any disclosure, copying, distribution or use of this e-mail
> and any attachments is prohibited. If you have received this e-mail in
> error, please notify Northwestern Mutual immediately by returning it to the
> sender and delete all copies from your system. Please be advised that
> communications with {SECURE MESSAGE} in the subject line have been sent
> using a secure messaging system. Communications that do not have this tag
> may not be secure and could be observed by a third party. Thank you for
> your cooperation.
>
>
>
>
> This e-mail and any attachments may contain confidential information of
> Northwestern Mutual. If you are not the intended recipient of this message,
> be aware that any disclosure, copying, distribution or use of this e-mail
> and any attachments is prohibited. If you have received this e-mail in
> error, please notify Northwestern Mutual immediately by returning it to the
> sender and delete all copies from your system. Please be advised that
> communications with {SECURE MESSAGE} in the subject line have been sent
> using a secure messaging system. Communications that do not have this tag
> may not be secure and could be observed by a third party. Thank you for
> your cooperation.
>
>
>
RE: Help with NiFi LookupRecord & MergeRecord Processors
Posted by "matthewmichala@northwesternmutual.com" <ma...@northwesternmutual.com>.
FYI, I did get the following responses from the Stack Overflow Nifi Tag Site - so until the DB Lookup Service is available, does anyone have any other workaround or custom built solutions:
Stack Overflow Reply 1:
When you use LookupRecord, what LookupService are you using? There currently is not a released implementation of a LookupService that works against a relatonial DB, although there is active work to implement one
Stack Overflow Reply 2:
MergeRecord understands records and can merge them together as proper JSON. In both cases though, merging is not joining, merging is writing content or records one after another to create larger flow files.
-----Original Message-----
From: matthewmichala@northwesternmutual.com <ma...@northwesternmutual.com>
Sent: Monday, March 11, 2019 5:51 PM
To: dev@nifi.apache.org
Subject: [EXTERNAL] Help with NiFi LookupRecord & MergeRecord Processors
NiFi Developers,
I'm a senior engineer at Northwestern Mutual in Milwaukee, and was just wondering if anyone can help us with the NiFi LookupRecord, or MergeRecord Processors?
We keep getting this error when trying to use any Lookup Service (NiFi or 3rd party) to append additional attributes to our initial DB2 Query:
Failed to lookup coordinates {LEGAL_ENTITY_ID_NUM=13366} in Lookup Service
Has anyone had any success merging data from 2 different sql queries based on a common Unique ID from both queries? We've tried LookupRecord which uses the Lookup Service, and also tried MergeRecord, and MergeContent - nothing seems to work??
Bottom line is - we have an initial main query (ID & Status) - then want to append an additional attribute (Name) from a 2nd query (which just contains ID & Name) - onto each record of the main query - after joining on ID.
Whenever we try using LookupRecord, and pass the ID as a parameter via ? marks, we keep getting the above error. We've gotten nowhere with MergeRecord too. We have learned how to MergeContent, but that is not really what we want (is just a workaround).
Thanks & Regards,
Matt Michala
Senior Engineer
Digital Workplace & Corp. Solutions
Northwestern Mutual (T07-150)
414-661-4668
Thanks & Regards,
Matt Michala
Senior Engineer
Digital Workplace & Corp. Solutions
Northwestern Mutual (T07-150)
414-661-4668
This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.
This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.