You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Ryan Hendrickson <ry...@gmail.com> on 2018/11/29 22:12:22 UTC
GetMongo - Pass-on Initial FlowFile?
Hi all,
I'm curious if the GetMongo processor can allow you can pass in a
FlowFile Document and enrich the original FlowFile document with the
results from the Query Result - or more generally store the result as a
NiFi Attribute, instead of replacing the content in the FlowFile.
I really want to "enrich" my results with data from Mongo, not replace
my FlowFile with the database record that matches.
I had thought I had figured out a way to do this in the past, but can't
seem to replicate it, so I figured I'd ask the community.
Thanks,
Ryan
Re: GetMongo - Pass-on Initial FlowFile?
Posted by Ryan Hendrickson <ry...@gmail.com>.
I was able to accomplish this by:
/---success---> *ReplaceText* (Prepend JSON Key)
---success--> \
/
\
*GetMongo*
-------> *Merge Content* (Combine on
Correlation Attribute Name, Binary Concat)
\
/
\---original---> *ReplaceText* (Prepend JSON Key)
---success--> /
Not the cleanest, but it gets it done.
Ryan
On Fri, Nov 30, 2018 at 3:43 PM Ryan Hendrickson <
ryan.andrew.hendrickson@gmail.com> wrote:
> I'm using the InferAvroSchema to generate the schema, such that I don't
> need to know about the JSON structure in advance.
>
> Another question.. Can the LookupRecord lookup based on the Mongo _id
> field?
>
> Using the GetMongo processor, I had the following as my query: { _id:
> ObjectId("${uuid}") }
>
> Is there an equivalent way to do that with LookupRecord and the
> MongoDBLookupService?
>
> Thanks,
> Ryan
>
> On Fri, Nov 30, 2018 at 11:19 AM Ryan Hendrickson <
> ryan.andrew.hendrickson@gmail.com> wrote:
>
>> My incoming FlowFIle is a valid JSON Object. The key names could be
>> anything, they're not defined.
>>
>> I need to add a top level object to every JSON Object, based on the
>> result from MongoDB.
>>
>> Is that possible with the JsonTreeReader/Schema, or do I need to know
>> what the fields are?
>>
>> Input:
>> { "key1" : "value", "key2" : value" }
>>
>> Output:
>> { "key1" : "value", "key2" : value", "mongorecord" : { "mongokey1" :
>> "mongovalue", "monogoKey2" : "mongovalue" } }
>>
>> Thanks,
>> Ryan
>>
>>
>> On Fri, Nov 30, 2018 at 10:54 AM Mike Thomsen <mi...@gmail.com>
>> wrote:
>>
>>> For the record side of things, you just need to create a schema that
>>> includes your existing JSON fields and a new branch that will have the
>>> enriched fields in it.
>>>
>>> On Fri, Nov 30, 2018 at 10:39 AM Ryan Hendrickson <
>>> ryan.andrew.hendrickson@gmail.com> wrote:
>>>
>>>> Hi Otto and Mike,
>>>> The LookupRecord does look fruitful, although I don't have a defined
>>>> schema for the JsonTreeReader. Is there a way to just keep it generic? I
>>>> know I have valid JSON already, I just want to add the result of the
>>>> MongoQuery to a specific Json Path in the FlowFile.
>>>>
>>>> The LookupAttribute also seems like it could be another avenue, but
>>>> it doesn't have the MongoDBLookupService in the list of Compatible
>>>> Controller Services.
>>>>
>>>> Ryan
>>>>
>>>> On Fri, Nov 30, 2018 at 10:24 AM Mike Thomsen <mi...@gmail.com>
>>>> wrote:
>>>>
>>>>> LookupAttribute + the MongoDBLookupService should be able to do that.
>>>>>
>>>>> On Thu, Nov 29, 2018 at 8:05 PM Otto Fowler <ot...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Sounds like you want to look at enrichment with the LookupRecord
>>>>>> processors and Mongo.
>>>>>>
>>>>>> https://community.hortonworks.com/articles/146198/data-flow-enrichment-with-nifi-part-3-lookuprecord.html
>>>>>>
>>>>>>
>>>>>> On November 29, 2018 at 17:12:38, Ryan Hendrickson (
>>>>>> ryan.andrew.hendrickson@gmail.com) wrote:
>>>>>>
>>>>>> Hi all,
>>>>>> I'm curious if the GetMongo processor can allow you can pass in a
>>>>>> FlowFile Document and enrich the original FlowFile document with the
>>>>>> results from the Query Result - or more generally store the result as a
>>>>>> NiFi Attribute, instead of replacing the content in the FlowFile.
>>>>>>
>>>>>> I really want to "enrich" my results with data from Mongo, not
>>>>>> replace my FlowFile with the database record that matches.
>>>>>>
>>>>>> I had thought I had figured out a way to do this in the past, but
>>>>>> can't seem to replicate it, so I figured I'd ask the community.
>>>>>>
>>>>>> Thanks,
>>>>>> Ryan
>>>>>>
>>>>>>
Re: GetMongo - Pass-on Initial FlowFile?
Posted by Ryan Hendrickson <ry...@gmail.com>.
I'm using the InferAvroSchema to generate the schema, such that I don't
need to know about the JSON structure in advance.
Another question.. Can the LookupRecord lookup based on the Mongo _id field?
Using the GetMongo processor, I had the following as my query: { _id:
ObjectId("${uuid}") }
Is there an equivalent way to do that with LookupRecord and the
MongoDBLookupService?
Thanks,
Ryan
On Fri, Nov 30, 2018 at 11:19 AM Ryan Hendrickson <
ryan.andrew.hendrickson@gmail.com> wrote:
> My incoming FlowFIle is a valid JSON Object. The key names could be
> anything, they're not defined.
>
> I need to add a top level object to every JSON Object, based on the result
> from MongoDB.
>
> Is that possible with the JsonTreeReader/Schema, or do I need to know what
> the fields are?
>
> Input:
> { "key1" : "value", "key2" : value" }
>
> Output:
> { "key1" : "value", "key2" : value", "mongorecord" : { "mongokey1" :
> "mongovalue", "monogoKey2" : "mongovalue" } }
>
> Thanks,
> Ryan
>
>
> On Fri, Nov 30, 2018 at 10:54 AM Mike Thomsen <mi...@gmail.com>
> wrote:
>
>> For the record side of things, you just need to create a schema that
>> includes your existing JSON fields and a new branch that will have the
>> enriched fields in it.
>>
>> On Fri, Nov 30, 2018 at 10:39 AM Ryan Hendrickson <
>> ryan.andrew.hendrickson@gmail.com> wrote:
>>
>>> Hi Otto and Mike,
>>> The LookupRecord does look fruitful, although I don't have a defined
>>> schema for the JsonTreeReader. Is there a way to just keep it generic? I
>>> know I have valid JSON already, I just want to add the result of the
>>> MongoQuery to a specific Json Path in the FlowFile.
>>>
>>> The LookupAttribute also seems like it could be another avenue, but
>>> it doesn't have the MongoDBLookupService in the list of Compatible
>>> Controller Services.
>>>
>>> Ryan
>>>
>>> On Fri, Nov 30, 2018 at 10:24 AM Mike Thomsen <mi...@gmail.com>
>>> wrote:
>>>
>>>> LookupAttribute + the MongoDBLookupService should be able to do that.
>>>>
>>>> On Thu, Nov 29, 2018 at 8:05 PM Otto Fowler <ot...@gmail.com>
>>>> wrote:
>>>>
>>>>> Sounds like you want to look at enrichment with the LookupRecord
>>>>> processors and Mongo.
>>>>>
>>>>> https://community.hortonworks.com/articles/146198/data-flow-enrichment-with-nifi-part-3-lookuprecord.html
>>>>>
>>>>>
>>>>> On November 29, 2018 at 17:12:38, Ryan Hendrickson (
>>>>> ryan.andrew.hendrickson@gmail.com) wrote:
>>>>>
>>>>> Hi all,
>>>>> I'm curious if the GetMongo processor can allow you can pass in a
>>>>> FlowFile Document and enrich the original FlowFile document with the
>>>>> results from the Query Result - or more generally store the result as a
>>>>> NiFi Attribute, instead of replacing the content in the FlowFile.
>>>>>
>>>>> I really want to "enrich" my results with data from Mongo, not
>>>>> replace my FlowFile with the database record that matches.
>>>>>
>>>>> I had thought I had figured out a way to do this in the past, but
>>>>> can't seem to replicate it, so I figured I'd ask the community.
>>>>>
>>>>> Thanks,
>>>>> Ryan
>>>>>
>>>>>
Re: GetMongo - Pass-on Initial FlowFile?
Posted by Ryan Hendrickson <ry...@gmail.com>.
My incoming FlowFIle is a valid JSON Object. The key names could be
anything, they're not defined.
I need to add a top level object to every JSON Object, based on the result
from MongoDB.
Is that possible with the JsonTreeReader/Schema, or do I need to know what
the fields are?
Input:
{ "key1" : "value", "key2" : value" }
Output:
{ "key1" : "value", "key2" : value", "mongorecord" : { "mongokey1" :
"mongovalue", "monogoKey2" : "mongovalue" } }
Thanks,
Ryan
On Fri, Nov 30, 2018 at 10:54 AM Mike Thomsen <mi...@gmail.com>
wrote:
> For the record side of things, you just need to create a schema that
> includes your existing JSON fields and a new branch that will have the
> enriched fields in it.
>
> On Fri, Nov 30, 2018 at 10:39 AM Ryan Hendrickson <
> ryan.andrew.hendrickson@gmail.com> wrote:
>
>> Hi Otto and Mike,
>> The LookupRecord does look fruitful, although I don't have a defined
>> schema for the JsonTreeReader. Is there a way to just keep it generic? I
>> know I have valid JSON already, I just want to add the result of the
>> MongoQuery to a specific Json Path in the FlowFile.
>>
>> The LookupAttribute also seems like it could be another avenue, but it
>> doesn't have the MongoDBLookupService in the list of Compatible Controller
>> Services.
>>
>> Ryan
>>
>> On Fri, Nov 30, 2018 at 10:24 AM Mike Thomsen <mi...@gmail.com>
>> wrote:
>>
>>> LookupAttribute + the MongoDBLookupService should be able to do that.
>>>
>>> On Thu, Nov 29, 2018 at 8:05 PM Otto Fowler <ot...@gmail.com>
>>> wrote:
>>>
>>>> Sounds like you want to look at enrichment with the LookupRecord
>>>> processors and Mongo.
>>>>
>>>> https://community.hortonworks.com/articles/146198/data-flow-enrichment-with-nifi-part-3-lookuprecord.html
>>>>
>>>>
>>>> On November 29, 2018 at 17:12:38, Ryan Hendrickson (
>>>> ryan.andrew.hendrickson@gmail.com) wrote:
>>>>
>>>> Hi all,
>>>> I'm curious if the GetMongo processor can allow you can pass in a
>>>> FlowFile Document and enrich the original FlowFile document with the
>>>> results from the Query Result - or more generally store the result as a
>>>> NiFi Attribute, instead of replacing the content in the FlowFile.
>>>>
>>>> I really want to "enrich" my results with data from Mongo, not
>>>> replace my FlowFile with the database record that matches.
>>>>
>>>> I had thought I had figured out a way to do this in the past, but
>>>> can't seem to replicate it, so I figured I'd ask the community.
>>>>
>>>> Thanks,
>>>> Ryan
>>>>
>>>>
Re: GetMongo - Pass-on Initial FlowFile?
Posted by Mike Thomsen <mi...@gmail.com>.
For the record side of things, you just need to create a schema that
includes your existing JSON fields and a new branch that will have the
enriched fields in it.
On Fri, Nov 30, 2018 at 10:39 AM Ryan Hendrickson <
ryan.andrew.hendrickson@gmail.com> wrote:
> Hi Otto and Mike,
> The LookupRecord does look fruitful, although I don't have a defined
> schema for the JsonTreeReader. Is there a way to just keep it generic? I
> know I have valid JSON already, I just want to add the result of the
> MongoQuery to a specific Json Path in the FlowFile.
>
> The LookupAttribute also seems like it could be another avenue, but it
> doesn't have the MongoDBLookupService in the list of Compatible Controller
> Services.
>
> Ryan
>
> On Fri, Nov 30, 2018 at 10:24 AM Mike Thomsen <mi...@gmail.com>
> wrote:
>
>> LookupAttribute + the MongoDBLookupService should be able to do that.
>>
>> On Thu, Nov 29, 2018 at 8:05 PM Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>>> Sounds like you want to look at enrichment with the LookupRecord
>>> processors and Mongo.
>>>
>>> https://community.hortonworks.com/articles/146198/data-flow-enrichment-with-nifi-part-3-lookuprecord.html
>>>
>>>
>>> On November 29, 2018 at 17:12:38, Ryan Hendrickson (
>>> ryan.andrew.hendrickson@gmail.com) wrote:
>>>
>>> Hi all,
>>> I'm curious if the GetMongo processor can allow you can pass in a
>>> FlowFile Document and enrich the original FlowFile document with the
>>> results from the Query Result - or more generally store the result as a
>>> NiFi Attribute, instead of replacing the content in the FlowFile.
>>>
>>> I really want to "enrich" my results with data from Mongo, not
>>> replace my FlowFile with the database record that matches.
>>>
>>> I had thought I had figured out a way to do this in the past, but
>>> can't seem to replicate it, so I figured I'd ask the community.
>>>
>>> Thanks,
>>> Ryan
>>>
>>>
Re: GetMongo - Pass-on Initial FlowFile?
Posted by Ryan Hendrickson <ry...@gmail.com>.
Hi Otto and Mike,
The LookupRecord does look fruitful, although I don't have a defined
schema for the JsonTreeReader. Is there a way to just keep it generic? I
know I have valid JSON already, I just want to add the result of the
MongoQuery to a specific Json Path in the FlowFile.
The LookupAttribute also seems like it could be another avenue, but it
doesn't have the MongoDBLookupService in the list of Compatible Controller
Services.
Ryan
On Fri, Nov 30, 2018 at 10:24 AM Mike Thomsen <mi...@gmail.com>
wrote:
> LookupAttribute + the MongoDBLookupService should be able to do that.
>
> On Thu, Nov 29, 2018 at 8:05 PM Otto Fowler <ot...@gmail.com>
> wrote:
>
>> Sounds like you want to look at enrichment with the LookupRecord
>> processors and Mongo.
>>
>> https://community.hortonworks.com/articles/146198/data-flow-enrichment-with-nifi-part-3-lookuprecord.html
>>
>>
>> On November 29, 2018 at 17:12:38, Ryan Hendrickson (
>> ryan.andrew.hendrickson@gmail.com) wrote:
>>
>> Hi all,
>> I'm curious if the GetMongo processor can allow you can pass in a
>> FlowFile Document and enrich the original FlowFile document with the
>> results from the Query Result - or more generally store the result as a
>> NiFi Attribute, instead of replacing the content in the FlowFile.
>>
>> I really want to "enrich" my results with data from Mongo, not replace
>> my FlowFile with the database record that matches.
>>
>> I had thought I had figured out a way to do this in the past, but
>> can't seem to replicate it, so I figured I'd ask the community.
>>
>> Thanks,
>> Ryan
>>
>>
Re: GetMongo - Pass-on Initial FlowFile?
Posted by Mike Thomsen <mi...@gmail.com>.
LookupAttribute + the MongoDBLookupService should be able to do that.
On Thu, Nov 29, 2018 at 8:05 PM Otto Fowler <ot...@gmail.com> wrote:
> Sounds like you want to look at enrichment with the LookupRecord
> processors and Mongo.
>
> https://community.hortonworks.com/articles/146198/data-flow-enrichment-with-nifi-part-3-lookuprecord.html
>
>
> On November 29, 2018 at 17:12:38, Ryan Hendrickson (
> ryan.andrew.hendrickson@gmail.com) wrote:
>
> Hi all,
> I'm curious if the GetMongo processor can allow you can pass in a
> FlowFile Document and enrich the original FlowFile document with the
> results from the Query Result - or more generally store the result as a
> NiFi Attribute, instead of replacing the content in the FlowFile.
>
> I really want to "enrich" my results with data from Mongo, not replace
> my FlowFile with the database record that matches.
>
> I had thought I had figured out a way to do this in the past, but can't
> seem to replicate it, so I figured I'd ask the community.
>
> Thanks,
> Ryan
>
>
Re: GetMongo - Pass-on Initial FlowFile?
Posted by Otto Fowler <ot...@gmail.com>.
Sounds like you want to look at enrichment with the LookupRecord processors
and Mongo.
https://community.hortonworks.com/articles/146198/data-flow-enrichment-with-nifi-part-3-lookuprecord.html
On November 29, 2018 at 17:12:38, Ryan Hendrickson (
ryan.andrew.hendrickson@gmail.com) wrote:
Hi all,
I'm curious if the GetMongo processor can allow you can pass in a
FlowFile Document and enrich the original FlowFile document with the
results from the Query Result - or more generally store the result as a
NiFi Attribute, instead of replacing the content in the FlowFile.
I really want to "enrich" my results with data from Mongo, not replace
my FlowFile with the database record that matches.
I had thought I had figured out a way to do this in the past, but can't
seem to replicate it, so I figured I'd ask the community.
Thanks,
Ryan