You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by James Wing <jv...@gmail.com> on 2016/07/01 06:41:59 UTC
Re: GetDynamoDB Processor returns only item attribute from DynamoDB
but not all attributes.
Mike,
I believe it was an intentional design choice to separate the hash/range
keys from the flowfile content for validation and handling. There is
currently no option in PutDynamoDB or GetDynamoDB to match the entire
flowfile content to the entire DynamoDB item, but I believe that is
certainly a possible extension.
Thanks,
James
On Thu, Jun 30, 2016 at 9:57 AM, Mike Connolly <mi...@oracle.com>
wrote:
> I checked out the master branch of Nifi and noticed the GetDynamoDB
> processor which was something I really needed for a workflow. I tried using
> it to return the json representation of a DynamoDB item. It looks like it
> will only return a json representation of a a single attribute of a
> DynamoDB item. I'm wondering it this was the intention or if there is a
> specific value you need to set in the processor configuration that will
> return the whole item. Anybody ever use this?
>
> Thanks,
>
> Mike Connolly
>
Re: GetDynamoDB Processor returns only item attribute from DynamoDB
but not all attributes.
Posted by Mike Connolly <mi...@oracle.com>.
Mark,
I this point on my project we are evaluating different workflow
technologies. I was asked to take a look at NiFi which looks pretty
cool. At some point the powers that be will make a determination as to
what software we will be using. If we go with NiFi I would definitely
work on solving this problem. I can't commit at this time since its not
in my tasking. I'll keep you posted.
Thanks,
Mike
On 07/05/2016 09:15 AM, Mark Payne wrote:
> Hi Mike,
>
> Personally, I'm not familiar with DynamoDB either, but a quick look at
> the source & configuration do indeed
> show that it expects the user to fetch only a single 'attribute' from
> a Dynamo Item. Looking at the AWS javadocs,
> it appears that this is the typical usage, as they don't seem to
> provide a method for obtaining the entire item
> as JSON (unless I'm missing something).
>
> One approach, though, would be to allow the 'Json Document attribute'
> property to be optional instead of required
> and if not specified, simply iterate over all of the keys in the
> Dynamo DB Item and create a JSON object ourselves
> that we write to the content of the FlowFile.
>
> Often what happens, though, is that a member of the NiFi community
> develops a processor to provide the functionality
> that they need and sometimes makes the processor a little more
> 'generic' to handle additional use cases, if it makes sense.
> As the community begins to use the processor, it begins to evolved
> over time to meet more and more use cases. This is
> a fairly new processor and so likely will evolve over time to meet
> more & more use cases. I think the use case of pulling back
> an entire DynamoDB item is very reasonable and likely will be needed
> by others.
>
> Is there any chance that you're interested in working on building out
> that capability and contributing it to the community?
> We would be more than happy to help shepherd you through the process
> of contributing it back to make NiFi's dynamo
> support better!
>
> Thanks
> -Mark
>
>
>> On Jul 5, 2016, at 8:51 AM, Mike Connolly <mike.connolly@oracle.com
>> <ma...@oracle.com>> wrote:
>>
>> James,
>> Hmmm. I'm not too familiar with DynamoDB or NiFi, but having a
>> processor that can only return one attribute at a time doesn't seem
>> very useful. I would think the processor would retrieve all the
>> attributes and package them into one variable and send it back as one
>> unit so when the user configures the processor they could do
>> something like set the json attribute value to "ALL" or something
>> like that. I tried modifying the processor source code to call
>> item.getJSON() to populate the return value with everything, but
>> something isn't liking { brace in the string value that gets sent
>> back. Its complaining about unacceptable characters in the string. I
>> might need to swich to MongoDB as my datasource and see if that works.
>>
>> Thanks,
>>
>> Mike
>>
>> On 07/01/2016 02:41 AM, James Wing wrote:
>>> Mike,
>>>
>>> I believe it was an intentional design choice to separate the
>>> hash/range keys from the flowfile content for validation and
>>> handling. There is currently no option in PutDynamoDB or
>>> GetDynamoDB to match the entire flowfile content to the entire
>>> DynamoDB item, but I believe that is certainly a possible extension.
>>>
>>> Thanks,
>>>
>>> James
>>>
>>> On Thu, Jun 30, 2016 at 9:57 AM, Mike Connolly
>>> <mike.connolly@oracle.com <ma...@oracle.com>> wrote:
>>>
>>> I checked out the master branch of Nifi and noticed the
>>> GetDynamoDB processor which was something I really needed for a
>>> workflow. I tried using it to return the json representation of
>>> a DynamoDB item. It looks like it will only return a json
>>> representation of a a single attribute of a DynamoDB item. I'm
>>> wondering it this was the intention or if there is a specific
>>> value you need to set in the processor configuration that will
>>> return the whole item. Anybody ever use this?
>>>
>>> Thanks,
>>>
>>> Mike Connolly
>>>
>>>
>>
>
Re: GetDynamoDB Processor returns only item attribute from DynamoDB but not all attributes.
Posted by Mark Payne <ma...@hotmail.com>.
Hi Mike,
Personally, I'm not familiar with DynamoDB either, but a quick look at the source & configuration do indeed
show that it expects the user to fetch only a single 'attribute' from a Dynamo Item. Looking at the AWS javadocs,
it appears that this is the typical usage, as they don't seem to provide a method for obtaining the entire item
as JSON (unless I'm missing something).
One approach, though, would be to allow the 'Json Document attribute' property to be optional instead of required
and if not specified, simply iterate over all of the keys in the Dynamo DB Item and create a JSON object ourselves
that we write to the content of the FlowFile.
Often what happens, though, is that a member of the NiFi community develops a processor to provide the functionality
that they need and sometimes makes the processor a little more 'generic' to handle additional use cases, if it makes sense.
As the community begins to use the processor, it begins to evolved over time to meet more and more use cases. This is
a fairly new processor and so likely will evolve over time to meet more & more use cases. I think the use case of pulling back
an entire DynamoDB item is very reasonable and likely will be needed by others.
Is there any chance that you're interested in working on building out that capability and contributing it to the community?
We would be more than happy to help shepherd you through the process of contributing it back to make NiFi's dynamo
support better!
Thanks
-Mark
> On Jul 5, 2016, at 8:51 AM, Mike Connolly <mi...@oracle.com> wrote:
>
> James,
> Hmmm. I'm not too familiar with DynamoDB or NiFi, but having a processor that can only return one attribute at a time doesn't seem very useful. I would think the processor would retrieve all the attributes and package them into one variable and send it back as one unit so when the user configures the processor they could do something like set the json attribute value to "ALL" or something like that. I tried modifying the processor source code to call item.getJSON() to populate the return value with everything, but something isn't liking { brace in the string value that gets sent back. Its complaining about unacceptable characters in the string. I might need to swich to MongoDB as my datasource and see if that works.
>
> Thanks,
>
> Mike
>
> On 07/01/2016 02:41 AM, James Wing wrote:
>> Mike,
>>
>> I believe it was an intentional design choice to separate the hash/range keys from the flowfile content for validation and handling. There is currently no option in PutDynamoDB or GetDynamoDB to match the entire flowfile content to the entire DynamoDB item, but I believe that is certainly a possible extension.
>>
>> Thanks,
>>
>> James
>>
>> On Thu, Jun 30, 2016 at 9:57 AM, Mike Connolly <mike.connolly@oracle.com <ma...@oracle.com>> wrote:
>> I checked out the master branch of Nifi and noticed the GetDynamoDB processor which was something I really needed for a workflow. I tried using it to return the json representation of a DynamoDB item. It looks like it will only return a json representation of a a single attribute of a DynamoDB item. I'm wondering it this was the intention or if there is a specific value you need to set in the processor configuration that will return the whole item. Anybody ever use this?
>>
>> Thanks,
>>
>> Mike Connolly
>>
>
Re: GetDynamoDB Processor returns only item attribute from DynamoDB
but not all attributes.
Posted by Mike Connolly <mi...@oracle.com>.
James,
Hmmm. I'm not too familiar with DynamoDB or NiFi, but having a
processor that can only return one attribute at a time doesn't seem very
useful. I would think the processor would retrieve all the attributes
and package them into one variable and send it back as one unit so when
the user configures the processor they could do something like set the
json attribute value to "ALL" or something like that. I tried modifying
the processor source code to call item.getJSON() to populate the return
value with everything, but something isn't liking { brace in the string
value that gets sent back. Its complaining about unacceptable characters
in the string. I might need to swich to MongoDB as my datasource and see
if that works.
Thanks,
Mike
On 07/01/2016 02:41 AM, James Wing wrote:
> Mike,
>
> I believe it was an intentional design choice to separate the
> hash/range keys from the flowfile content for validation and
> handling. There is currently no option in PutDynamoDB or GetDynamoDB
> to match the entire flowfile content to the entire DynamoDB item, but
> I believe that is certainly a possible extension.
>
> Thanks,
>
> James
>
> On Thu, Jun 30, 2016 at 9:57 AM, Mike Connolly
> <mike.connolly@oracle.com <ma...@oracle.com>> wrote:
>
> I checked out the master branch of Nifi and noticed the
> GetDynamoDB processor which was something I really needed for a
> workflow. I tried using it to return the json representation of a
> DynamoDB item. It looks like it will only return a json
> representation of a a single attribute of a DynamoDB item. I'm
> wondering it this was the intention or if there is a specific
> value you need to set in the processor configuration that will
> return the whole item. Anybody ever use this?
>
> Thanks,
>
> Mike Connolly
>
>