You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Keith Lim <Ke...@ds-iq.com> on 2016/05/12 18:52:58 UTC
How to extract scalar info of json array using EvaluateJsonPath
processor?
I have a flow file with json array and would like to use EvaluateJsonPath processor the extract the item count of that array. Does the Nifi Expression Language in combination of json path feature supports this without writing script? What is the syntax?
Thanks,
Keith
Re: How to extract scalar info of json array using EvaluateJsonPath
processor?
Posted by Keith Lim <Ke...@ds-iq.com>.
Thanks Aldrin for providing a clear sample in such a short time. That Advanced menu implementation is very eye-opening. :) I appreciate your great effort.
Thanks,
Keith
________________________________
From: Aldrin Piri <al...@gmail.com>
Sent: Thursday, May 12, 2016 6:36 PM
To: users@nifi.apache.org
Subject: Re: How to extract scalar info of json array using EvaluateJsonPath processor?
Hi Keith,
I threw together a quick example of how this is done and made it available as a GitHub Gist [1]. In that template, the core logic of transforming the evaluated JsonPath expression result is in the "Count IDs as Attribute" processor which is making use of the 'Advanced' menu for UpdateAttribute. As a caveat, this is quite quick and dirty, but illustrates how you can extract values from content to attributes for your case.
One thing I would like to highlight is that Expression Language (EL) is exclusively against FlowFile attributes. The pattern that typically happens is an extraction/selection of key features/characteristics from the content, promoted as attributes for further manipulation and handling. This approach is taken in the referenced template.
I additionally noticed that I was also incorrect in my previous statement and overlooked the length() operator for the JsonPath library [2]. This, however, does not seem to apply to expressions for EvaluateJsonPath and will likely need some additional inspection. I have created an issue for that [3].
Please let us know if you have any additional questions!
--aldrin
[1] https://gist.github.com/apiri/0e2d0c9b1a7a4f109fbc91da56693d30
[https://avatars3.githubusercontent.com/u/502889?v=3&s=400]<https://gist.github.com/apiri/0e2d0c9b1a7a4f109fbc91da56693d30>
Generates a sample JSON flowfile which then has: 1) An array of values extracted to an attribute 2) Using the 'Advanced' menu of UpdateAttributes, determines if the result IDs are non-empty and then performs a count using allDelinatedValues to create an id<https://gist.github.com/apiri/0e2d0c9b1a7a4f109fbc91da56693d30>
gist.github.com
Generates a sample JSON flowfile which then has: 1) An array of values extracted to an attribute 2) Using the 'Advanced' menu of UpdateAttributes, determines if the result IDs are non-empty and t...
[2] https://github.com/jayway/JsonPath#functions
[3] https://issues.apache.org/jira/browse/NIFI-1875
On Thu, May 12, 2016 at 2:52 PM, Keith Lim <Ke...@ds-iq.com>> wrote:
Hi Aldrin,
I am still not able to get this to work. I see that the expression language guide (http://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html) mentions how to work with flowfile attributes, but not on the flowfile content itself. What is the builtin variable that points to the flowfile content that I can used with the expression language?
For illustration, let's say I have the below json as the content of a flowfile. What processor can I used to take this flowfile as input and how can I extract the count of item in the json array "results" below. I want to assign it to a user defined attribute in the processor.
{ "results" : [ { "name" : "Jane Doe", "id" : "1" }, { "name" : "John Doe", "id" : "2" } ] }
Thanks for all your help.
Thanks,
keith
________________________________
From: Aldrin Piri <al...@gmail.com>>
Sent: Thursday, May 12, 2016 1:32 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: How to extract scalar info of json array using EvaluateJsonPath processor?
Hi Keith,
Scanning over some of the docs, it does not appear that JsonPath supports a count operator but could possibly be used to extract from your source document that could possibly be manipulated using something like allDelinatedValues [1]. Certainly not the most elegant approach, but could work.
If you would like some additional help, a sample to work from would be nice to give some more concrete assistance.
[1] http://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#alldelineatedvalues
On Thu, May 12, 2016 at 11:52 AM, Keith Lim <Ke...@ds-iq.com>> wrote:
I have a flow file with json array and would like to use EvaluateJsonPath processor the extract the item count of that array. Does the Nifi Expression Language in combination of json path feature supports this without writing script? What is the syntax?
Thanks,
Keith
Re: How to extract scalar info of json array using EvaluateJsonPath processor?
Posted by Aldrin Piri <al...@gmail.com>.
Hi Keith,
I threw together a quick example of how this is done and made it available
as a GitHub Gist [1]. In that template, the core logic of transforming the
evaluated JsonPath expression result is in the "Count IDs as Attribute"
processor which is making use of the 'Advanced' menu for UpdateAttribute.
As a caveat, this is quite quick and dirty, but illustrates how you can
extract values from content to attributes for your case.
One thing I would like to highlight is that Expression Language (EL) is
exclusively against FlowFile attributes. The pattern that typically
happens is an extraction/selection of key features/characteristics from the
content, promoted as attributes for further manipulation and handling.
This approach is taken in the referenced template.
I additionally noticed that I was also incorrect in my previous statement
and overlooked the length() operator for the JsonPath library [2]. This,
however, does not seem to apply to expressions for EvaluateJsonPath and
will likely need some additional inspection. I have created an issue for
that [3].
Please let us know if you have any additional questions!
--aldrin
[1] https://gist.github.com/apiri/0e2d0c9b1a7a4f109fbc91da56693d30
[2] https://github.com/jayway/JsonPath#functions
[3] https://issues.apache.org/jira/browse/NIFI-1875
On Thu, May 12, 2016 at 2:52 PM, Keith Lim <Ke...@ds-iq.com> wrote:
> Hi Aldrin,
>
> I am still not able to get this to work. I see that the expression
> language guide (
> http://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html)
> mentions how to work with flowfile attributes, but not on the flowfile
> content itself. What is the builtin variable that points to the flowfile
> content that I can used with the expression language?
>
> For illustration, let's say I have the below json as the content of a
> flowfile. What processor can I used to take this flowfile as input and how
> can I extract the count of item in the json array "results" below. I want
> to assign it to a user defined attribute in the processor.
>
> { "results" : [ { "name" : "Jane Doe", "id" : "1" }, { "name" :
> "John Doe", "id" : "2" } ] }
>
> Thanks for all your help.
>
> Thanks,
> keith
> ------------------------------
> *From:* Aldrin Piri <al...@gmail.com>
> *Sent:* Thursday, May 12, 2016 1:32 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: How to extract scalar info of json array using
> EvaluateJsonPath processor?
>
> Hi Keith,
>
> Scanning over some of the docs, it does not appear that JsonPath supports
> a count operator but could possibly be used to extract from your source
> document that could possibly be manipulated using something like
> allDelinatedValues [1]. Certainly not the most elegant approach, but could
> work.
>
> If you would like some additional help, a sample to work from would be
> nice to give some more concrete assistance.
>
> [1]
> http://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#alldelineatedvalues
>
> On Thu, May 12, 2016 at 11:52 AM, Keith Lim <Ke...@ds-iq.com> wrote:
>
>> I have a flow file with json array and would like to use EvaluateJsonPath
>> processor the extract the item count of that array. Does the Nifi
>> Expression Language in combination of json path feature supports this without
>> writing script? What is the syntax?
>>
>> Thanks,
>> Keith
>>
>>
>>
>
Re: How to extract scalar info of json array using EvaluateJsonPath
processor?
Posted by Keith Lim <Ke...@ds-iq.com>.
Hi Aldrin,
I am still not able to get this to work. I see that the expression language guide (http://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html) mentions how to work with flowfile attributes, but not on the flowfile content itself. What is the builtin variable that points to the flowfile content that I can used with the expression language?
For illustration, let's say I have the below json as the content of a flowfile. What processor can I used to take this flowfile as input and how can I extract the count of item in the json array "results" below. I want to assign it to a user defined attribute in the processor.
{ "results" : [ { "name" : "Jane Doe", "id" : "1" }, { "name" : "John Doe", "id" : "2" } ] }
Thanks for all your help.
Thanks,
keith
________________________________
From: Aldrin Piri <al...@gmail.com>
Sent: Thursday, May 12, 2016 1:32 PM
To: users@nifi.apache.org
Subject: Re: How to extract scalar info of json array using EvaluateJsonPath processor?
Hi Keith,
Scanning over some of the docs, it does not appear that JsonPath supports a count operator but could possibly be used to extract from your source document that could possibly be manipulated using something like allDelinatedValues [1]. Certainly not the most elegant approach, but could work.
If you would like some additional help, a sample to work from would be nice to give some more concrete assistance.
[1] http://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#alldelineatedvalues
On Thu, May 12, 2016 at 11:52 AM, Keith Lim <Ke...@ds-iq.com>> wrote:
I have a flow file with json array and would like to use EvaluateJsonPath processor the extract the item count of that array. Does the Nifi Expression Language in combination of json path feature supports this without writing script? What is the syntax?
Thanks,
Keith
Re: How to extract scalar info of json array using EvaluateJsonPath processor?
Posted by Aldrin Piri <al...@gmail.com>.
Hi Keith,
Scanning over some of the docs, it does not appear that JsonPath supports a
count operator but could possibly be used to extract from your source
document that could possibly be manipulated using something like
allDelinatedValues [1]. Certainly not the most elegant approach, but could
work.
If you would like some additional help, a sample to work from would be nice
to give some more concrete assistance.
[1]
http://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#alldelineatedvalues
On Thu, May 12, 2016 at 11:52 AM, Keith Lim <Ke...@ds-iq.com> wrote:
> I have a flow file with json array and would like to use EvaluateJsonPath
> processor the extract the item count of that array. Does the Nifi
> Expression Language in combination of json path feature supports this without
> writing script? What is the syntax?
>
> Thanks,
> Keith
>
>
>