You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Angel Kafazov <ak...@cst-bg.net> on 2016/07/01 12:08:53 UTC
Re: Multiple JSON fields validation with EvaluateJsonPath
Hi Mark,
The EvaluateJsonPath always sets the attribute with empty string value,
regardless if the JSON field exists or not. It is only possible to check if
the field has a empty value or not, but not possible to check if it
actually exists. Am I missing something?
Best Regards,
Angel
On Thu, Jun 30, 2016 at 7:31 PM, Mark Payne <ma...@hotmail.com> wrote:
> Hi Angel,
>
> I don't know the reasoning behind only using 'unmatched' for FlowFile
> Content, but I am guessing
> it has to do with the idea that the FlowFile is not modified when routed
> to this relationship and the
> author probably wanted to avoid the ambiguity that would occur when some
> paths matched but
> others didn't.
>
> That being said, the pattern that is typically followed is to use
> EvaluateJsonPath with a destination
> of attribute and then to follow this processor with a RouteOnAttribute
> processor to check if all of the
> attributes were added.
>
> Thanks
> -Mark
>
>
> On Jun 30, 2016, at 12:05 PM, Angel Kafazov <ak...@cst-bg.net> wrote:
>
> Hello,
>
> I am considering using EvaluateJsonPath processor to check if JSON
> formatted FlowFile content contains a number of predefined fields. The
> processor defines three relationships: failure, unmatched and matched as
> follows:
>
> failure FlowFiles are routed to this relationship when the JsonPath
> cannot be evaluated against the content of the FlowFile; for instance, if
> the FlowFile is not valid JSON
> unmatched FlowFiles are routed to this relationship when the JsonPath
> does not match the content of the FlowFile and the Destination is set to
> flowfile-content
> matched FlowFiles are routed to this relationship when the JsonPath is
> successfully evaluated and the FlowFile is modified as a result
>
> If the flowfile-content Destination is used, the existance only one field
> can be checked. To validate multiple fields, flowfile-attribute destination
> must be used, this means however that the unmatched relationship will not
> be used.
>
> This actually prevents one from being able to check existance multiple
> JSON fields in the content, if I understand correctly. What is the reason
> for this implementation? Is there any other processor that could be used to
> achieve the task?
>
> Regards,
> Angel
>
>
> Angel Kafazov
> CST LTD
> Bul. Bulgaria, blok 10, 6500 Svilengrad
> Bulgaria
> +359877336002 (BG Mobile)
> +4961509729845 (DE Landline)
> Email: akafazov@cst-bg.net
> http://cst-bg.net
>
>
>
Re: Multiple JSON fields validation with EvaluateJsonPath
Posted by Mark Payne <ma...@hotmail.com>.
Excellent! Yes, a pull request via github is typically the easiest way to contribute back.
> On Jul 1, 2016, at 10:06 AM, Angel Kafazov <ak...@cst-bg.net> wrote:
>
> Hi Mark,
>
> Sure, I would like to do it. I registered account in Jira. Should I create a pull request on the github repo when done?
>
> Regards,
> Angel
>
>
> On Fri, Jul 1, 2016 at 4:43 PM, Mark Payne <markap14@hotmail.com <ma...@hotmail.com>> wrote:
> Angel,
>
> OK, sounds like we've got a good path forward. I created a JIRA for this [1].
>
> Is this by chance something that you would be able / interested in working on?
>
> Thanks
> -Mark
>
> [1] https://issues.apache.org/jira/browse/NIFI-2161 <https://issues.apache.org/jira/browse/NIFI-2161>
>
>
>
>> On Jul 1, 2016, at 9:10 AM, Angel Kafazov <akafazov@cst-bg.net <ma...@cst-bg.net>> wrote:
>>
>> Hi Mark,
>>
>> Yes, I think this option would be exactly what I am looking for. It would make sense to add it, as there doesn't seem to be any other way around this using any of the available processors.
>>
>> Best Regards,
>> Angel
>>
>>
>> On Fri, Jul 1, 2016 at 4:01 PM, Mark Payne <markap14@hotmail.com <ma...@hotmail.com>> wrote:
>> Ah, I did miss that. In the cases that I have used it, an empty string would not be a valid value in the JSON, so I
>> have been able to route on whether or not the value is 'null or empty' by checking ${myAttribute:isEmpty()}.
>> Would it help if there were a new option in this Processor for the "Null Value Representation" such that if the value
>> does not exist, no attribute is even added? Then you'd be able to simply check for the existence of the attribute.
>>
>> Thanks
>> -Mark
>>
>>> On Jul 1, 2016, at 8:08 AM, Angel Kafazov <akafazov@cst-bg.net <ma...@cst-bg.net>> wrote:
>>>
>>> Hi Mark,
>>>
>>> The EvaluateJsonPath always sets the attribute with empty string value, regardless if the JSON field exists or not. It is only possible to check if the field has a empty value or not, but not possible to check if it actually exists. Am I missing something?
>>>
>>> Best Regards,
>>> Angel
>>>
>>>
>>> On Thu, Jun 30, 2016 at 7:31 PM, Mark Payne <markap14@hotmail.com <ma...@hotmail.com>> wrote:
>>> Hi Angel,
>>>
>>> I don't know the reasoning behind only using 'unmatched' for FlowFile Content, but I am guessing
>>> it has to do with the idea that the FlowFile is not modified when routed to this relationship and the
>>> author probably wanted to avoid the ambiguity that would occur when some paths matched but
>>> others didn't.
>>>
>>> That being said, the pattern that is typically followed is to use EvaluateJsonPath with a destination
>>> of attribute and then to follow this processor with a RouteOnAttribute processor to check if all of the
>>> attributes were added.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>>> On Jun 30, 2016, at 12:05 PM, Angel Kafazov <akafazov@cst-bg.net <ma...@cst-bg.net>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> I am considering using EvaluateJsonPath processor to check if JSON formatted FlowFile content contains a number of predefined fields. The processor defines three relationships: failure, unmatched and matched as follows:
>>>>
>>>> failure FlowFiles are routed to this relationship when the JsonPath cannot be evaluated against the content of the FlowFile; for instance, if the FlowFile is not valid JSON
>>>> unmatched FlowFiles are routed to this relationship when the JsonPath does not match the content of the FlowFile and the Destination is set to flowfile-content
>>>> matched FlowFiles are routed to this relationship when the JsonPath is successfully evaluated and the FlowFile is modified as a result
>>>>
>>>> If the flowfile-content Destination is used, the existance only one field can be checked. To validate multiple fields, flowfile-attribute destination must be used, this means however that the unmatched relationship will not be used.
>>>>
>>>> This actually prevents one from being able to check existance multiple JSON fields in the content, if I understand correctly. What is the reason for this implementation? Is there any other processor that could be used to achieve the task?
>>>>
>>>> Regards,
>>>> Angel
>>>>
>>>>
>>>> Angel Kafazov
>>>> CST LTD
>>>> Bul. Bulgaria, blok 10, 6500 Svilengrad
>>>> Bulgaria
>>>> +359877336002 <tel:%2B359877336002> (BG Mobile)
>>>> +4961509729845 <tel:%2B4961509729845> (DE Landline)
>>>> Email: akafazov@cst-bg.net <ma...@cst-bg.net>
>>>> http://cst-bg.net <http://cst-bg.net/>
>>>
>>
>>
>
>
Re: Multiple JSON fields validation with EvaluateJsonPath
Posted by Angel Kafazov <ak...@cst-bg.net>.
Hi Mark,
Sure, I would like to do it. I registered account in Jira. Should I create
a pull request on the github repo when done?
Regards,
Angel
On Fri, Jul 1, 2016 at 4:43 PM, Mark Payne <ma...@hotmail.com> wrote:
> Angel,
>
> OK, sounds like we've got a good path forward. I created a JIRA for this
> [1].
>
> Is this by chance something that you would be able / interested in working
> on?
>
> Thanks
> -Mark
>
> [1] https://issues.apache.org/jira/browse/NIFI-2161
>
>
>
> On Jul 1, 2016, at 9:10 AM, Angel Kafazov <ak...@cst-bg.net> wrote:
>
> Hi Mark,
>
> Yes, I think this option would be exactly what I am looking for. It would
> make sense to add it, as there doesn't seem to be any other way around this
> using any of the available processors.
>
> Best Regards,
> Angel
>
>
> On Fri, Jul 1, 2016 at 4:01 PM, Mark Payne <ma...@hotmail.com> wrote:
>
>> Ah, I did miss that. In the cases that I have used it, an empty string
>> would not be a valid value in the JSON, so I
>> have been able to route on whether or not the value is 'null or empty' by
>> checking ${myAttribute:isEmpty()}.
>> Would it help if there were a new option in this Processor for the "Null
>> Value Representation" such that if the value
>> does not exist, no attribute is even added? Then you'd be able to simply
>> check for the existence of the attribute.
>>
>> Thanks
>> -Mark
>>
>> On Jul 1, 2016, at 8:08 AM, Angel Kafazov <ak...@cst-bg.net> wrote:
>>
>> Hi Mark,
>>
>> The EvaluateJsonPath always sets the attribute with empty string value,
>> regardless if the JSON field exists or not. It is only possible to check if
>> the field has a empty value or not, but not possible to check if it
>> actually exists. Am I missing something?
>>
>> Best Regards,
>> Angel
>>
>>
>> On Thu, Jun 30, 2016 at 7:31 PM, Mark Payne <ma...@hotmail.com> wrote:
>>
>>> Hi Angel,
>>>
>>> I don't know the reasoning behind only using 'unmatched' for FlowFile
>>> Content, but I am guessing
>>> it has to do with the idea that the FlowFile is not modified when routed
>>> to this relationship and the
>>> author probably wanted to avoid the ambiguity that would occur when some
>>> paths matched but
>>> others didn't.
>>>
>>> That being said, the pattern that is typically followed is to use
>>> EvaluateJsonPath with a destination
>>> of attribute and then to follow this processor with a RouteOnAttribute
>>> processor to check if all of the
>>> attributes were added.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> On Jun 30, 2016, at 12:05 PM, Angel Kafazov <ak...@cst-bg.net> wrote:
>>>
>>> Hello,
>>>
>>> I am considering using EvaluateJsonPath processor to check if JSON
>>> formatted FlowFile content contains a number of predefined fields. The
>>> processor defines three relationships: failure, unmatched and matched as
>>> follows:
>>>
>>> failure FlowFiles are routed to this relationship when the JsonPath
>>> cannot be evaluated against the content of the FlowFile; for instance, if
>>> the FlowFile is not valid JSON
>>> unmatched FlowFiles are routed to this relationship when the JsonPath
>>> does not match the content of the FlowFile and the Destination is set to
>>> flowfile-content
>>> matched FlowFiles are routed to this relationship when the JsonPath is
>>> successfully evaluated and the FlowFile is modified as a result
>>>
>>> If the flowfile-content Destination is used, the existance only one
>>> field can be checked. To validate multiple fields, flowfile-attribute
>>> destination must be used, this means however that the unmatched
>>> relationship will not be used.
>>>
>>> This actually prevents one from being able to check existance multiple
>>> JSON fields in the content, if I understand correctly. What is the reason
>>> for this implementation? Is there any other processor that could be used to
>>> achieve the task?
>>>
>>> Regards,
>>> Angel
>>>
>>>
>>> Angel Kafazov
>>> CST LTD
>>> Bul. Bulgaria, blok 10, 6500 Svilengrad
>>> Bulgaria
>>> +359877336002 (BG Mobile)
>>> +4961509729845 (DE Landline)
>>> Email: akafazov@cst-bg.net
>>> http://cst-bg.net
>>>
>>>
>>>
>>
>>
>
>
Re: Multiple JSON fields validation with EvaluateJsonPath
Posted by Mark Payne <ma...@hotmail.com>.
Angel,
OK, sounds like we've got a good path forward. I created a JIRA for this [1].
Is this by chance something that you would be able / interested in working on?
Thanks
-Mark
[1] https://issues.apache.org/jira/browse/NIFI-2161 <https://issues.apache.org/jira/browse/NIFI-2161>
> On Jul 1, 2016, at 9:10 AM, Angel Kafazov <ak...@cst-bg.net> wrote:
>
> Hi Mark,
>
> Yes, I think this option would be exactly what I am looking for. It would make sense to add it, as there doesn't seem to be any other way around this using any of the available processors.
>
> Best Regards,
> Angel
>
>
> On Fri, Jul 1, 2016 at 4:01 PM, Mark Payne <markap14@hotmail.com <ma...@hotmail.com>> wrote:
> Ah, I did miss that. In the cases that I have used it, an empty string would not be a valid value in the JSON, so I
> have been able to route on whether or not the value is 'null or empty' by checking ${myAttribute:isEmpty()}.
> Would it help if there were a new option in this Processor for the "Null Value Representation" such that if the value
> does not exist, no attribute is even added? Then you'd be able to simply check for the existence of the attribute.
>
> Thanks
> -Mark
>
>> On Jul 1, 2016, at 8:08 AM, Angel Kafazov <akafazov@cst-bg.net <ma...@cst-bg.net>> wrote:
>>
>> Hi Mark,
>>
>> The EvaluateJsonPath always sets the attribute with empty string value, regardless if the JSON field exists or not. It is only possible to check if the field has a empty value or not, but not possible to check if it actually exists. Am I missing something?
>>
>> Best Regards,
>> Angel
>>
>>
>> On Thu, Jun 30, 2016 at 7:31 PM, Mark Payne <markap14@hotmail.com <ma...@hotmail.com>> wrote:
>> Hi Angel,
>>
>> I don't know the reasoning behind only using 'unmatched' for FlowFile Content, but I am guessing
>> it has to do with the idea that the FlowFile is not modified when routed to this relationship and the
>> author probably wanted to avoid the ambiguity that would occur when some paths matched but
>> others didn't.
>>
>> That being said, the pattern that is typically followed is to use EvaluateJsonPath with a destination
>> of attribute and then to follow this processor with a RouteOnAttribute processor to check if all of the
>> attributes were added.
>>
>> Thanks
>> -Mark
>>
>>
>>> On Jun 30, 2016, at 12:05 PM, Angel Kafazov <akafazov@cst-bg.net <ma...@cst-bg.net>> wrote:
>>>
>>> Hello,
>>>
>>> I am considering using EvaluateJsonPath processor to check if JSON formatted FlowFile content contains a number of predefined fields. The processor defines three relationships: failure, unmatched and matched as follows:
>>>
>>> failure FlowFiles are routed to this relationship when the JsonPath cannot be evaluated against the content of the FlowFile; for instance, if the FlowFile is not valid JSON
>>> unmatched FlowFiles are routed to this relationship when the JsonPath does not match the content of the FlowFile and the Destination is set to flowfile-content
>>> matched FlowFiles are routed to this relationship when the JsonPath is successfully evaluated and the FlowFile is modified as a result
>>>
>>> If the flowfile-content Destination is used, the existance only one field can be checked. To validate multiple fields, flowfile-attribute destination must be used, this means however that the unmatched relationship will not be used.
>>>
>>> This actually prevents one from being able to check existance multiple JSON fields in the content, if I understand correctly. What is the reason for this implementation? Is there any other processor that could be used to achieve the task?
>>>
>>> Regards,
>>> Angel
>>>
>>>
>>> Angel Kafazov
>>> CST LTD
>>> Bul. Bulgaria, blok 10, 6500 Svilengrad
>>> Bulgaria
>>> +359877336002 <tel:%2B359877336002> (BG Mobile)
>>> +4961509729845 <tel:%2B4961509729845> (DE Landline)
>>> Email: akafazov@cst-bg.net <ma...@cst-bg.net>
>>> http://cst-bg.net <http://cst-bg.net/>
>>
>
>
Re: Multiple JSON fields validation with EvaluateJsonPath
Posted by Angel Kafazov <ak...@cst-bg.net>.
Hi Mark,
Yes, I think this option would be exactly what I am looking for. It would
make sense to add it, as there doesn't seem to be any other way around this
using any of the available processors.
Best Regards,
Angel
On Fri, Jul 1, 2016 at 4:01 PM, Mark Payne <ma...@hotmail.com> wrote:
> Ah, I did miss that. In the cases that I have used it, an empty string
> would not be a valid value in the JSON, so I
> have been able to route on whether or not the value is 'null or empty' by
> checking ${myAttribute:isEmpty()}.
> Would it help if there were a new option in this Processor for the "Null
> Value Representation" such that if the value
> does not exist, no attribute is even added? Then you'd be able to simply
> check for the existence of the attribute.
>
> Thanks
> -Mark
>
> On Jul 1, 2016, at 8:08 AM, Angel Kafazov <ak...@cst-bg.net> wrote:
>
> Hi Mark,
>
> The EvaluateJsonPath always sets the attribute with empty string value,
> regardless if the JSON field exists or not. It is only possible to check if
> the field has a empty value or not, but not possible to check if it
> actually exists. Am I missing something?
>
> Best Regards,
> Angel
>
>
> On Thu, Jun 30, 2016 at 7:31 PM, Mark Payne <ma...@hotmail.com> wrote:
>
>> Hi Angel,
>>
>> I don't know the reasoning behind only using 'unmatched' for FlowFile
>> Content, but I am guessing
>> it has to do with the idea that the FlowFile is not modified when routed
>> to this relationship and the
>> author probably wanted to avoid the ambiguity that would occur when some
>> paths matched but
>> others didn't.
>>
>> That being said, the pattern that is typically followed is to use
>> EvaluateJsonPath with a destination
>> of attribute and then to follow this processor with a RouteOnAttribute
>> processor to check if all of the
>> attributes were added.
>>
>> Thanks
>> -Mark
>>
>>
>> On Jun 30, 2016, at 12:05 PM, Angel Kafazov <ak...@cst-bg.net> wrote:
>>
>> Hello,
>>
>> I am considering using EvaluateJsonPath processor to check if JSON
>> formatted FlowFile content contains a number of predefined fields. The
>> processor defines three relationships: failure, unmatched and matched as
>> follows:
>>
>> failure FlowFiles are routed to this relationship when the JsonPath
>> cannot be evaluated against the content of the FlowFile; for instance, if
>> the FlowFile is not valid JSON
>> unmatched FlowFiles are routed to this relationship when the JsonPath
>> does not match the content of the FlowFile and the Destination is set to
>> flowfile-content
>> matched FlowFiles are routed to this relationship when the JsonPath is
>> successfully evaluated and the FlowFile is modified as a result
>>
>> If the flowfile-content Destination is used, the existance only one field
>> can be checked. To validate multiple fields, flowfile-attribute destination
>> must be used, this means however that the unmatched relationship will not
>> be used.
>>
>> This actually prevents one from being able to check existance multiple
>> JSON fields in the content, if I understand correctly. What is the reason
>> for this implementation? Is there any other processor that could be used to
>> achieve the task?
>>
>> Regards,
>> Angel
>>
>>
>> Angel Kafazov
>> CST LTD
>> Bul. Bulgaria, blok 10, 6500 Svilengrad
>> Bulgaria
>> +359877336002 (BG Mobile)
>> +4961509729845 (DE Landline)
>> Email: akafazov@cst-bg.net
>> http://cst-bg.net
>>
>>
>>
>
>
Re: Multiple JSON fields validation with EvaluateJsonPath
Posted by Mark Payne <ma...@hotmail.com>.
Ah, I did miss that. In the cases that I have used it, an empty string would not be a valid value in the JSON, so I
have been able to route on whether or not the value is 'null or empty' by checking ${myAttribute:isEmpty()}.
Would it help if there were a new option in this Processor for the "Null Value Representation" such that if the value
does not exist, no attribute is even added? Then you'd be able to simply check for the existence of the attribute.
Thanks
-Mark
> On Jul 1, 2016, at 8:08 AM, Angel Kafazov <ak...@cst-bg.net> wrote:
>
> Hi Mark,
>
> The EvaluateJsonPath always sets the attribute with empty string value, regardless if the JSON field exists or not. It is only possible to check if the field has a empty value or not, but not possible to check if it actually exists. Am I missing something?
>
> Best Regards,
> Angel
>
>
> On Thu, Jun 30, 2016 at 7:31 PM, Mark Payne <markap14@hotmail.com <ma...@hotmail.com>> wrote:
> Hi Angel,
>
> I don't know the reasoning behind only using 'unmatched' for FlowFile Content, but I am guessing
> it has to do with the idea that the FlowFile is not modified when routed to this relationship and the
> author probably wanted to avoid the ambiguity that would occur when some paths matched but
> others didn't.
>
> That being said, the pattern that is typically followed is to use EvaluateJsonPath with a destination
> of attribute and then to follow this processor with a RouteOnAttribute processor to check if all of the
> attributes were added.
>
> Thanks
> -Mark
>
>
>> On Jun 30, 2016, at 12:05 PM, Angel Kafazov <akafazov@cst-bg.net <ma...@cst-bg.net>> wrote:
>>
>> Hello,
>>
>> I am considering using EvaluateJsonPath processor to check if JSON formatted FlowFile content contains a number of predefined fields. The processor defines three relationships: failure, unmatched and matched as follows:
>>
>> failure FlowFiles are routed to this relationship when the JsonPath cannot be evaluated against the content of the FlowFile; for instance, if the FlowFile is not valid JSON
>> unmatched FlowFiles are routed to this relationship when the JsonPath does not match the content of the FlowFile and the Destination is set to flowfile-content
>> matched FlowFiles are routed to this relationship when the JsonPath is successfully evaluated and the FlowFile is modified as a result
>>
>> If the flowfile-content Destination is used, the existance only one field can be checked. To validate multiple fields, flowfile-attribute destination must be used, this means however that the unmatched relationship will not be used.
>>
>> This actually prevents one from being able to check existance multiple JSON fields in the content, if I understand correctly. What is the reason for this implementation? Is there any other processor that could be used to achieve the task?
>>
>> Regards,
>> Angel
>>
>>
>> Angel Kafazov
>> CST LTD
>> Bul. Bulgaria, blok 10, 6500 Svilengrad
>> Bulgaria
>> +359877336002 <tel:%2B359877336002> (BG Mobile)
>> +4961509729845 <tel:%2B4961509729845> (DE Landline)
>> Email: akafazov@cst-bg.net <ma...@cst-bg.net>
>> http://cst-bg.net <http://cst-bg.net/>
>