Unexpected JSON Path behaviour with NiFi Expression Language in UpdateAttribute (and confusing ReplaceText output)

[Chris Sampson <ch...@naimuri.com>]

Hi,

(Copied form Slack channel)

I’m seeing some confusing behaviours with the Expression Language json path functions and ReplaceText for adding JSON into a file (1.11.3):
given input of (read into a FlowFile attribute named record_json via the EvaluateJsonPath processor):
{
  "person": {
    "addresses": [
      {"current": true, "postcode": "test"}
    ]
  },
  "metadata": {}
}
UpdateAttribute to set address_1 as ${record_json:jsonPath('$.person.addresses[?(@.current)].address_1')} results in a value of []. yet I’d have expected an empty String (as the json path doesn’t exist) - doing the same but for postcode correctly returns a value of test
UpdateAttribute to try and update record_json with ${record_json:jsonPathPut('$.metadata', 'a_key', 'a_value')} results in the following output:
{
  "person": {
    "addresses": [
      {"current": true, "postcode": "test"}
    ]
  },
  "metadata": {},
  "metadata": {"a_key": "a_value"}
}
i.e. there’s a duplicated metadata object, where I’d have expected the new key/value pair to be added to the existing object instead (but can’t use jsonPathAdd as that’s for arrays, not objects
Also, when I try to then ReplaceText to put the attribute back into the file content (for various reasons I need it as both file content and an attribute later in the Flow), the above example output seems to lose the additional metadata object and just the blank object remains, i.e. a ReplaceText processor with settings Search Value = (?s)$.*$, Replacement Value = ${record_json}, Replacement Strategy = Regex Replace, Evaluation Mode = Entire text and the above example output gives file content of:
{
  "person": {
    "addresses": [
      {"current": true, "postcode": "test"}
    ]
  },
  "metadata": {}
}



Cheers,

-- 
Chris Sampson

t: +44 7867 843675
e: chris.sampson@naimuri.com

Re: Unexpected JSON Path behaviour with NiFi Expression Language in UpdateAttribute (and confusing ReplaceText output)

[Chris Sampson <ch...@naimuri.com>]

A little further investigation/tinkering:

I suspect the ReplaceText issue is that the JSON is being parsed and the duplicate metadata field is causing a problem, then one of them is being thrown away
adding a teplaceFirst function call the remove the ,"metadata":{} from the attribute first has fixed the issue… but that’s not particularly great (and caused by the original metadata duplication caused by jsonPathPut)


additionally jsonPathPut can only add scalars, so I can’t create the metadata object as part of UpdateAttribtue process, it needs to exist beforehand (which I thankfully have the opportunity to do, but it’s frustrating)
would be helpful if jsonPathPut could either put objects as well as scalars or, given a non-existent nested path, e.g. $.metadata.a_key, created the intermediate objects in order to add the key
if I create & initialise the metadata object prior to the UpdateAttribute processor, e.g. $.metadata.is_initialised: true, then the jsonPathPut for a_key: a_value works as expected (i.e. uses the existing object, not by creating a new one); but then trying to tidy up the unwanted field with a jsonPathDelete('$.metadata.is_initialised') call in UpdateAttribute results in all fields of metadata being deleted (i.e. a_key is also deleted)! Although this only seems to happen when the object from which I’m trying to delete data using jsonPathDelete has previously had data added to it with jsonPathPut - objects that hadn’t previously been modified can have individual fields deleted without issue.


So I’m having to fudge things a fair amount in order to manipulate the JSON within the attribute (create/add fields to an object, delete fields from the object

Cheers,

-- 
Chris Sampson

t: +44 7867 843675
e: chris.sampson@naimuri.com

> On 6 Mar 2020, at 12:45, Chris Sampson <ch...@naimuri.com> wrote:
> 
> Hi,
> 
> (Copied form Slack channel)
> 
> I’m seeing some confusing behaviours with the Expression Language json path functions and ReplaceText for adding JSON into a file (1.11.3):
> given input of (read into a FlowFile attribute named record_json via the EvaluateJsonPath processor):
> {
>   "person": {
>     "addresses": [
>       {"current": true, "postcode": "test"}
>     ]
>   },
>   "metadata": {}
> }
> UpdateAttribute to set address_1 as ${record_json:jsonPath('$.person.addresses[?(@.current)].address_1')} results in a value of []. yet I’d have expected an empty String (as the json path doesn’t exist) - doing the same but for postcode correctly returns a value of test
> UpdateAttribute to try and update record_json with ${record_json:jsonPathPut('$.metadata', 'a_key', 'a_value')} results in the following output:
> {
>   "person": {
>     "addresses": [
>       {"current": true, "postcode": "test"}
>     ]
>   },
>   "metadata": {},
>   "metadata": {"a_key": "a_value"}
> }
> i.e. there’s a duplicated metadata object, where I’d have expected the new key/value pair to be added to the existing object instead (but can’t use jsonPathAdd as that’s for arrays, not objects
> Also, when I try to then ReplaceText to put the attribute back into the file content (for various reasons I need it as both file content and an attribute later in the Flow), the above example output seems to lose the additional metadata object and just the blank object remains, i.e. a ReplaceText processor with settings Search Value = (?s)$.*$, Replacement Value = ${record_json}, Replacement Strategy = Regex Replace, Evaluation Mode = Entire text and the above example output gives file content of:
> {
>   "person": {
>     "addresses": [
>       {"current": true, "postcode": "test"}
>     ]
>   },
>   "metadata": {}
> }
> 
> 
> 
> Cheers,
> 
> -- 
> Chris Sampson
> 
> t: +44 7867 843675
> e: chris.sampson@naimuri.com <ma...@naimuri.com>