[jira] [Created] (NIFI-7194) Record Path (and Readers) should be able to select/filter on fields within arrays

["Chris Sampson (Jira)" <ji...@apache.org>]

Chris Sampson created NIFI-7194:
-----------------------------------

             Summary: Record Path (and Readers) should be able to select/filter on fields within arrays
                 Key: NIFI-7194
                 URL: https://issues.apache.org/jira/browse/NIFI-7194
             Project: Apache NiFi
          Issue Type: Improvement
    Affects Versions: 1.11.1
            Reporter: Chris Sampson


Given a sample input like

{code:json}
{
	"field": "value",
	"a_date": "2020-02-01",
	"ids": [
		{
			"id": "1",
			"id_date": "2020-02-24"
		},
		{
			"id": "2",
			"id_date": "2020-02-23"
		}
	],
	"another": {
		"some_date": "2020-02-02"
	}
}
{code}

It would be useful if one could use NiFi's Record Path to generically reference fields nested within arrays as well as top-level/nested objects in order to make updates to their values, e.g. as part of a call to the UpdateRecord processor.

For example, to reference all non-array fields whose name contains `date`, one could currently use: ` //*[matchesRegex(fieldName(.), '(^|.+_)date(_.+|$)')]`; however there is no equivalent for referencing the date fields within the arrays.

Such a RecordPath could look like `//[0..-1]/*[matchesRegex(fieldName(.), '(^|.+_)date(_.+|$)')]`, however this is currently marked as invalid by the UpdateRecord processor validation (and presumably therefore doesn't work). One tricky addition to this would be how to handle multiple levels of nesting of arrays/objects and how much descending through the Record tree should be possible.

An addition to this would be to allow for matching of Map fields too, e.g. with something like `//[]/*[matchesRegex(fieldName(.), '(^|.+_)date(_.+|$)')]`.


An example use-case for this is where incoming data structure is not set/known and may contain arrays (or even maps) with date fields where the flow is trying to support multiple date formats (e.g. yyyy-MM-dd and dd/MM/yyyy) but wants to harmonise all dates to a single format for further processing (e.g. dd/MM/yyyy).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)