[jira] [Created] (NIFI-7197) In-place replacement in LookupRecord processor

["Pierre Villard (Jira)" <ji...@apache.org>]

Pierre Villard created NIFI-7197:
------------------------------------

             Summary: In-place replacement in LookupRecord processor
                 Key: NIFI-7197
                 URL: https://issues.apache.org/jira/browse/NIFI-7197
             Project: Apache NiFi
          Issue Type: Improvement
          Components: Extensions
            Reporter: Pierre Villard
            Assignee: Pierre Villard


One of the promise of the record-oriented processors is to avoid any data splitting to increase overall performances. This JIRA is about improving the {{LookupRecord}} processor to support lookup replacement when the lookup key field is located into an array of the input data.

Here is an example of data:

{code:java}
  {
    "locales": [
      {
        "language" : "fr",
        "region" : "CA"
      },
      {
        "language" : "fr",
        "region" : "FR"
      }
    ]
  }
{code}

If we want to replace language for each occurrence in the array, we'd need to slip the data, do the lookup and then merge back everything together.

Unfortunately we can't use something like:

{code:java}
Result RecordPath: /locales[*]/language
key: /locales[*]/language
{code}

Because the provided result record path returns multiple fields.

With this JIRA, I suggest to add a new boolean property for "in-place replacement" that would default to false to preserve current behavior. It would also allow to do multi replacements at one time.

If true, the following requirements would be made:
- the lookup service should be a single key lookup service
- replacement will only support simple types

Then, every dynamic property containing record path would be evaluated and the corresponding field will be used as the key with the lookup service and the field will be updated with the value returned by the lookup service.

Example:

{code:java}
[
  {
    "foo" : {
      "foo" : "key"
    },
    "locales": [
      {
        "language" : "fr",
        "region" : "CA"
      },
      {
        "language" : "fr",
        "region" : "FR"
      }
    ]
  }, 
    {
      "foo" : {
        "foo" : "key"
      },
      "locales": [
        {
          "language" : "fr",
          "region" : "CA"
        },
        {
          "language" : "fr",
          "region" : "FR"
        }
      ]
    }
]
{code}

With a lookup service containing:

{code:java}
fr => French
CA => Canada
FR => France
key => value
{code}

And the following dynamic properties:

{code:java}
- lookupLanguage => /locales[*]/language
- lookupRegion => /locales[*]/region
- lookupFoo => /foo/foo
{code}

should give

{code:java}
[
  {
    "foo" : {
      "foo" : "value"
    },
    "locales": [
      {
        "language" : "French",
        "region" : "Canada"
      },
      {
        "language" : "French",
        "region" : "France"
      }
    ]
  },  {
      "foo" : {
        "foo" : "value"
      },
      "locales": [
        {
          "language" : "French",
          "region" : "Canada"
        },
        {
          "language" : "French",
          "region" : "France"
        }
      ]
    }
]
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)