You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Darren Govoni <da...@ontrenet.com> on 2020/03/31 12:40:18 UTC

Adding Nested Properties/JSON

Hi,
   I want to use Nifi to design a flow that modifies, updates, etc a nested JSON document (or that can finally output one at the end).

For example:

{
   "name":"this and that",
   "field":"value",
   "others": [
       {"name":"here and there"},
       ...
   ]
}

What's the best approach to this using Nifi?

Thanks in advance!
Darren

Re: Adding Nested Properties/JSON

Posted by Etienne Jouvin <la...@gmail.com>.
With Jolt transformation, you can do something like :


Input
{
  "name": "this and that",
  "field": "value"
}

Transformation 1
[
   {
      "operation":"modify-overwrite-beta",
      "spec":{
         "others":{
            "[0]":[
               {
                  "name":"here and there"
               }
            ]
         }
      }
   }
]

Transformation 2
[
   {
      "operation":"modify-overwrite-beta",
      "spec":{
         "others":{
            "[1]":[
               {
                  "name":"one and two"
               }
            ]
         }
      }
   }
]



Or in one raw :
[
   {
      "operation":"modify-overwrite-beta",
      "spec":{
         "others":{
            "[0]":[
               {
                  "name":"here and there"
               }
            ],
            "[1]":[
               {
                  "name":"one and two"
               }
            ]
         }
      }
   }
]

But with this, the difficult part is to send the processor the value [0],
[1] ....
You have to put a variable wih the complete value and put it in the JOLT
specification like
  [
   {
      "operation":"modify-overwrite-beta",
      "spec":{
         "others":{
            "${myVar}":[
               {
                  "name":"here and there"
               }
            ]
         }
      }
   }
]
Where myVar contains [0]

But may be you the value to add is not a constant.
This will be another issue, but possible (I do it lot of time)



<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Garanti
sans virus. www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Le mar. 31 mars 2020 à 15:40, Darren Govoni <da...@ontrenet.com> a écrit :

> Sure. Thank you.
>
> Processor #1 creates this JSON
>
> {
>    "name":"this and that",
>    "field":"value"
> }
>
> passes to Processor #2 which adds a record to a sub-field
>
> {
>    "name":"this and that",
>    "field":"value",
>    "others": [
>       {"name":"here and there"}
>     ]
> }
>
> passes to Processor #3 which also adds a record to "others".
>
> {
>    "name":"this and that",
>    "field":"value",
>    "others": [
>       {"name":"here and there"},
>       {"name":"one and two"},
>     ]
> }
>
> which is the final output. So it's more building a JSON than transforming,
> sorta.
> ------------------------------
> *From:* Etienne Jouvin <la...@gmail.com>
> *Sent:* Tuesday, March 31, 2020 9:37 AM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Re: Adding Nested Properties/JSON
>
> Can you post example of input and expected result.
>
> For adding, you can use default or modify-overwrite-beta
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Garanti
> sans virus. www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_5258469381324177961_x_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> Le mar. 31 mars 2020 à 15:30, Darren Govoni <da...@ontrenet.com> a
> écrit :
>
> Hi. Thank you.
>
> In looking at the Jolt docs these are the operations:
>
> shift, sort, cardinality, modify-default-beta, modify-overwrite-beta,
> modify-define-beta, or remove
>
> I primarily need "add" such that I can add nested elements or add elements
> to an array already in the JSON.
>
> Can a single Jolt processor do this? Or do I need to merge two inputs to
> join them into a single JSON?
>
> thanks in advance!
> Darren
>
>
> ------------------------------
> *From:* Etienne Jouvin <la...@gmail.com>
> *Sent:* Tuesday, March 31, 2020 8:52 AM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Re: Adding Nested Properties/JSON
>
> Hello.
>
> Jolt transformation.
>
> Etienne
>
> Le mar. 31 mars 2020 à 14:40, Darren Govoni <da...@ontrenet.com> a
> écrit :
>
> Hi,
>    I want to use Nifi to design a flow that modifies, updates, etc a
> nested JSON document (or that can finally output one at the end).
>
> For example:
>
> {
>    "name":"this and that",
>    "field":"value",
>    "others": [
>        {"name":"here and there"},
>        ...
>    ]
> }
>
> What's the best approach to this using Nifi?
>
> Thanks in advance!
> Darren
>
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Garanti
> sans virus. www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_5258469381324177961_x_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>

Re: Adding Nested Properties/JSON

Posted by Darren Govoni <da...@ontrenet.com>.
Sure. Thank you.

Processor #1 creates this JSON

{
   "name":"this and that",
   "field":"value"
}

passes to Processor #2 which adds a record to a sub-field

{
   "name":"this and that",
   "field":"value",
   "others": [
      {"name":"here and there"}
    ]
}

passes to Processor #3 which also adds a record to "others".

{
   "name":"this and that",
   "field":"value",
   "others": [
      {"name":"here and there"},
      {"name":"one and two"},
    ]
}

which is the final output. So it's more building a JSON than transforming, sorta.
________________________________
From: Etienne Jouvin <la...@gmail.com>
Sent: Tuesday, March 31, 2020 9:37 AM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Adding Nested Properties/JSON

Can you post example of input and expected result.

For adding, you can use default or modify-overwrite-beta

[https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif]<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>  Garanti sans virus. www.avast.com<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>

Le mar. 31 mars 2020 à 15:30, Darren Govoni <da...@ontrenet.com>> a écrit :
Hi. Thank you.

In looking at the Jolt docs these are the operations:

shift, sort, cardinality, modify-default-beta, modify-overwrite-beta, modify-define-beta, or remove

I primarily need "add" such that I can add nested elements or add elements to an array already in the JSON.

Can a single Jolt processor do this? Or do I need to merge two inputs to join them into a single JSON?

thanks in advance!
Darren


________________________________
From: Etienne Jouvin <la...@gmail.com>>
Sent: Tuesday, March 31, 2020 8:52 AM
To: users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>
Subject: Re: Adding Nested Properties/JSON

Hello.

Jolt transformation.

Etienne

Le mar. 31 mars 2020 à 14:40, Darren Govoni <da...@ontrenet.com>> a écrit :
Hi,
   I want to use Nifi to design a flow that modifies, updates, etc a nested JSON document (or that can finally output one at the end).

For example:

{
   "name":"this and that",
   "field":"value",
   "others": [
       {"name":"here and there"},
       ...
   ]
}

What's the best approach to this using Nifi?

Thanks in advance!
Darren

[https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif]<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>  Garanti sans virus. www.avast.com<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>

Re: Adding Nested Properties/JSON

Posted by Etienne Jouvin <la...@gmail.com>.
Can you post example of input and expected result.

For adding, you can use default or modify-overwrite-beta

<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Garanti
sans virus. www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Le mar. 31 mars 2020 à 15:30, Darren Govoni <da...@ontrenet.com> a écrit :

> Hi. Thank you.
>
> In looking at the Jolt docs these are the operations:
>
> shift, sort, cardinality, modify-default-beta, modify-overwrite-beta,
> modify-define-beta, or remove
>
> I primarily need "add" such that I can add nested elements or add elements
> to an array already in the JSON.
>
> Can a single Jolt processor do this? Or do I need to merge two inputs to
> join them into a single JSON?
>
> thanks in advance!
> Darren
>
>
> ------------------------------
> *From:* Etienne Jouvin <la...@gmail.com>
> *Sent:* Tuesday, March 31, 2020 8:52 AM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Re: Adding Nested Properties/JSON
>
> Hello.
>
> Jolt transformation.
>
> Etienne
>
> Le mar. 31 mars 2020 à 14:40, Darren Govoni <da...@ontrenet.com> a
> écrit :
>
> Hi,
>    I want to use Nifi to design a flow that modifies, updates, etc a
> nested JSON document (or that can finally output one at the end).
>
> For example:
>
> {
>    "name":"this and that",
>    "field":"value",
>    "others": [
>        {"name":"here and there"},
>        ...
>    ]
> }
>
> What's the best approach to this using Nifi?
>
> Thanks in advance!
> Darren
>
>
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Garanti
sans virus. www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: Adding Nested Properties/JSON

Posted by Darren Govoni <da...@ontrenet.com>.
Hi. Thank you.

In looking at the Jolt docs these are the operations:

shift, sort, cardinality, modify-default-beta, modify-overwrite-beta, modify-define-beta, or remove

I primarily need "add" such that I can add nested elements or add elements to an array already in the JSON.

Can a single Jolt processor do this? Or do I need to merge two inputs to join them into a single JSON?

thanks in advance!
Darren


________________________________
From: Etienne Jouvin <la...@gmail.com>
Sent: Tuesday, March 31, 2020 8:52 AM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Adding Nested Properties/JSON

Hello.

Jolt transformation.

Etienne

Le mar. 31 mars 2020 à 14:40, Darren Govoni <da...@ontrenet.com>> a écrit :
Hi,
   I want to use Nifi to design a flow that modifies, updates, etc a nested JSON document (or that can finally output one at the end).

For example:

{
   "name":"this and that",
   "field":"value",
   "others": [
       {"name":"here and there"},
       ...
   ]
}

What's the best approach to this using Nifi?

Thanks in advance!
Darren

Re: Adding Nested Properties/JSON

Posted by Etienne Jouvin <la...@gmail.com>.
Hello.

Jolt transformation.

Etienne

Le mar. 31 mars 2020 à 14:40, Darren Govoni <da...@ontrenet.com> a écrit :

> Hi,
>    I want to use Nifi to design a flow that modifies, updates, etc a
> nested JSON document (or that can finally output one at the end).
>
> For example:
>
> {
>    "name":"this and that",
>    "field":"value",
>    "others": [
>        {"name":"here and there"},
>        ...
>    ]
> }
>
> What's the best approach to this using Nifi?
>
> Thanks in advance!
> Darren
>