You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Darren Govoni <da...@ontrenet.com> on 2020/03/31 12:40:18 UTC
Adding Nested Properties/JSON
Hi,
I want to use Nifi to design a flow that modifies, updates, etc a nested JSON document (or that can finally output one at the end).
For example:
{
"name":"this and that",
"field":"value",
"others": [
{"name":"here and there"},
...
]
}
What's the best approach to this using Nifi?
Thanks in advance!
Darren
Re: Adding Nested Properties/JSON
Posted by Etienne Jouvin <la...@gmail.com>.
With Jolt transformation, you can do something like :
Input
{
"name": "this and that",
"field": "value"
}
Transformation 1
[
{
"operation":"modify-overwrite-beta",
"spec":{
"others":{
"[0]":[
{
"name":"here and there"
}
]
}
}
}
]
Transformation 2
[
{
"operation":"modify-overwrite-beta",
"spec":{
"others":{
"[1]":[
{
"name":"one and two"
}
]
}
}
}
]
Or in one raw :
[
{
"operation":"modify-overwrite-beta",
"spec":{
"others":{
"[0]":[
{
"name":"here and there"
}
],
"[1]":[
{
"name":"one and two"
}
]
}
}
}
]
But with this, the difficult part is to send the processor the value [0],
[1] ....
You have to put a variable wih the complete value and put it in the JOLT
specification like
[
{
"operation":"modify-overwrite-beta",
"spec":{
"others":{
"${myVar}":[
{
"name":"here and there"
}
]
}
}
}
]
Where myVar contains [0]
But may be you the value to add is not a constant.
This will be another issue, but possible (I do it lot of time)
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Garanti
sans virus. www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Le mar. 31 mars 2020 à 15:40, Darren Govoni <da...@ontrenet.com> a écrit :
> Sure. Thank you.
>
> Processor #1 creates this JSON
>
> {
> "name":"this and that",
> "field":"value"
> }
>
> passes to Processor #2 which adds a record to a sub-field
>
> {
> "name":"this and that",
> "field":"value",
> "others": [
> {"name":"here and there"}
> ]
> }
>
> passes to Processor #3 which also adds a record to "others".
>
> {
> "name":"this and that",
> "field":"value",
> "others": [
> {"name":"here and there"},
> {"name":"one and two"},
> ]
> }
>
> which is the final output. So it's more building a JSON than transforming,
> sorta.
> ------------------------------
> *From:* Etienne Jouvin <la...@gmail.com>
> *Sent:* Tuesday, March 31, 2020 9:37 AM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Re: Adding Nested Properties/JSON
>
> Can you post example of input and expected result.
>
> For adding, you can use default or modify-overwrite-beta
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Garanti
> sans virus. www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_5258469381324177961_x_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> Le mar. 31 mars 2020 à 15:30, Darren Govoni <da...@ontrenet.com> a
> écrit :
>
> Hi. Thank you.
>
> In looking at the Jolt docs these are the operations:
>
> shift, sort, cardinality, modify-default-beta, modify-overwrite-beta,
> modify-define-beta, or remove
>
> I primarily need "add" such that I can add nested elements or add elements
> to an array already in the JSON.
>
> Can a single Jolt processor do this? Or do I need to merge two inputs to
> join them into a single JSON?
>
> thanks in advance!
> Darren
>
>
> ------------------------------
> *From:* Etienne Jouvin <la...@gmail.com>
> *Sent:* Tuesday, March 31, 2020 8:52 AM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Re: Adding Nested Properties/JSON
>
> Hello.
>
> Jolt transformation.
>
> Etienne
>
> Le mar. 31 mars 2020 à 14:40, Darren Govoni <da...@ontrenet.com> a
> écrit :
>
> Hi,
> I want to use Nifi to design a flow that modifies, updates, etc a
> nested JSON document (or that can finally output one at the end).
>
> For example:
>
> {
> "name":"this and that",
> "field":"value",
> "others": [
> {"name":"here and there"},
> ...
> ]
> }
>
> What's the best approach to this using Nifi?
>
> Thanks in advance!
> Darren
>
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Garanti
> sans virus. www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_5258469381324177961_x_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
Re: Adding Nested Properties/JSON
Posted by Darren Govoni <da...@ontrenet.com>.
Sure. Thank you.
Processor #1 creates this JSON
{
"name":"this and that",
"field":"value"
}
passes to Processor #2 which adds a record to a sub-field
{
"name":"this and that",
"field":"value",
"others": [
{"name":"here and there"}
]
}
passes to Processor #3 which also adds a record to "others".
{
"name":"this and that",
"field":"value",
"others": [
{"name":"here and there"},
{"name":"one and two"},
]
}
which is the final output. So it's more building a JSON than transforming, sorta.
________________________________
From: Etienne Jouvin <la...@gmail.com>
Sent: Tuesday, March 31, 2020 9:37 AM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Adding Nested Properties/JSON
Can you post example of input and expected result.
For adding, you can use default or modify-overwrite-beta
[https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif]<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Garanti sans virus. www.avast.com<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Le mar. 31 mars 2020 à 15:30, Darren Govoni <da...@ontrenet.com>> a écrit :
Hi. Thank you.
In looking at the Jolt docs these are the operations:
shift, sort, cardinality, modify-default-beta, modify-overwrite-beta, modify-define-beta, or remove
I primarily need "add" such that I can add nested elements or add elements to an array already in the JSON.
Can a single Jolt processor do this? Or do I need to merge two inputs to join them into a single JSON?
thanks in advance!
Darren
________________________________
From: Etienne Jouvin <la...@gmail.com>>
Sent: Tuesday, March 31, 2020 8:52 AM
To: users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>
Subject: Re: Adding Nested Properties/JSON
Hello.
Jolt transformation.
Etienne
Le mar. 31 mars 2020 à 14:40, Darren Govoni <da...@ontrenet.com>> a écrit :
Hi,
I want to use Nifi to design a flow that modifies, updates, etc a nested JSON document (or that can finally output one at the end).
For example:
{
"name":"this and that",
"field":"value",
"others": [
{"name":"here and there"},
...
]
}
What's the best approach to this using Nifi?
Thanks in advance!
Darren
[https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif]<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Garanti sans virus. www.avast.com<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Re: Adding Nested Properties/JSON
Posted by Etienne Jouvin <la...@gmail.com>.
Can you post example of input and expected result.
For adding, you can use default or modify-overwrite-beta
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Garanti
sans virus. www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Le mar. 31 mars 2020 à 15:30, Darren Govoni <da...@ontrenet.com> a écrit :
> Hi. Thank you.
>
> In looking at the Jolt docs these are the operations:
>
> shift, sort, cardinality, modify-default-beta, modify-overwrite-beta,
> modify-define-beta, or remove
>
> I primarily need "add" such that I can add nested elements or add elements
> to an array already in the JSON.
>
> Can a single Jolt processor do this? Or do I need to merge two inputs to
> join them into a single JSON?
>
> thanks in advance!
> Darren
>
>
> ------------------------------
> *From:* Etienne Jouvin <la...@gmail.com>
> *Sent:* Tuesday, March 31, 2020 8:52 AM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Re: Adding Nested Properties/JSON
>
> Hello.
>
> Jolt transformation.
>
> Etienne
>
> Le mar. 31 mars 2020 à 14:40, Darren Govoni <da...@ontrenet.com> a
> écrit :
>
> Hi,
> I want to use Nifi to design a flow that modifies, updates, etc a
> nested JSON document (or that can finally output one at the end).
>
> For example:
>
> {
> "name":"this and that",
> "field":"value",
> "others": [
> {"name":"here and there"},
> ...
> ]
> }
>
> What's the best approach to this using Nifi?
>
> Thanks in advance!
> Darren
>
>
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Garanti
sans virus. www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Re: Adding Nested Properties/JSON
Posted by Darren Govoni <da...@ontrenet.com>.
Hi. Thank you.
In looking at the Jolt docs these are the operations:
shift, sort, cardinality, modify-default-beta, modify-overwrite-beta, modify-define-beta, or remove
I primarily need "add" such that I can add nested elements or add elements to an array already in the JSON.
Can a single Jolt processor do this? Or do I need to merge two inputs to join them into a single JSON?
thanks in advance!
Darren
________________________________
From: Etienne Jouvin <la...@gmail.com>
Sent: Tuesday, March 31, 2020 8:52 AM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Adding Nested Properties/JSON
Hello.
Jolt transformation.
Etienne
Le mar. 31 mars 2020 à 14:40, Darren Govoni <da...@ontrenet.com>> a écrit :
Hi,
I want to use Nifi to design a flow that modifies, updates, etc a nested JSON document (or that can finally output one at the end).
For example:
{
"name":"this and that",
"field":"value",
"others": [
{"name":"here and there"},
...
]
}
What's the best approach to this using Nifi?
Thanks in advance!
Darren
Re: Adding Nested Properties/JSON
Posted by Etienne Jouvin <la...@gmail.com>.
Hello.
Jolt transformation.
Etienne
Le mar. 31 mars 2020 à 14:40, Darren Govoni <da...@ontrenet.com> a écrit :
> Hi,
> I want to use Nifi to design a flow that modifies, updates, etc a
> nested JSON document (or that can finally output one at the end).
>
> For example:
>
> {
> "name":"this and that",
> "field":"value",
> "others": [
> {"name":"here and there"},
> ...
> ]
> }
>
> What's the best approach to this using Nifi?
>
> Thanks in advance!
> Darren
>