You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Chris Herssens <ch...@gmail.com> on 2017/10/12 09:04:36 UTC

JoltTransformJSON with array of jsons

Hello,

Is it possible to use the JoltTransformJson processor with an array of
jsons as input

Regards,

Chris

Re: JoltTransformJSON with array of jsons

Posted by Chris Herssens <ch...@gmail.com>.
Hello Mark,

Thanks for the information. The transformation works.

Regards,

Chris

On Thu, Oct 12, 2017 at 1:26 PM, Mark Rachelski <ma...@acommerce.asia> wrote:

> Pierre,
> I never noticed the Advanced UI button... thank you.
>
> Chris,
>
> I am very confident this can be done. I just don't have the time to show
> you how right now.
>
> From memory, the right side of your jolt spec needs to direct the values
> into the same node... then it will automatically build up an array as
> output. The Jolt syntax takes a while to get your head wrapped around.
> Start by reading through the spec found here: https://github.com/
> bazaarvoice/jolt/blob/master/jolt-core/src/main/java/com/
> bazaarvoice/jolt/Shiftr.java.
>
> I was able to ask for help on StackOverflow and this link is an example
> that a very nice person helped with that is manipulating arrays both on the
> input and output side. https://stackoverflow.com/questions/45961341/
> simplifying-google-sheet-json-using-jolt
>
> Mark.
>
> On Thu, Oct 12, 2017 at 5:28 PM Pierre Villard <
> pierre.villard.fr@gmail.com> wrote:
>
>> I won't be able to provide more help (I'm always doing really simple
>> things with Jolt), sorry Chris.
>> Maybe others can chime in and give some hints.
>>
>> Just as a comment Mark, in the Jolt processor configuration view, you
>> have an "Advanced UI" button available. If you click on it, it'll open a UI
>> allowing you to test your Jolt specification just as you can do on the web
>> page you provided. Maybe you already noticed the Advanced UI and feel it's
>> not providing enough features... but just wanted to say it in this thread
>> in case you missed it.
>>
>> Pierre
>>
>> 2017-10-12 12:08 GMT+02:00 Chris Herssens <ch...@gmail.com>:
>>
>>> I tried this, but I get only one json back. Where each element has a
>>> array of values. I need as result an  array of json. The transformation
>>> should be done on each json.
>>>
>>> Op 12 okt. 2017 12:01 p.m. schreef "Mark Rachelski" <mark@acommerce.asia
>>> >:
>>>
>>> Chris,
>>>>
>>>> This does not work because your Jolt Specification is saying you are
>>>> looking for a top-level element called timestamp_start. Try something like
>>>> this:
>>>>
>>>> [
>>>>   {
>>>>     "operation" : "shift",
>>>>     "spec": {
>>>>       "*": {
>>>>       "timestamp_start" : "timestamp_start",
>>>>       "timestamp_end" : "timestamp_end",
>>>>       "custom_primitives" : {
>>>>           "*": "&" }
>>>>      }
>>>>     }
>>>>   }
>>>> ]
>>>>
>>>> to allow for the array.
>>>>
>>>> Also, I recommend you don't develop your jolt specification inside
>>>> NiFi... NiFi is the runtime processor.
>>>>
>>>> From a development environment just use this site: http://jolt-demo.
>>>> appspot.com/#inception where you can paste in your starting document
>>>> and specification to find a spec that does what you want.   Then after the
>>>> spec works, you can put it into your JoltTransformJson processor.
>>>>
>>>> Mar.k
>>>>
>>>> On Thu, Oct 12, 2017 at 4:52 PM Chris Herssens <
>>>> chris.herssens@gmail.com> wrote:
>>>>
>>>>> Hello Pierre,
>>>>>
>>>>> I want to flatten the json content.  How to I do this ?
>>>>> Now  I use the chain DSL with jolt spec
>>>>>
>>>>> [
>>>>>   {
>>>>>     "operation" : "shift",
>>>>>     "spec": {
>>>>>       "timestamp_start" : "timestamp_start",
>>>>>       "timestamp_end" : "timestamp_end",
>>>>>       "custom_primitives" : {
>>>>>           "*": "&" }
>>>>>     }
>>>>>   }
>>>>> ]
>>>>>
>>>>> this doesn't work with an array of json only with a single json
>>>>>
>>>>> Regards,
>>>>>
>>>>> Chris
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Oct 12, 2017 at 11:26 AM, Pierre Villard <
>>>>> pierre.villard.fr@gmail.com> wrote:
>>>>>
>>>>>> Hi Chris,
>>>>>>
>>>>>> Yes it is.
>>>>>>
>>>>>> For example, if I want to add a field in all JSON of my array - I can
>>>>>> do (with default DSL) :
>>>>>>
>>>>>> {
>>>>>>   "*": {
>>>>>>     "mynewfield":"mynewvalue"
>>>>>>   }
>>>>>> }
>>>>>>
>>>>>> I'm clearly not a Jolt expert, but others can help you if you're
>>>>>> trying to do something more complex.
>>>>>>
>>>>>> Pierre
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2017-10-12 11:04 GMT+02:00 Chris Herssens <ch...@gmail.com>:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Is it possible to use the JoltTransformJson processor with an array
>>>>>>> of jsons as input
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Chris
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>

Re: JoltTransformJSON with array of jsons

Posted by Mark Rachelski <ma...@acommerce.asia>.
Pierre,
I never noticed the Advanced UI button... thank you.

Chris,

I am very confident this can be done. I just don't have the time to show
you how right now.

From memory, the right side of your jolt spec needs to direct the values
into the same node... then it will automatically build up an array as
output. The Jolt syntax takes a while to get your head wrapped around.
Start by reading through the spec found here:
https://github.com/bazaarvoice/jolt/blob/master/jolt-core/src/main/java/com/bazaarvoice/jolt/Shiftr.java
.

I was able to ask for help on StackOverflow and this link is an example
that a very nice person helped with that is manipulating arrays both on the
input and output side.
https://stackoverflow.com/questions/45961341/simplifying-google-sheet-json-using-jolt

Mark.

On Thu, Oct 12, 2017 at 5:28 PM Pierre Villard <pi...@gmail.com>
wrote:

> I won't be able to provide more help (I'm always doing really simple
> things with Jolt), sorry Chris.
> Maybe others can chime in and give some hints.
>
> Just as a comment Mark, in the Jolt processor configuration view, you have
> an "Advanced UI" button available. If you click on it, it'll open a UI
> allowing you to test your Jolt specification just as you can do on the web
> page you provided. Maybe you already noticed the Advanced UI and feel it's
> not providing enough features... but just wanted to say it in this thread
> in case you missed it.
>
> Pierre
>
> 2017-10-12 12:08 GMT+02:00 Chris Herssens <ch...@gmail.com>:
>
>> I tried this, but I get only one json back. Where each element has a
>> array of values. I need as result an  array of json. The transformation
>> should be done on each json.
>>
>> Op 12 okt. 2017 12:01 p.m. schreef "Mark Rachelski" <mark@acommerce.asia
>> >:
>>
>> Chris,
>>>
>>> This does not work because your Jolt Specification is saying you are
>>> looking for a top-level element called timestamp_start. Try something like
>>> this:
>>>
>>> [
>>>   {
>>>     "operation" : "shift",
>>>     "spec": {
>>>       "*": {
>>>       "timestamp_start" : "timestamp_start",
>>>       "timestamp_end" : "timestamp_end",
>>>       "custom_primitives" : {
>>>           "*": "&" }
>>>      }
>>>     }
>>>   }
>>> ]
>>>
>>> to allow for the array.
>>>
>>> Also, I recommend you don't develop your jolt specification inside
>>> NiFi... NiFi is the runtime processor.
>>>
>>> From a development environment just use this site:
>>> http://jolt-demo.appspot.com/#inception where you can paste in your
>>> starting document and specification to find a spec that does what you want.
>>>   Then after the spec works, you can put it into your JoltTransformJson
>>> processor.
>>>
>>> Mar.k
>>>
>>> On Thu, Oct 12, 2017 at 4:52 PM Chris Herssens <ch...@gmail.com>
>>> wrote:
>>>
>>>> Hello Pierre,
>>>>
>>>> I want to flatten the json content.  How to I do this ?
>>>> Now  I use the chain DSL with jolt spec
>>>>
>>>> [
>>>>   {
>>>>     "operation" : "shift",
>>>>     "spec": {
>>>>       "timestamp_start" : "timestamp_start",
>>>>       "timestamp_end" : "timestamp_end",
>>>>       "custom_primitives" : {
>>>>           "*": "&" }
>>>>     }
>>>>   }
>>>> ]
>>>>
>>>> this doesn't work with an array of json only with a single json
>>>>
>>>> Regards,
>>>>
>>>> Chris
>>>>
>>>>
>>>>
>>>> On Thu, Oct 12, 2017 at 11:26 AM, Pierre Villard <
>>>> pierre.villard.fr@gmail.com> wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> Yes it is.
>>>>>
>>>>> For example, if I want to add a field in all JSON of my array - I can
>>>>> do (with default DSL) :
>>>>>
>>>>> {
>>>>>   "*": {
>>>>>     "mynewfield":"mynewvalue"
>>>>>   }
>>>>> }
>>>>>
>>>>> I'm clearly not a Jolt expert, but others can help you if you're
>>>>> trying to do something more complex.
>>>>>
>>>>> Pierre
>>>>>
>>>>>
>>>>>
>>>>> 2017-10-12 11:04 GMT+02:00 Chris Herssens <ch...@gmail.com>:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Is it possible to use the JoltTransformJson processor with an array
>>>>>> of jsons as input
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>
>>>>>
>>>>
>

Re: JoltTransformJSON with array of jsons

Posted by Pierre Villard <pi...@gmail.com>.
I won't be able to provide more help (I'm always doing really simple things
with Jolt), sorry Chris.
Maybe others can chime in and give some hints.

Just as a comment Mark, in the Jolt processor configuration view, you have
an "Advanced UI" button available. If you click on it, it'll open a UI
allowing you to test your Jolt specification just as you can do on the web
page you provided. Maybe you already noticed the Advanced UI and feel it's
not providing enough features... but just wanted to say it in this thread
in case you missed it.

Pierre

2017-10-12 12:08 GMT+02:00 Chris Herssens <ch...@gmail.com>:

> I tried this, but I get only one json back. Where each element has a array
> of values. I need as result an  array of json. The transformation should be
> done on each json.
>
> Op 12 okt. 2017 12:01 p.m. schreef "Mark Rachelski" <ma...@acommerce.asia>:
>
> Chris,
>>
>> This does not work because your Jolt Specification is saying you are
>> looking for a top-level element called timestamp_start. Try something like
>> this:
>>
>> [
>>   {
>>     "operation" : "shift",
>>     "spec": {
>>       "*": {
>>       "timestamp_start" : "timestamp_start",
>>       "timestamp_end" : "timestamp_end",
>>       "custom_primitives" : {
>>           "*": "&" }
>>      }
>>     }
>>   }
>> ]
>>
>> to allow for the array.
>>
>> Also, I recommend you don't develop your jolt specification inside
>> NiFi... NiFi is the runtime processor.
>>
>> From a development environment just use this site:
>> http://jolt-demo.appspot.com/#inception where you can paste in your
>> starting document and specification to find a spec that does what you want.
>>   Then after the spec works, you can put it into your JoltTransformJson
>> processor.
>>
>> Mar.k
>>
>> On Thu, Oct 12, 2017 at 4:52 PM Chris Herssens <ch...@gmail.com>
>> wrote:
>>
>>> Hello Pierre,
>>>
>>> I want to flatten the json content.  How to I do this ?
>>> Now  I use the chain DSL with jolt spec
>>>
>>> [
>>>   {
>>>     "operation" : "shift",
>>>     "spec": {
>>>       "timestamp_start" : "timestamp_start",
>>>       "timestamp_end" : "timestamp_end",
>>>       "custom_primitives" : {
>>>           "*": "&" }
>>>     }
>>>   }
>>> ]
>>>
>>> this doesn't work with an array of json only with a single json
>>>
>>> Regards,
>>>
>>> Chris
>>>
>>>
>>>
>>> On Thu, Oct 12, 2017 at 11:26 AM, Pierre Villard <
>>> pierre.villard.fr@gmail.com> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> Yes it is.
>>>>
>>>> For example, if I want to add a field in all JSON of my array - I can
>>>> do (with default DSL) :
>>>>
>>>> {
>>>>   "*": {
>>>>     "mynewfield":"mynewvalue"
>>>>   }
>>>> }
>>>>
>>>> I'm clearly not a Jolt expert, but others can help you if you're trying
>>>> to do something more complex.
>>>>
>>>> Pierre
>>>>
>>>>
>>>>
>>>> 2017-10-12 11:04 GMT+02:00 Chris Herssens <ch...@gmail.com>:
>>>>
>>>>> Hello,
>>>>>
>>>>> Is it possible to use the JoltTransformJson processor with an array of
>>>>> jsons as input
>>>>>
>>>>> Regards,
>>>>>
>>>>> Chris
>>>>>
>>>>
>>>>
>>>

Re: JoltTransformJSON with array of jsons

Posted by Chris Herssens <ch...@gmail.com>.
I tried this, but I get only one json back. Where each element has a array
of values. I need as result an  array of json. The transformation should be
done on each json.

Op 12 okt. 2017 12:01 p.m. schreef "Mark Rachelski" <ma...@acommerce.asia>:

> Chris,
>
> This does not work because your Jolt Specification is saying you are
> looking for a top-level element called timestamp_start. Try something like
> this:
>
> [
>   {
>     "operation" : "shift",
>     "spec": {
>       "*": {
>       "timestamp_start" : "timestamp_start",
>       "timestamp_end" : "timestamp_end",
>       "custom_primitives" : {
>           "*": "&" }
>      }
>     }
>   }
> ]
>
> to allow for the array.
>
> Also, I recommend you don't develop your jolt specification inside NiFi...
> NiFi is the runtime processor.
>
> From a development environment just use this site: http://jolt-demo.
> appspot.com/#inception where you can paste in your starting document and
> specification to find a spec that does what you want.   Then after the spec
> works, you can put it into your JoltTransformJson processor.
>
> Mar.k
>
> On Thu, Oct 12, 2017 at 4:52 PM Chris Herssens <ch...@gmail.com>
> wrote:
>
>> Hello Pierre,
>>
>> I want to flatten the json content.  How to I do this ?
>> Now  I use the chain DSL with jolt spec
>>
>> [
>>   {
>>     "operation" : "shift",
>>     "spec": {
>>       "timestamp_start" : "timestamp_start",
>>       "timestamp_end" : "timestamp_end",
>>       "custom_primitives" : {
>>           "*": "&" }
>>     }
>>   }
>> ]
>>
>> this doesn't work with an array of json only with a single json
>>
>> Regards,
>>
>> Chris
>>
>>
>>
>> On Thu, Oct 12, 2017 at 11:26 AM, Pierre Villard <
>> pierre.villard.fr@gmail.com> wrote:
>>
>>> Hi Chris,
>>>
>>> Yes it is.
>>>
>>> For example, if I want to add a field in all JSON of my array - I can do
>>> (with default DSL) :
>>>
>>> {
>>>   "*": {
>>>     "mynewfield":"mynewvalue"
>>>   }
>>> }
>>>
>>> I'm clearly not a Jolt expert, but others can help you if you're trying
>>> to do something more complex.
>>>
>>> Pierre
>>>
>>>
>>>
>>> 2017-10-12 11:04 GMT+02:00 Chris Herssens <ch...@gmail.com>:
>>>
>>>> Hello,
>>>>
>>>> Is it possible to use the JoltTransformJson processor with an array of
>>>> jsons as input
>>>>
>>>> Regards,
>>>>
>>>> Chris
>>>>
>>>
>>>
>>

Re: JoltTransformJSON with array of jsons

Posted by Mark Rachelski <ma...@acommerce.asia>.
Chris,

This does not work because your Jolt Specification is saying you are
looking for a top-level element called timestamp_start. Try something like
this:

[
  {
    "operation" : "shift",
    "spec": {
      "*": {
      "timestamp_start" : "timestamp_start",
      "timestamp_end" : "timestamp_end",
      "custom_primitives" : {
          "*": "&" }
     }
    }
  }
]

to allow for the array.

Also, I recommend you don't develop your jolt specification inside NiFi...
NiFi is the runtime processor.

From a development environment just use this site:
http://jolt-demo.appspot.com/#inception where you can paste in your
starting document and specification to find a spec that does what you want.
  Then after the spec works, you can put it into your JoltTransformJson
processor.

Mar.k

On Thu, Oct 12, 2017 at 4:52 PM Chris Herssens <ch...@gmail.com>
wrote:

> Hello Pierre,
>
> I want to flatten the json content.  How to I do this ?
> Now  I use the chain DSL with jolt spec
>
> [
>   {
>     "operation" : "shift",
>     "spec": {
>       "timestamp_start" : "timestamp_start",
>       "timestamp_end" : "timestamp_end",
>       "custom_primitives" : {
>           "*": "&" }
>     }
>   }
> ]
>
> this doesn't work with an array of json only with a single json
>
> Regards,
>
> Chris
>
>
>
> On Thu, Oct 12, 2017 at 11:26 AM, Pierre Villard <
> pierre.villard.fr@gmail.com> wrote:
>
>> Hi Chris,
>>
>> Yes it is.
>>
>> For example, if I want to add a field in all JSON of my array - I can do
>> (with default DSL) :
>>
>> {
>>   "*": {
>>     "mynewfield":"mynewvalue"
>>   }
>> }
>>
>> I'm clearly not a Jolt expert, but others can help you if you're trying
>> to do something more complex.
>>
>> Pierre
>>
>>
>>
>> 2017-10-12 11:04 GMT+02:00 Chris Herssens <ch...@gmail.com>:
>>
>>> Hello,
>>>
>>> Is it possible to use the JoltTransformJson processor with an array of
>>> jsons as input
>>>
>>> Regards,
>>>
>>> Chris
>>>
>>
>>
>

Re: JoltTransformJSON with array of jsons

Posted by Chris Herssens <ch...@gmail.com>.
Hello Pierre,

I want to flatten the json content.  How to I do this ?
Now  I use the chain DSL with jolt spec

[
  {
    "operation" : "shift",
    "spec": {
      "timestamp_start" : "timestamp_start",
      "timestamp_end" : "timestamp_end",
      "custom_primitives" : {
          "*": "&" }
    }
  }
]

this doesn't work with an array of json only with a single json

Regards,

Chris



On Thu, Oct 12, 2017 at 11:26 AM, Pierre Villard <
pierre.villard.fr@gmail.com> wrote:

> Hi Chris,
>
> Yes it is.
>
> For example, if I want to add a field in all JSON of my array - I can do
> (with default DSL) :
>
> {
>   "*": {
>     "mynewfield":"mynewvalue"
>   }
> }
>
> I'm clearly not a Jolt expert, but others can help you if you're trying to
> do something more complex.
>
> Pierre
>
>
>
> 2017-10-12 11:04 GMT+02:00 Chris Herssens <ch...@gmail.com>:
>
>> Hello,
>>
>> Is it possible to use the JoltTransformJson processor with an array of
>> jsons as input
>>
>> Regards,
>>
>> Chris
>>
>
>

Re: JoltTransformJSON with array of jsons

Posted by Pierre Villard <pi...@gmail.com>.
Hi Chris,

Yes it is.

For example, if I want to add a field in all JSON of my array - I can do
(with default DSL) :

{
  "*": {
    "mynewfield":"mynewvalue"
  }
}

I'm clearly not a Jolt expert, but others can help you if you're trying to
do something more complex.

Pierre



2017-10-12 11:04 GMT+02:00 Chris Herssens <ch...@gmail.com>:

> Hello,
>
> Is it possible to use the JoltTransformJson processor with an array of
> jsons as input
>
> Regards,
>
> Chris
>