You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Steve Champagne <ch...@gmail.com> on 2018/08/27 22:52:44 UTC
Wrapping a JSON string
Hello,
I'm ingesting some JSON data that I'd like to wrap in a json_string field
as a string type. I tried using a JsonPathReader with a dynamic property
'json_string' and a value of $, but I seem to be getting back a string
version of the JSON:
{"partition_date":"2018-01-01T00:00:00.000Z","json_string":"{@timestamp=2018-01-01T00:00:00.000Z,
id=1, name=John}"}
I was wondering if there was a way that I could do this and preserve the
raw JSON format?
Thanks,
Steve
Re: Wrapping a JSON string
Posted by Matt Burgess <ma...@apache.org>.
Steve,
Some good news may come in the form of the JoltTransformRecord processor
which should make the next release [1].
Regards,
Matt
[1] https://issues.apache.org/jira/browse/NIFI-5353
On Tue, Sep 4, 2018 at 3:25 PM Steve Champagne <ch...@gmail.com> wrote:
> I was hoping to use the record processors otherwise I would definitely use
> jolt. I was eventually able to get what I was after using a ScriptedReader.
> I do like 'payload' better than 'json_string' though. :)
>
> Thanks,
> Steve
>
> On Mon, Sep 3, 2018 at 3:09 AM DEHAY Aurelien <au...@faurecia.com>
> wrote:
>
>> Hello.
>>
>>
>>
>> You will have to remove the “ in the wrapped json, and process it like a
>> standard string, so I would use the text processors instead of a json one.
>>
>>
>>
>> We did something different from our side: we use encodeJSON do
>> encapsulate the json payload, encoded (“ => \” mainly) in a payload string:
>>
>> Extract the value with evaluatejsonpath with payload attribute as $
>>
>> Update attribute with payload = ${payload:escapeJson()}
>>
>> And a jolt processor with default operation like:
>>
>> [
>>
>> {
>>
>> "operation": "shift",
>>
>> "spec": {
>>
>> "PLANT": "DE_PLANT"
>>
>> }
>>
>> },
>>
>> {
>>
>> "operation": "default",
>>
>> "spec": {
>>
>> "PAYLOAD": "${payload}"
>>
>> }
>>
>> }
>>
>> ]
>>
>>
>>
>>
>>
>>
>>
>>
>> *Aurélien DEHAY *Big Data Architect
>> +33 616 815 441
>>
>> aurelien.dehay@faurecia.com
>>
>> 2 rue Hennape - 92735 Nanterre Cedex – France
>>
>> [image: Faurecia_inspiring_mobility_logo-RVB_150]
>>
>>
>>
>> *From:* Steve Champagne [mailto:champagst@gmail.com]
>> *Sent:* mardi 28 août 2018 00:53
>> *To:* users@nifi.apache.org
>> *Subject:* Wrapping a JSON string
>>
>>
>>
>> Hello,
>>
>>
>>
>> I'm ingesting some JSON data that I'd like to wrap in a json_string field
>> as a string type. I tried using a JsonPathReader with a dynamic property
>> 'json_string' and a value of $, but I seem to be getting back a string
>> version of the JSON:
>>
>>
>>
>> {"partition_date":"2018-01-01T00:00:00.000Z","json_string":"{@timestamp=2018-01-01T00:00:00.000Z,
>> id=1, name=John}"}
>>
>>
>>
>> I was wondering if there was a way that I could do this and preserve the
>> raw JSON format?
>>
>>
>>
>> Thanks,
>>
>> Steve
>>
>> This electronic transmission (and any attachments thereto) is intended
>> solely for the use of the addressee(s). It may contain confidential or
>> legally privileged information. If you are not the intended recipient of
>> this message, you must delete it immediately and notify the sender. Any
>> unauthorized use or disclosure of this message is strictly prohibited.
>> Faurecia does not guarantee the integrity of this transmission and shall
>> therefore never be liable if the message is altered or falsified nor for
>> any virus, interception or damage to your system.
>>
>
Re: Wrapping a JSON string
Posted by Steve Champagne <ch...@gmail.com>.
I was hoping to use the record processors otherwise I would definitely use
jolt. I was eventually able to get what I was after using a ScriptedReader.
I do like 'payload' better than 'json_string' though. :)
Thanks,
Steve
On Mon, Sep 3, 2018 at 3:09 AM DEHAY Aurelien <au...@faurecia.com>
wrote:
> Hello.
>
>
>
> You will have to remove the “ in the wrapped json, and process it like a
> standard string, so I would use the text processors instead of a json one.
>
>
>
> We did something different from our side: we use encodeJSON do encapsulate
> the json payload, encoded (“ => \” mainly) in a payload string:
>
> Extract the value with evaluatejsonpath with payload attribute as $
>
> Update attribute with payload = ${payload:escapeJson()}
>
> And a jolt processor with default operation like:
>
> [
>
> {
>
> "operation": "shift",
>
> "spec": {
>
> "PLANT": "DE_PLANT"
>
> }
>
> },
>
> {
>
> "operation": "default",
>
> "spec": {
>
> "PAYLOAD": "${payload}"
>
> }
>
> }
>
> ]
>
>
>
>
>
>
>
>
> *Aurélien DEHAY *Big Data Architect
> +33 616 815 441
>
> aurelien.dehay@faurecia.com
>
> 2 rue Hennape - 92735 Nanterre Cedex – France
>
> [image: Faurecia_inspiring_mobility_logo-RVB_150]
>
>
>
> *From:* Steve Champagne [mailto:champagst@gmail.com]
> *Sent:* mardi 28 août 2018 00:53
> *To:* users@nifi.apache.org
> *Subject:* Wrapping a JSON string
>
>
>
> Hello,
>
>
>
> I'm ingesting some JSON data that I'd like to wrap in a json_string field
> as a string type. I tried using a JsonPathReader with a dynamic property
> 'json_string' and a value of $, but I seem to be getting back a string
> version of the JSON:
>
>
>
> {"partition_date":"2018-01-01T00:00:00.000Z","json_string":"{@timestamp=2018-01-01T00:00:00.000Z,
> id=1, name=John}"}
>
>
>
> I was wondering if there was a way that I could do this and preserve the
> raw JSON format?
>
>
>
> Thanks,
>
> Steve
>
> This electronic transmission (and any attachments thereto) is intended
> solely for the use of the addressee(s). It may contain confidential or
> legally privileged information. If you are not the intended recipient of
> this message, you must delete it immediately and notify the sender. Any
> unauthorized use or disclosure of this message is strictly prohibited.
> Faurecia does not guarantee the integrity of this transmission and shall
> therefore never be liable if the message is altered or falsified nor for
> any virus, interception or damage to your system.
>
RE: Wrapping a JSON string
Posted by DEHAY Aurelien <au...@faurecia.com>.
Hello.
You will have to remove the “ in the wrapped json, and process it like a standard string, so I would use the text processors instead of a json one.
We did something different from our side: we use encodeJSON do encapsulate the json payload, encoded (“ => \” mainly) in a payload string:
Extract the value with evaluatejsonpath with payload attribute as $
Update attribute with payload = ${payload:escapeJson()}
And a jolt processor with default operation like:
[
{
"operation": "shift",
"spec": {
"PLANT": "DE_PLANT"
}
},
{
"operation": "default",
"spec": {
"PAYLOAD": "${payload}"
}
}
]
Aurélien DEHAY
Big Data Architect
+33 616 815 441
aurelien.dehay@faurecia.com
2 rue Hennape - 92735 Nanterre Cedex – France
[Faurecia_inspiring_mobility_logo-RVB_150]
From: Steve Champagne [mailto:champagst@gmail.com]
Sent: mardi 28 août 2018 00:53
To: users@nifi.apache.org
Subject: Wrapping a JSON string
Hello,
I'm ingesting some JSON data that I'd like to wrap in a json_string field as a string type. I tried using a JsonPathReader with a dynamic property 'json_string' and a value of $, but I seem to be getting back a string version of the JSON:
{"partition_date":"2018-01-01T00:00:00.000Z","json_string":"{@timestamp=2018-01-01T00:00:00.000Z, id=1, name=John}"}
I was wondering if there was a way that I could do this and preserve the raw JSON format?
Thanks,
Steve
This electronic transmission (and any attachments thereto) is intended solely for the use of the addressee(s). It may contain confidential or legally privileged information. If you are not the intended recipient of this message, you must delete it immediately and notify the sender. Any unauthorized use or disclosure of this message is strictly prohibited. Faurecia does not guarantee the integrity of this transmission and shall therefore never be liable if the message is altered or falsified nor for any virus, interception or damage to your system.