You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Chris Campbell <ck...@live.com> on 2015/11/06 23:58:56 UTC
Question on EvaluateJSONPath
I'm converting CSV files to JSON and I want to extract only desired records
from the files based on the first column.
I used ConvertCSVtoAvro and then ConvertAvroToJSON that that gave me JSON
output that looks good.
[{"field1": "rtapushevq", "field2": "CONFCOAT_SIDE2", "field3": "PVA4",
"field4": "ata", "field5": "NO", "field6": "151030.000027", "field7": "1",
"field8": "28364681B620RR1519501012427601", "field9": "GFE_POWER",
"field10": "GM"},{"field1": "rtapushevq", "field2": "SELSOLDERINSP",
"field3": "SELSOLDERINSP1", "field4": "ata", "field5": "NO", "field6":
"151030.000040", "field7": "1", "field8": "533052945601", "field9":
"CHARGER_LVL", "field10": "BMW"},{"field1": "rtapushevq", "field2":
"LVL_ASSY11", "field3": "LVL_STA11", "field4": "ata", "field5": "NO",
.
However, when I try to extract the value of field1 into an attribute using
EvaluateJSONPath (looking at $.field1) it doesn't appear to give me what I
want. I suspect that is because the JSON is formatted as an array rather
than one JSON object per input record.
Can you suggest a way to process the JSON output record-by-record?
Re: Question on EvaluateJSONPath
Posted by Joe Witt <jo...@gmail.com>.
Chris,
The idea here is that you had a 'bunch of events in one thing' and now
you're telling nifi ok let's break that back into their specific
events, do some stuff, then turn it into what you want.
Thanks
Joe
On Fri, Nov 6, 2015 at 6:25 PM, Chris Campbell <ck...@live.com> wrote:
> Thanks Joe!
>
> That sounds like the right way to approach it. I guess I expected that NiFi would treat each record like an event rather than as an object.
>
> I'll give it a try.
>
> -----Original Message-----
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: Friday, November 6, 2015 6:07 PM
> To: users@nifi.apache.org
> Subject: Re: Question on EvaluateJSONPath
>
> Chris,
>
> Let me frame it differently and lets see if that does what you'd like.
>
> Input is a CSV file.
>
> Transform is to filter out CSV entries which do not match desired criteria.
>
> Output is a JSON file.
>
> If that is correct how about this flow:
>
> SplitText (1 line at time)
> RouteText (to find matching lines and filter out the rest) MergeContent to smash the entries back together Transform bundles to AVRO then JSON
>
> Would that take care of it?
>
> Thanks
> Joe
>
> On Fri, Nov 6, 2015 at 5:58 PM, Chris Campbell <ck...@live.com> wrote:
>> I’m converting CSV files to JSON and I want to extract only desired
>> records from the files based on the first column.
>>
>>
>>
>> I used ConvertCSVtoAvro and then ConvertAvroToJSON that that gave me
>> JSON output that looks good.
>>
>>
>>
>> [{"field1": "rtapushevq", "field2": "CONFCOAT_SIDE2", "field3":
>> "PVA4",
>> "field4": "ata", "field5": "NO", "field6": "151030.000027", "field7":
>> "1",
>> "field8": "28364681B620RR1519501012427601", "field9": "GFE_POWER",
>> "field10": "GM"},{"field1": "rtapushevq", "field2": "SELSOLDERINSP",
>> "field3": "SELSOLDERINSP1", "field4": "ata", "field5": "NO", "field6":
>> "151030.000040", "field7": "1", "field8": "533052945601", "field9":
>> "CHARGER_LVL", "field10": "BMW"},{"field1": "rtapushevq", "field2":
>> "LVL_ASSY11", "field3": "LVL_STA11", "field4": "ata", "field5": "NO",
>>
>> …
>>
>>
>>
>> However, when I try to extract the value of field1 into an attribute
>> using EvaluateJSONPath (looking at $.field1) it doesn’t appear to give
>> me what I want. I suspect that is because the JSON is formatted as an
>> array rather than one JSON object per input record.
>>
>>
>>
>> Can you suggest a way to process the JSON output record-by-record?
RE: Question on EvaluateJSONPath
Posted by Chris Campbell <ck...@live.com>.
Thanks Joe!
That sounds like the right way to approach it. I guess I expected that NiFi would treat each record like an event rather than as an object.
I'll give it a try.
-----Original Message-----
From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: Friday, November 6, 2015 6:07 PM
To: users@nifi.apache.org
Subject: Re: Question on EvaluateJSONPath
Chris,
Let me frame it differently and lets see if that does what you'd like.
Input is a CSV file.
Transform is to filter out CSV entries which do not match desired criteria.
Output is a JSON file.
If that is correct how about this flow:
SplitText (1 line at time)
RouteText (to find matching lines and filter out the rest) MergeContent to smash the entries back together Transform bundles to AVRO then JSON
Would that take care of it?
Thanks
Joe
On Fri, Nov 6, 2015 at 5:58 PM, Chris Campbell <ck...@live.com> wrote:
> I’m converting CSV files to JSON and I want to extract only desired
> records from the files based on the first column.
>
>
>
> I used ConvertCSVtoAvro and then ConvertAvroToJSON that that gave me
> JSON output that looks good.
>
>
>
> [{"field1": "rtapushevq", "field2": "CONFCOAT_SIDE2", "field3":
> "PVA4",
> "field4": "ata", "field5": "NO", "field6": "151030.000027", "field7":
> "1",
> "field8": "28364681B620RR1519501012427601", "field9": "GFE_POWER",
> "field10": "GM"},{"field1": "rtapushevq", "field2": "SELSOLDERINSP",
> "field3": "SELSOLDERINSP1", "field4": "ata", "field5": "NO", "field6":
> "151030.000040", "field7": "1", "field8": "533052945601", "field9":
> "CHARGER_LVL", "field10": "BMW"},{"field1": "rtapushevq", "field2":
> "LVL_ASSY11", "field3": "LVL_STA11", "field4": "ata", "field5": "NO",
>
> …
>
>
>
> However, when I try to extract the value of field1 into an attribute
> using EvaluateJSONPath (looking at $.field1) it doesn’t appear to give
> me what I want. I suspect that is because the JSON is formatted as an
> array rather than one JSON object per input record.
>
>
>
> Can you suggest a way to process the JSON output record-by-record?
Re: Question on EvaluateJSONPath
Posted by Joe Witt <jo...@gmail.com>.
Chris,
Let me frame it differently and lets see if that does what you'd like.
Input is a CSV file.
Transform is to filter out CSV entries which do not match desired criteria.
Output is a JSON file.
If that is correct how about this flow:
SplitText (1 line at time)
RouteText (to find matching lines and filter out the rest)
MergeContent to smash the entries back together
Transform bundles to AVRO then JSON
Would that take care of it?
Thanks
Joe
On Fri, Nov 6, 2015 at 5:58 PM, Chris Campbell <ck...@live.com> wrote:
> I’m converting CSV files to JSON and I want to extract only desired records
> from the files based on the first column.
>
>
>
> I used ConvertCSVtoAvro and then ConvertAvroToJSON that that gave me JSON
> output that looks good.
>
>
>
> [{"field1": "rtapushevq", "field2": "CONFCOAT_SIDE2", "field3": "PVA4",
> "field4": "ata", "field5": "NO", "field6": "151030.000027", "field7": "1",
> "field8": "28364681B620RR1519501012427601", "field9": "GFE_POWER",
> "field10": "GM"},{"field1": "rtapushevq", "field2": "SELSOLDERINSP",
> "field3": "SELSOLDERINSP1", "field4": "ata", "field5": "NO", "field6":
> "151030.000040", "field7": "1", "field8": "533052945601", "field9":
> "CHARGER_LVL", "field10": "BMW"},{"field1": "rtapushevq", "field2":
> "LVL_ASSY11", "field3": "LVL_STA11", "field4": "ata", "field5": "NO",
>
> …
>
>
>
> However, when I try to extract the value of field1 into an attribute using
> EvaluateJSONPath (looking at $.field1) it doesn’t appear to give me what I
> want. I suspect that is because the JSON is formatted as an array rather
> than one JSON object per input record.
>
>
>
> Can you suggest a way to process the JSON output record-by-record?