You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Janu Thevan <ts...@gmail.com> on 2015/12/04 06:04:41 UTC
Fail convert csv file to avro and avro to json
Hi all,
I am new to software engineering field and i'm a student.
I'm try building a simple data flow which can read csv file from the local
system, post it in the HTTP as a JSON message format.
For that i'm using GetFile, convertCSVTOAvro, ConvertAvroToJSON and
PostHTTP respectively. The problem i'm facing trouble when writing avro
schema.
my csv file contain data as
1234,701,temperature,4.504342,20.44345,2.3,4.50345
i write the avro schema as like
{
"type" :"record",
"name" : "event",
"fields" : [
{"name":"metaData_timestamp","type" : "long"},
{"name":"metaData_sensorId","type" : "int"},
{"name":"metaData_sensorName","type" : "string"},
{"name":"correlationData_longitude","type" : "double"},
{"name":"correlationData_latitude","type" : "double"},
{"name":"humidity","type" : "float"},
{"name":"sensorValue","type" : "double"}
]
}
and the format of json i need is
{
"event": {
"metaData": {
"timestamp": 4354643,
"sensorId": 701,
"sensorName": "temperature"
},
"correlationData": {
"longitude": 4.504343,
"latitude": 20.44345
},
"payloadData": {
"humidity": 2.3,
"sensorValue": 4.504343
}
}
}
I google to sorted out the problem, but i couldn't.
Or suggest me how to create json message in Nifi.
Thank you in advance.
Best regards,
Janu Thevan.
Re: Fail convert csv file to avro and avro to json
Posted by Bryan Bende <bb...@gmail.com>.
I think the issue is that ConvertCsvToAvro is going to produce a flat Avro
record, which then means ConvertAvroToJson is going to produce a flat JSON
document.
Avro schemas support nesting [1], but I'm not sure how you can go from the
CSV to a schema with nesting with in NiFi.
Another approach might be to use ExtractText and ReplaceText to parse the
CSV and create the JSON (no Avro involved). There is an example template
that shows how this could be done here [2].
Take a look at the "csv" property of ExtractText and then look at the
"Replacement Value" property of ReplaceText.
[1]
http://stackoverflow.com/questions/11764287/how-to-nest-records-in-an-avro-schema
[2]
https://cwiki.apache.org/confluence/download/attachments/57904847/CsvToJSON.xml?version=1&modificationDate=1442927496000&api=v2
On Fri, Dec 4, 2015 at 10:11 AM, Joe Percivall <
joepercivall@yahoo.com.invalid> wrote:
> Hey Janu,
>
> Is there a specific error message or bulletin you're running into when
> trying to convert?
>
> Also there is a new processor in master called AttributesToJSON but I
> believe it only is able to create flat JSON objects. If you're flexible
> with your JSON output that may work out too.
>
> Joe
>
>
> - - - - - -
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joepercivall@yahoo.com
>
>
>
>
> On Friday, December 4, 2015 12:06 AM, Janu Thevan <ts...@gmail.com>
> wrote:
>
>
>
> Hi all,
>
>
> I am new to software engineering field and i'm a student.
>
> I'm try building a simple data flow which can read csv file from the local
> system, post it in the HTTP as a JSON message format.
>
> For that i'm using GetFile, convertCSVTOAvro, ConvertAvroToJSON and
> PostHTTP respectively. The problem i'm facing trouble when writing avro
> schema.
>
> my csv file contain data as
> 1234,701,temperature,4.504342,20.44345,2.3,4.50345
>
> i write the avro schema as like
> {
> "type" :"record",
> "name" : "event",
> "fields" : [
> {"name":"metaData_timestamp","type" : "long"},
> {"name":"metaData_sensorId","type" : "int"},
> {"name":"metaData_sensorName","type" : "string"},
> {"name":"correlationData_longitude","type" : "double"},
> {"name":"correlationData_latitude","type" : "double"},
> {"name":"humidity","type" : "float"},
> {"name":"sensorValue","type" : "double"}
> ]
> }
>
> and the format of json i need is
> {
> "event": {
> "metaData": {
> "timestamp": 4354643,
> "sensorId": 701,
> "sensorName": "temperature"
> },
> "correlationData": {
> "longitude": 4.504343,
> "latitude": 20.44345
> },
> "payloadData": {
> "humidity": 2.3,
> "sensorValue": 4.504343
> }
> }
> }
>
> I google to sorted out the problem, but i couldn't.
>
> Or suggest me how to create json message in Nifi.
>
> Thank you in advance.
>
> Best regards,
> Janu Thevan.
>
Re: Fail convert csv file to avro and avro to json
Posted by Joe Percivall <jo...@yahoo.com.INVALID>.
Hey Janu,
Is there a specific error message or bulletin you're running into when trying to convert?
Also there is a new processor in master called AttributesToJSON but I believe it only is able to create flat JSON objects. If you're flexible with your JSON output that may work out too.
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com
On Friday, December 4, 2015 12:06 AM, Janu Thevan <ts...@gmail.com> wrote:
Hi all,
I am new to software engineering field and i'm a student.
I'm try building a simple data flow which can read csv file from the local
system, post it in the HTTP as a JSON message format.
For that i'm using GetFile, convertCSVTOAvro, ConvertAvroToJSON and
PostHTTP respectively. The problem i'm facing trouble when writing avro
schema.
my csv file contain data as
1234,701,temperature,4.504342,20.44345,2.3,4.50345
i write the avro schema as like
{
"type" :"record",
"name" : "event",
"fields" : [
{"name":"metaData_timestamp","type" : "long"},
{"name":"metaData_sensorId","type" : "int"},
{"name":"metaData_sensorName","type" : "string"},
{"name":"correlationData_longitude","type" : "double"},
{"name":"correlationData_latitude","type" : "double"},
{"name":"humidity","type" : "float"},
{"name":"sensorValue","type" : "double"}
]
}
and the format of json i need is
{
"event": {
"metaData": {
"timestamp": 4354643,
"sensorId": 701,
"sensorName": "temperature"
},
"correlationData": {
"longitude": 4.504343,
"latitude": 20.44345
},
"payloadData": {
"humidity": 2.3,
"sensorValue": 4.504343
}
}
}
I google to sorted out the problem, but i couldn't.
Or suggest me how to create json message in Nifi.
Thank you in advance.
Best regards,
Janu Thevan.