You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Etienne Jouvin <la...@gmail.com> on 2020/03/11 09:35:37 UTC
Apache NiFi 1.9.2 InferAvroSchema on csv file header with :
Hello all.
Just in case someone "can test".
I have NiFi 1.9.2 and need to convert CSV to JSON. I do not planned to
upgrade for now (because of deployment procedure)
In the CSV, I have a column with value like prop:Name
i set true for the property Get CSV Header Definition From Data
The processor failed because of the name.
But if I use a convertRecord with a CSV Reader, that infer schema, and a
JSON writer, this is working fine.
Not the same algorithm to get infer schema from InferAvroSchema and the
reader ?
Regards
Etienne Jouvin
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Garanti
sans virus. www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Re: Apache NiFi 1.9.2 InferAvroSchema on csv file header with :
Posted by Etienne Jouvin <la...@gmail.com>.
Ok thanks.
Finally in my case, this is not an issue.
Because the CSV reader does the job.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Garanti
sans virus. www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Le mer. 11 mars 2020 à 10:48, Edward Armes <ed...@gmail.com> a
écrit :
> Hi Jouvin,
>
> I believe you are correct that the inferAvroSchema and the convert record
> processor do work differently. I believe this is because the
> inferAvroSchema uses Apache Kite and the convert record derives the schema
> from the record reader itself.
>
> As an aside I have also noticed that when you use a validateRecord with a
> different types of reader and writer record handlers (i.e. json in avro
> out). You get different results l, while I'm not surprised by this I think
> it's worth just flagging up, for future reference.
>
> Edward
>
> On Wed, 11 Mar 2020, 09:35 Etienne Jouvin, <la...@gmail.com>
> wrote:
>
>> Hello all.
>>
>> Just in case someone "can test".
>>
>> I have NiFi 1.9.2 and need to convert CSV to JSON. I do not planned to
>> upgrade for now (because of deployment procedure)
>> In the CSV, I have a column with value like prop:Name
>>
>> i set true for the property Get CSV Header Definition From Data
>>
>> The processor failed because of the name.
>>
>> But if I use a convertRecord with a CSV Reader, that infer schema, and a
>> JSON writer, this is working fine.
>>
>> Not the same algorithm to get infer schema from InferAvroSchema and the
>> reader ?
>>
>> Regards
>>
>> Etienne Jouvin
>>
>>
>>
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Garanti
>> sans virus. www.avast.com
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
>> <#m_-8081117188909953864_m_1566800260412186955_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>
>
Re: Apache NiFi 1.9.2 InferAvroSchema on csv file header with :
Posted by Edward Armes <ed...@gmail.com>.
Hi Jouvin,
I believe you are correct that the inferAvroSchema and the convert record
processor do work differently. I believe this is because the
inferAvroSchema uses Apache Kite and the convert record derives the schema
from the record reader itself.
As an aside I have also noticed that when you use a validateRecord with a
different types of reader and writer record handlers (i.e. json in avro
out). You get different results l, while I'm not surprised by this I think
it's worth just flagging up, for future reference.
Edward
On Wed, 11 Mar 2020, 09:35 Etienne Jouvin, <la...@gmail.com> wrote:
> Hello all.
>
> Just in case someone "can test".
>
> I have NiFi 1.9.2 and need to convert CSV to JSON. I do not planned to
> upgrade for now (because of deployment procedure)
> In the CSV, I have a column with value like prop:Name
>
> i set true for the property Get CSV Header Definition From Data
>
> The processor failed because of the name.
>
> But if I use a convertRecord with a CSV Reader, that infer schema, and a
> JSON writer, this is working fine.
>
> Not the same algorithm to get infer schema from InferAvroSchema and the
> reader ?
>
> Regards
>
> Etienne Jouvin
>
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Garanti
> sans virus. www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_1566800260412186955_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>