You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Chris Herssens <ch...@gmail.com> on 2017/10/11 11:55:57 UTC
convert avro schema to another schema
Hello All,
I would like to convert an avro schema to another avro schema. Since Nifi
reads the avro data from kafka, I can't use the ConvertAvroSchema
processor. Which processor can I use ?
Is it possible to use ConsumeKafkaRecord processor for that ? If yes how
do we specify the dynamic properties ? If possible can you give me an
example ?
Regards,
Chris
Re: convert avro schema to another schema
Posted by Chris Herssens <ch...@gmail.com>.
Hello Matt,
I tried to use the UpdateRecord processor, but it seems that it doesn't
work with map types.
Suppose the name of the map is "primitives" and has a key "a".
If I define the property /a = /primitives/a ,
the value of "a" is always null
Regards,
Chris
On Wed, Oct 11, 2017 at 6:18 PM, Matt Burgess <ma...@apache.org> wrote:
> If you know the input and output schemas, you should be able to use
> UpdateRecord for this. You would have a user-defined property for each
> output field (including flattened field names), whose value would be
> the RecordPath to the field in the map. I believe for any fields that
> are in the input schema but not the output schema, they are ignored
> (but don't quote me on this, give it a try ;) so you could effectively
> "convert" the record by updating outgoing fields with the values from
> the nested fields in the input record.
>
> Regards,
> Matt
>
> On Wed, Oct 11, 2017 at 12:10 PM, Bryan Bende <bb...@gmail.com> wrote:
> > Casting string to integer wouldn't be a problem, but flattening a map
> would be.
> >
> > I think there was a previous thread where a FlattenRecord processor
> > was discussed, and I believe there was agreement that it would be
> > useful, but it just hasn't been developed yet.
> >
> > If you had FlattenRecord then you would probably bring the data in
> > from ConsumeKafkaRecord keeping the original schema (using a writer
> > with "inherit schema from reader") and then later in the flow do a
> > FlattenRecord.
> >
> > As a current option, you might be able to do the flattening yourself
> > with an ExecuteScript processor, or maybe a scripted record writer
> > that did the flattening on the way out.
> >
> >
> > On Wed, Oct 11, 2017 at 11:22 AM, Chris Herssens
> > <ch...@gmail.com> wrote:
> >> Hello Bryan,
> >>
> >> In my case the schemas are not compatible, In Schema A I have a map
> type and
> >> I want to flatten this map. Also I want to cast string to integer.
> >>
> >> Chris
> >>
> >> On Wed, Oct 11, 2017 at 5:03 PM, Bryan Bende <bb...@gmail.com> wrote:
> >>>
> >>> Chris,
> >>>
> >>> Any processor that uses a record reader and record writer can
> >>> inherently do schema conversion by using schema A for the reader, and
> >>> schema B for the writer, assuming the schemas are compatible.
> >>>
> >>> Compatible in this sense would mean one of the following...
> >>>
> >>> - Schema B has the same field names as schema A, but with some of the
> >>> filed types being different
> >>> - Schema B has a subset of the fields in schema A, possibly some
> >>> changing type as above
> >>> - Schema B has additional fields and they have default values since
> >>> they won't exist in the records coming from schema A
> >>>
> >>> If you have Avro data in Kafka that already has the schema embedded in
> >>> it, then you can use ConsumeKafkaRecord with an AvroReader and set the
> >>> Schema Access Strategy to "Embedded Avro", and then use a
> >>> AvroRecordSetWriter and set the Schema Access Strategy to one of the
> >>> other options like Schema Name (which needs a schema registry) or
> >>> Schema Text which allows you to enter in a full schema.
> >>>
> >>> You could also do the same thing anywhere else in your flow using
> >>> ConvertRecord.
> >>>
> >>> Thanks,
> >>>
> >>> Bryan
> >>>
> >>>
> >>> On Wed, Oct 11, 2017 at 7:55 AM, Chris Herssens
> >>> <ch...@gmail.com> wrote:
> >>> > Hello All,
> >>> >
> >>> > I would like to convert an avro schema to another avro schema.
> Since
> >>> > Nifi
> >>> > reads the avro data from kafka, I can't use the ConvertAvroSchema
> >>> > processor.
> >>> > Which processor can I use ?
> >>> > Is it possible to use ConsumeKafkaRecord processor for that ? If yes
> how
> >>> > do
> >>> > we specify the dynamic properties ? If possible can you give me an
> >>> > example
> >>> > ?
> >>> >
> >>> > Regards,
> >>> >
> >>> > Chris
> >>
> >>
>
Re: convert avro schema to another schema
Posted by Matt Burgess <ma...@apache.org>.
If you know the input and output schemas, you should be able to use
UpdateRecord for this. You would have a user-defined property for each
output field (including flattened field names), whose value would be
the RecordPath to the field in the map. I believe for any fields that
are in the input schema but not the output schema, they are ignored
(but don't quote me on this, give it a try ;) so you could effectively
"convert" the record by updating outgoing fields with the values from
the nested fields in the input record.
Regards,
Matt
On Wed, Oct 11, 2017 at 12:10 PM, Bryan Bende <bb...@gmail.com> wrote:
> Casting string to integer wouldn't be a problem, but flattening a map would be.
>
> I think there was a previous thread where a FlattenRecord processor
> was discussed, and I believe there was agreement that it would be
> useful, but it just hasn't been developed yet.
>
> If you had FlattenRecord then you would probably bring the data in
> from ConsumeKafkaRecord keeping the original schema (using a writer
> with "inherit schema from reader") and then later in the flow do a
> FlattenRecord.
>
> As a current option, you might be able to do the flattening yourself
> with an ExecuteScript processor, or maybe a scripted record writer
> that did the flattening on the way out.
>
>
> On Wed, Oct 11, 2017 at 11:22 AM, Chris Herssens
> <ch...@gmail.com> wrote:
>> Hello Bryan,
>>
>> In my case the schemas are not compatible, In Schema A I have a map type and
>> I want to flatten this map. Also I want to cast string to integer.
>>
>> Chris
>>
>> On Wed, Oct 11, 2017 at 5:03 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>
>>> Chris,
>>>
>>> Any processor that uses a record reader and record writer can
>>> inherently do schema conversion by using schema A for the reader, and
>>> schema B for the writer, assuming the schemas are compatible.
>>>
>>> Compatible in this sense would mean one of the following...
>>>
>>> - Schema B has the same field names as schema A, but with some of the
>>> filed types being different
>>> - Schema B has a subset of the fields in schema A, possibly some
>>> changing type as above
>>> - Schema B has additional fields and they have default values since
>>> they won't exist in the records coming from schema A
>>>
>>> If you have Avro data in Kafka that already has the schema embedded in
>>> it, then you can use ConsumeKafkaRecord with an AvroReader and set the
>>> Schema Access Strategy to "Embedded Avro", and then use a
>>> AvroRecordSetWriter and set the Schema Access Strategy to one of the
>>> other options like Schema Name (which needs a schema registry) or
>>> Schema Text which allows you to enter in a full schema.
>>>
>>> You could also do the same thing anywhere else in your flow using
>>> ConvertRecord.
>>>
>>> Thanks,
>>>
>>> Bryan
>>>
>>>
>>> On Wed, Oct 11, 2017 at 7:55 AM, Chris Herssens
>>> <ch...@gmail.com> wrote:
>>> > Hello All,
>>> >
>>> > I would like to convert an avro schema to another avro schema. Since
>>> > Nifi
>>> > reads the avro data from kafka, I can't use the ConvertAvroSchema
>>> > processor.
>>> > Which processor can I use ?
>>> > Is it possible to use ConsumeKafkaRecord processor for that ? If yes how
>>> > do
>>> > we specify the dynamic properties ? If possible can you give me an
>>> > example
>>> > ?
>>> >
>>> > Regards,
>>> >
>>> > Chris
>>
>>
Re: convert avro schema to another schema
Posted by Bryan Bende <bb...@gmail.com>.
Casting string to integer wouldn't be a problem, but flattening a map would be.
I think there was a previous thread where a FlattenRecord processor
was discussed, and I believe there was agreement that it would be
useful, but it just hasn't been developed yet.
If you had FlattenRecord then you would probably bring the data in
from ConsumeKafkaRecord keeping the original schema (using a writer
with "inherit schema from reader") and then later in the flow do a
FlattenRecord.
As a current option, you might be able to do the flattening yourself
with an ExecuteScript processor, or maybe a scripted record writer
that did the flattening on the way out.
On Wed, Oct 11, 2017 at 11:22 AM, Chris Herssens
<ch...@gmail.com> wrote:
> Hello Bryan,
>
> In my case the schemas are not compatible, In Schema A I have a map type and
> I want to flatten this map. Also I want to cast string to integer.
>
> Chris
>
> On Wed, Oct 11, 2017 at 5:03 PM, Bryan Bende <bb...@gmail.com> wrote:
>>
>> Chris,
>>
>> Any processor that uses a record reader and record writer can
>> inherently do schema conversion by using schema A for the reader, and
>> schema B for the writer, assuming the schemas are compatible.
>>
>> Compatible in this sense would mean one of the following...
>>
>> - Schema B has the same field names as schema A, but with some of the
>> filed types being different
>> - Schema B has a subset of the fields in schema A, possibly some
>> changing type as above
>> - Schema B has additional fields and they have default values since
>> they won't exist in the records coming from schema A
>>
>> If you have Avro data in Kafka that already has the schema embedded in
>> it, then you can use ConsumeKafkaRecord with an AvroReader and set the
>> Schema Access Strategy to "Embedded Avro", and then use a
>> AvroRecordSetWriter and set the Schema Access Strategy to one of the
>> other options like Schema Name (which needs a schema registry) or
>> Schema Text which allows you to enter in a full schema.
>>
>> You could also do the same thing anywhere else in your flow using
>> ConvertRecord.
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Wed, Oct 11, 2017 at 7:55 AM, Chris Herssens
>> <ch...@gmail.com> wrote:
>> > Hello All,
>> >
>> > I would like to convert an avro schema to another avro schema. Since
>> > Nifi
>> > reads the avro data from kafka, I can't use the ConvertAvroSchema
>> > processor.
>> > Which processor can I use ?
>> > Is it possible to use ConsumeKafkaRecord processor for that ? If yes how
>> > do
>> > we specify the dynamic properties ? If possible can you give me an
>> > example
>> > ?
>> >
>> > Regards,
>> >
>> > Chris
>
>
Re: convert avro schema to another schema
Posted by Chris Herssens <ch...@gmail.com>.
Hello Bryan,
In my case the schemas are not compatible, In Schema A I have a map type
and I want to flatten this map. Also I want to cast string to integer.
Chris
On Wed, Oct 11, 2017 at 5:03 PM, Bryan Bende <bb...@gmail.com> wrote:
> Chris,
>
> Any processor that uses a record reader and record writer can
> inherently do schema conversion by using schema A for the reader, and
> schema B for the writer, assuming the schemas are compatible.
>
> Compatible in this sense would mean one of the following...
>
> - Schema B has the same field names as schema A, but with some of the
> filed types being different
> - Schema B has a subset of the fields in schema A, possibly some
> changing type as above
> - Schema B has additional fields and they have default values since
> they won't exist in the records coming from schema A
>
> If you have Avro data in Kafka that already has the schema embedded in
> it, then you can use ConsumeKafkaRecord with an AvroReader and set the
> Schema Access Strategy to "Embedded Avro", and then use a
> AvroRecordSetWriter and set the Schema Access Strategy to one of the
> other options like Schema Name (which needs a schema registry) or
> Schema Text which allows you to enter in a full schema.
>
> You could also do the same thing anywhere else in your flow using
> ConvertRecord.
>
> Thanks,
>
> Bryan
>
>
> On Wed, Oct 11, 2017 at 7:55 AM, Chris Herssens
> <ch...@gmail.com> wrote:
> > Hello All,
> >
> > I would like to convert an avro schema to another avro schema. Since
> Nifi
> > reads the avro data from kafka, I can't use the ConvertAvroSchema
> processor.
> > Which processor can I use ?
> > Is it possible to use ConsumeKafkaRecord processor for that ? If yes how
> do
> > we specify the dynamic properties ? If possible can you give me an
> example
> > ?
> >
> > Regards,
> >
> > Chris
>
Re: convert avro schema to another schema
Posted by Bryan Bende <bb...@gmail.com>.
Chris,
Any processor that uses a record reader and record writer can
inherently do schema conversion by using schema A for the reader, and
schema B for the writer, assuming the schemas are compatible.
Compatible in this sense would mean one of the following...
- Schema B has the same field names as schema A, but with some of the
filed types being different
- Schema B has a subset of the fields in schema A, possibly some
changing type as above
- Schema B has additional fields and they have default values since
they won't exist in the records coming from schema A
If you have Avro data in Kafka that already has the schema embedded in
it, then you can use ConsumeKafkaRecord with an AvroReader and set the
Schema Access Strategy to "Embedded Avro", and then use a
AvroRecordSetWriter and set the Schema Access Strategy to one of the
other options like Schema Name (which needs a schema registry) or
Schema Text which allows you to enter in a full schema.
You could also do the same thing anywhere else in your flow using ConvertRecord.
Thanks,
Bryan
On Wed, Oct 11, 2017 at 7:55 AM, Chris Herssens
<ch...@gmail.com> wrote:
> Hello All,
>
> I would like to convert an avro schema to another avro schema. Since Nifi
> reads the avro data from kafka, I can't use the ConvertAvroSchema processor.
> Which processor can I use ?
> Is it possible to use ConsumeKafkaRecord processor for that ? If yes how do
> we specify the dynamic properties ? If possible can you give me an example
> ?
>
> Regards,
>
> Chris