You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Joe Witt <jo...@gmail.com> on 2017/09/05 13:26:03 UTC
Re: Nifi kafka ser/deser problem
Hello
Can you share the details of how you're serializing the data? We'll
need to understand the configuration of the record writer of the
publish kafka process to help discuss about reading it back (be that
in nifi or otherwise).
Since we support multiple modes of encoding the schema information
this is the most likely source of difficulty.
Thanks
Joe
On Tue, Sep 5, 2017 at 8:53 AM, <ph...@orange.com> wrote:
> Hello, My SW environment à Nifi 1.4 compiled from sources , on Ubuntu
> 14.04 and a running Confluent 3.3 Platform (running Confluent Registry
> and kafkaStreams )
>
> I have a process group with a PublishKafkaRecord_0_10 that serialize a
> record ( with 3 fields)
>
> And another process group with a ConsumeKafkaRecord_0_10 that deserialise
> the record . that works nicely with reader/writer and with the Confluent
> Registry storing the Avro MetaData.
>
> I do not have ser/deser problems because I think Nifi does both jobs ……
>
>
>
> But I got problems of deserialization if I put some java Kafka client in
> the game … ( try to consume from Confluent Kafka Streams )
>
> I got some deserialization exception :
>
> org.apache.kafka.common.errors.SerializationException: Error deserializing
> Avro message for id -1
>
>
>
> Any trick to force another serialization ( nifi side) or deserialization (
> confluent side ) ?
>
>
>
> Best
>
> Phil
>
>
>
> _________________________________________________________________________________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu
> ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou
> falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and
> delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been
> modified, changed or falsified.
> Thank you.
RE: Nifi kafka ser/deser problem
Posted by ph...@orange.com.
Hello Andrew
Yes ..I trace the call to confluent registry and I see the get on the
subject ( Confluent registry side)
And I also see the binary
content on a consumer shell console (kafka-console-consumer )
---------------
room
room1147.100,3.246▒NF
My question was much more about the fact to have to write or not a decoder Confluent side …..
phil
From: Andrew Psaltis [mailto:psaltis.andrew@gmail.com]
Sent: mardi 5 septembre 2017 20:09
To: users@nifi.apache.org
Subject: Re: Nifi kafka ser/deser problem
Phil,
A couple of quick questions. With your Java application that is consuming data have you configured the KafkaAvroDecoder for the Value? Have you also tried consuming that data from the command line using the kafka-avro-console-consumer (that ships with the Confluent Schema Registry)?
Thanks,
Andrew
On Tue, Sep 5, 2017 at 10:47 AM, <ph...@orange.com>> wrote:
Hello Bryan,
Schema Write Strategy : Confluent Schema Registry Reference
Schema Access strategy : Use 'Schema Name' Property
Schema registry : ConfluentSchemaRegistry
Schema Name : topic7
Schema Text : ${avro.schema}
Compression Format : NONE
Thanks
Phil
-----Original Message-----
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: mardi 5 septembre 2017 16:39
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Nifi kafka ser/deser problem
Phil,
What is the "Schema Write Strategy" set to on the AvroRecordSetWriter?
-Bryan
On Tue, Sep 5, 2017 at 10:23 AM, <ph...@orange.com>> wrote:
> Hello Joe
>
> This json content
> {"type":"room","id":"room11","attributes":{"position":"47.100,3.246","
> surface":223,"norme":"NF"}}
>
> is the input of the PublishKafkaRecord_0_10 processor. This processor have among others the following properties set :
>
> Record Reader : JsonTreeReader --> pointing to the
> ConfluentSchemaRegistry Record Writer : AvroRecordSetWriter ->
> pointing to the confluentSchemaRegistry
>
>
> I trace the call to confluent registry and I see the get on the
> subject ( Confluent registry side) And I also see the binary
> content on a consumer shell console (kafka-console-consumer )
> ---------------
> room
> room1147.100,3.246▒NF
> -------------------------------
>
> Thanks
> Phil
>
> -----Original Message-----
> From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
> Sent: mardi 5 septembre 2017 15:26
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: Nifi kafka ser/deser problem
>
> Hello
>
> Can you share the details of how you're serializing the data? We'll need to understand the configuration of the record writer of the publish kafka process to help discuss about reading it back (be that in nifi or otherwise).
>
> Since we support multiple modes of encoding the schema information this is the most likely source of difficulty.
>
> Thanks
> Joe
>
> On Tue, Sep 5, 2017 at 8:53 AM, <ph...@orange.com>> wrote:
>> Hello, My SW environment à Nifi 1.4 compiled from sources , on Ubuntu
>> 14.04 and a running Confluent 3.3 Platform (running Confluent
>> Registry and kafkaStreams )
>>
>> I have a process group with a PublishKafkaRecord_0_10 that serialize
>> a record ( with 3 fields)
>>
>> And another process group with a ConsumeKafkaRecord_0_10 that
>> deserialise the record . that works nicely with reader/writer and with the Confluent
>> Registry storing the Avro MetaData.
>>
>> I do not have ser/deser problems because I think Nifi does both jobs
>> ……
>>
>>
>>
>> But I got problems of deserialization if I put some java Kafka client in
>> the game … ( try to consume from Confluent Kafka Streams )
>>
>> I got some deserialization exception :
>>
>> org.apache.kafka.common.errors.SerializationException: Error
>> deserializing Avro message for id -1
>>
>>
>>
>> Any trick to force another serialization ( nifi side) or
>> deserialization ( confluent side ) ?
>>
>>
>>
>> Best
>>
>> Phil
>>
>>
>>
>> _____________________________________________________________________
>> _ ___________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations
>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
>> exploites ou copies sans autorisation. Si vous avez recu ce message
>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi
>> que les pieces jointes. Les messages electroniques etant susceptibles
>> d'alteration, Orange decline toute responsabilite si ce message a ete
>> altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or
>> privileged information that may be protected by law; they should not
>> be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender
>> and delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have
>> been modified, changed or falsified.
>> Thank you.
>
> ______________________________________________________________________
> ___________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
> exploites ou copies sans autorisation. Si vous avez recu ce message
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or
> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
--
Thanks,
Andrew
Subscribe to my book: Streaming Data<http://manning.com/psaltis>
[https://static.licdn.com/scds/common/u/img/webpromo/btn_viewmy_160x25.png]<https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
twiiter: @itmdata<http://twitter.com/intent/user?screen_name=itmdata>
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
Re: Nifi kafka ser/deser problem
Posted by Andrew Psaltis <ps...@gmail.com>.
Phil,
A couple of quick questions. With your Java application that is consuming
data have you configured the KafkaAvroDecoder for the Value? Have you also
tried consuming that data from the command line using
the kafka-avro-console-consumer (that ships with the Confluent Schema
Registry)?
Thanks,
Andrew
On Tue, Sep 5, 2017 at 10:47 AM, <ph...@orange.com> wrote:
> Hello Bryan,
>
> Schema Write Strategy : Confluent Schema Registry Reference
> Schema Access strategy : Use 'Schema Name' Property
> Schema registry : ConfluentSchemaRegistry
> Schema Name : topic7
> Schema Text : ${avro.schema}
> Compression Format : NONE
>
> Thanks
> Phil
>
>
> -----Original Message-----
> From: Bryan Bende [mailto:bbende@gmail.com]
> Sent: mardi 5 septembre 2017 16:39
> To: users@nifi.apache.org
> Subject: Re: Nifi kafka ser/deser problem
>
> Phil,
>
> What is the "Schema Write Strategy" set to on the AvroRecordSetWriter?
>
> -Bryan
>
> On Tue, Sep 5, 2017 at 10:23 AM, <ph...@orange.com> wrote:
> > Hello Joe
> >
> > This json content
> > {"type":"room","id":"room11","attributes":{"position":"47.100,3.246","
> > surface":223,"norme":"NF"}}
> >
> > is the input of the PublishKafkaRecord_0_10 processor. This processor
> have among others the following properties set :
> >
> > Record Reader : JsonTreeReader --> pointing to the
> > ConfluentSchemaRegistry Record Writer : AvroRecordSetWriter ->
> > pointing to the confluentSchemaRegistry
> >
> >
> > I trace the call to confluent registry and I see the get on the
> > subject ( Confluent registry side) And I also see the binary
> > content on a consumer shell console (kafka-console-consumer )
> > ---------------
> > room
> > room1147.100,3.246▒NF
> > -------------------------------
> >
> > Thanks
> > Phil
> >
> > -----Original Message-----
> > From: Joe Witt [mailto:joe.witt@gmail.com]
> > Sent: mardi 5 septembre 2017 15:26
> > To: users@nifi.apache.org
> > Subject: Re: Nifi kafka ser/deser problem
> >
> > Hello
> >
> > Can you share the details of how you're serializing the data? We'll
> need to understand the configuration of the record writer of the publish
> kafka process to help discuss about reading it back (be that in nifi or
> otherwise).
> >
> > Since we support multiple modes of encoding the schema information this
> is the most likely source of difficulty.
> >
> > Thanks
> > Joe
> >
> > On Tue, Sep 5, 2017 at 8:53 AM, <ph...@orange.com> wrote:
> >> Hello, My SW environment à Nifi 1.4 compiled from sources , on
> Ubuntu
> >> 14.04 and a running Confluent 3.3 Platform (running Confluent
> >> Registry and kafkaStreams )
> >>
> >> I have a process group with a PublishKafkaRecord_0_10 that serialize
> >> a record ( with 3 fields)
> >>
> >> And another process group with a ConsumeKafkaRecord_0_10 that
> >> deserialise the record . that works nicely with reader/writer and with
> the Confluent
> >> Registry storing the Avro MetaData.
> >>
> >> I do not have ser/deser problems because I think Nifi does both jobs
> >> ……
> >>
> >>
> >>
> >> But I got problems of deserialization if I put some java Kafka client
> in
> >> the game … ( try to consume from Confluent Kafka Streams )
> >>
> >> I got some deserialization exception :
> >>
> >> org.apache.kafka.common.errors.SerializationException: Error
> >> deserializing Avro message for id -1
> >>
> >>
> >>
> >> Any trick to force another serialization ( nifi side) or
> >> deserialization ( confluent side ) ?
> >>
> >>
> >>
> >> Best
> >>
> >> Phil
> >>
> >>
> >>
> >> _____________________________________________________________________
> >> _ ___________________________________________________
> >>
> >> Ce message et ses pieces jointes peuvent contenir des informations
> >> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
> >> exploites ou copies sans autorisation. Si vous avez recu ce message
> >> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi
> >> que les pieces jointes. Les messages electroniques etant susceptibles
> >> d'alteration, Orange decline toute responsabilite si ce message a ete
> >> altere, deforme ou falsifie. Merci.
> >>
> >> This message and its attachments may contain confidential or
> >> privileged information that may be protected by law; they should not
> >> be distributed, used or copied without authorisation.
> >> If you have received this email in error, please notify the sender
> >> and delete this message and its attachments.
> >> As emails may be altered, Orange is not liable for messages that have
> >> been modified, changed or falsified.
> >> Thank you.
> >
> > ______________________________________________________________________
> > ___________________________________________________
> >
> > Ce message et ses pieces jointes peuvent contenir des informations
> > confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
> > exploites ou copies sans autorisation. Si vous avez recu ce message
> > par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que
> les pieces jointes. Les messages electroniques etant susceptibles
> d'alteration, Orange decline toute responsabilite si ce message a ete
> altere, deforme ou falsifie. Merci.
> >
> > This message and its attachments may contain confidential or
> > privileged information that may be protected by law; they should not be
> distributed, used or copied without authorisation.
> > If you have received this email in error, please notify the sender and
> delete this message and its attachments.
> > As emails may be altered, Orange is not liable for messages that have
> been modified, changed or falsified.
> > Thank you.
> >
>
> ____________________________________________________________
> _____________________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
> recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou
> falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and
> delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been
> modified, changed or falsified.
> Thank you.
>
>
--
Thanks,
Andrew
Subscribe to my book: Streaming Data <http://manning.com/psaltis>
<https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
Re: Nifi kafka ser/deser problem
Posted by Mark Payne <ma...@hotmail.com>.
Excellent! Glad you were able to get things moving!
On Sep 6, 2017, at 2:37 AM, <ph...@orange.com>> <ph...@orange.com>> wrote:
Hi Mark, It works like a charm!
Thank you very very much …. ☺
I understand better now the ser/deser and registries …..( confluent ,HWX or built in)
I was thinking wrongly that it was not necessary to write my own ser/deser code ( java Stream Application- Confluent Side)
Phil
Tanks again
From: Mark Payne [mailto:markap14@hotmail.com]
Sent: mardi 5 septembre 2017 22:22
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Nifi kafka ser/deser problem
Hi Phil,
I have just uploaded a template to [1] that will illustrate integrating NiFi
with Kafka using the Confluent Schema Registry.
The template will be at the bottom of the linked page. The instructions
there should illustrate how to publish a compatible schema to the Confluent
Schema Registry and then use NiFi to write data to the Kafka topic in a
compatible way. The template generates some CSV data and then writes it to
Kafka in Avro based on the Confluent Schema Registry.
Please check out the template and see if that helps you get going!
Thanks
-Mark
[1] https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
Example Dataflow Templates - Apache NiFi - Dashboard<https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates>
cwiki.apache.org<http://cwiki.apache.org>
Apache NiFi provides users the ability to build very large and complex DataFlows using NiFi. This is achieved by using the basic components: Processor, Funnel, Input ...
________________________________
From: philippe.gibert@orange.com<ma...@orange.com> <ph...@orange.com>>
Sent: Tuesday, September 5, 2017 9:47 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: RE: Nifi kafka ser/deser problem
Hello Bryan,
Schema Write Strategy : Confluent Schema Registry Reference
Schema Access strategy : Use 'Schema Name' Property
Schema registry : ConfluentSchemaRegistry
Schema Name : topic7
Schema Text : ${avro.schema}
Compression Format : NONE
Thanks
Phil
-----Original Message-----
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: mardi 5 septembre 2017 16:39
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Nifi kafka ser/deser problem
Phil,
What is the "Schema Write Strategy" set to on the AvroRecordSetWriter?
-Bryan
On Tue, Sep 5, 2017 at 10:23 AM, <ph...@orange.com>> wrote:
> Hello Joe
>
> This json content
> {"type":"room","id":"room11","attributes":{"position":"47.100,3.246","
> surface":223,"norme":"NF"}}
>
> is the input of the PublishKafkaRecord_0_10 processor. This processor have among others the following properties set :
>
> Record Reader : JsonTreeReader --> pointing to the
> ConfluentSchemaRegistry Record Writer : AvroRecordSetWriter ->
> pointing to the confluentSchemaRegistry
>
>
> I trace the call to confluent registry and I see the get on the
> subject ( Confluent registry side) And I also see the binary
> content on a consumer shell console (kafka-console-consumer )
> ---------------
> room
> room1147.100,3.246▒NF
> -------------------------------
>
> Thanks
> Phil
>
> -----Original Message-----
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: mardi 5 septembre 2017 15:26
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: Nifi kafka ser/deser problem
>
> Hello
>
> Can you share the details of how you're serializing the data? We'll need to understand the configuration of the record writer of the publish kafka process to help discuss about reading it back (be that in nifi or otherwise).
>
> Since we support multiple modes of encoding the schema information this is the most likely source of difficulty.
>
> Thanks
> Joe
>
> On Tue, Sep 5, 2017 at 8:53 AM, <ph...@orange.com>> wrote:
>> Hello, My SW environment à Nifi 1.4 compiled from sources , on Ubuntu
>> 14.04 and a running Confluent 3.3 Platform (running Confluent
>> Registry and kafkaStreams )
>>
>> I have a process group with a PublishKafkaRecord_0_10 that serialize
>> a record ( with 3 fields)
>>
>> And another process group with a ConsumeKafkaRecord_0_10 that
>> deserialise the record . that works nicely with reader/writer and with the Confluent
>> Registry storing the Avro MetaData.
>>
>> I do not have ser/deser problems because I think Nifi does both jobs
>> ……
>>
>>
>>
>> But I got problems of deserialization if I put some java Kafka client in
>> the game … ( try to consume from Confluent Kafka Streams )
>>
>> I got some deserialization exception :
>>
>> org.apache.kafka.common.errors.SerializationException: Error
>> deserializing Avro message for id -1
>>
>>
>>
>> Any trick to force another serialization ( nifi side) or
>> deserialization ( confluent side ) ?
>>
>>
>>
>> Best
>>
>> Phil
>>
>>
>>
>> _____________________________________________________________________
>> _ ___________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations
>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
>> exploites ou copies sans autorisation. Si vous avez recu ce message
>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi
>> que les pieces jointes. Les messages electroniques etant susceptibles
>> d'alteration, Orange decline toute responsabilite si ce message a ete
>> altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or
>> privileged information that may be protected by law; they should not
>> be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender
>> and delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have
>> been modified, changed or falsified.
>> Thank you.
>
> ______________________________________________________________________
> ___________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
> exploites ou copies sans autorisation. Si vous avez recu ce message
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or
> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
RE: Nifi kafka ser/deser problem
Posted by ph...@orange.com.
Hi Mark, It works like a charm!
Thank you very very much …. ☺
I understand better now the ser/deser and registries …..( confluent ,HWX or built in)
I was thinking wrongly that it was not necessary to write my own ser/deser code ( java Stream Application- Confluent Side)
Phil
Tanks again
From: Mark Payne [mailto:markap14@hotmail.com]
Sent: mardi 5 septembre 2017 22:22
To: users@nifi.apache.org
Subject: Re: Nifi kafka ser/deser problem
Hi Phil,
I have just uploaded a template to [1] that will illustrate integrating NiFi
with Kafka using the Confluent Schema Registry.
The template will be at the bottom of the linked page. The instructions
there should illustrate how to publish a compatible schema to the Confluent
Schema Registry and then use NiFi to write data to the Kafka topic in a
compatible way. The template generates some CSV data and then writes it to
Kafka in Avro based on the Confluent Schema Registry.
Please check out the template and see if that helps you get going!
Thanks
-Mark
[1] https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
Example Dataflow Templates - Apache NiFi - Dashboard<https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates>
cwiki.apache.org
Apache NiFi provides users the ability to build very large and complex DataFlows using NiFi. This is achieved by using the basic components: Processor, Funnel, Input ...
________________________________
From: philippe.gibert@orange.com<ma...@orange.com> <ph...@orange.com>>
Sent: Tuesday, September 5, 2017 9:47 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: RE: Nifi kafka ser/deser problem
Hello Bryan,
Schema Write Strategy : Confluent Schema Registry Reference
Schema Access strategy : Use 'Schema Name' Property
Schema registry : ConfluentSchemaRegistry
Schema Name : topic7
Schema Text : ${avro.schema}
Compression Format : NONE
Thanks
Phil
-----Original Message-----
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: mardi 5 septembre 2017 16:39
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Nifi kafka ser/deser problem
Phil,
What is the "Schema Write Strategy" set to on the AvroRecordSetWriter?
-Bryan
On Tue, Sep 5, 2017 at 10:23 AM, <ph...@orange.com>> wrote:
> Hello Joe
>
> This json content
> {"type":"room","id":"room11","attributes":{"position":"47.100,3.246","
> surface":223,"norme":"NF"}}
>
> is the input of the PublishKafkaRecord_0_10 processor. This processor have among others the following properties set :
>
> Record Reader : JsonTreeReader --> pointing to the
> ConfluentSchemaRegistry Record Writer : AvroRecordSetWriter ->
> pointing to the confluentSchemaRegistry
>
>
> I trace the call to confluent registry and I see the get on the
> subject ( Confluent registry side) And I also see the binary
> content on a consumer shell console (kafka-console-consumer )
> ---------------
> room
> room1147.100,3.246▒NF
> -------------------------------
>
> Thanks
> Phil
>
> -----Original Message-----
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: mardi 5 septembre 2017 15:26
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: Nifi kafka ser/deser problem
>
> Hello
>
> Can you share the details of how you're serializing the data? We'll need to understand the configuration of the record writer of the publish kafka process to help discuss about reading it back (be that in nifi or otherwise).
>
> Since we support multiple modes of encoding the schema information this is the most likely source of difficulty.
>
> Thanks
> Joe
>
> On Tue, Sep 5, 2017 at 8:53 AM, <ph...@orange.com>> wrote:
>> Hello, My SW environment à Nifi 1.4 compiled from sources , on Ubuntu
>> 14.04 and a running Confluent 3.3 Platform (running Confluent
>> Registry and kafkaStreams )
>>
>> I have a process group with a PublishKafkaRecord_0_10 that serialize
>> a record ( with 3 fields)
>>
>> And another process group with a ConsumeKafkaRecord_0_10 that
>> deserialise the record . that works nicely with reader/writer and with the Confluent
>> Registry storing the Avro MetaData.
>>
>> I do not have ser/deser problems because I think Nifi does both jobs
>> ……
>>
>>
>>
>> But I got problems of deserialization if I put some java Kafka client in
>> the game … ( try to consume from Confluent Kafka Streams )
>>
>> I got some deserialization exception :
>>
>> org.apache.kafka.common.errors.SerializationException: Error
>> deserializing Avro message for id -1
>>
>>
>>
>> Any trick to force another serialization ( nifi side) or
>> deserialization ( confluent side ) ?
>>
>>
>>
>> Best
>>
>> Phil
>>
>>
>>
>> _____________________________________________________________________
>> _ ___________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations
>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
>> exploites ou copies sans autorisation. Si vous avez recu ce message
>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi
>> que les pieces jointes. Les messages electroniques etant susceptibles
>> d'alteration, Orange decline toute responsabilite si ce message a ete
>> altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or
>> privileged information that may be protected by law; they should not
>> be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender
>> and delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have
>> been modified, changed or falsified.
>> Thank you.
>
> ______________________________________________________________________
> ___________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
> exploites ou copies sans autorisation. Si vous avez recu ce message
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or
> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
Re: Nifi kafka ser/deser problem
Posted by Mark Payne <ma...@hotmail.com>.
Hi Phil,
I have just uploaded a template to [1] that will illustrate integrating NiFi
with Kafka using the Confluent Schema Registry.
The template will be at the bottom of the linked page. The instructions
there should illustrate how to publish a compatible schema to the Confluent
Schema Registry and then use NiFi to write data to the Kafka topic in a
compatible way. The template generates some CSV data and then writes it to
Kafka in Avro based on the Confluent Schema Registry.
Please check out the template and see if that helps you get going!
Thanks
-Mark
[1] https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
Example Dataflow Templates - Apache NiFi - Dashboard<https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates>
cwiki.apache.org
Apache NiFi provides users the ability to build very large and complex DataFlows using NiFi. This is achieved by using the basic components: Processor, Funnel, Input ...
________________________________
From: philippe.gibert@orange.com <ph...@orange.com>
Sent: Tuesday, September 5, 2017 9:47 AM
To: users@nifi.apache.org
Subject: RE: Nifi kafka ser/deser problem
Hello Bryan,
Schema Write Strategy : Confluent Schema Registry Reference
Schema Access strategy : Use 'Schema Name' Property
Schema registry : ConfluentSchemaRegistry
Schema Name : topic7
Schema Text : ${avro.schema}
Compression Format : NONE
Thanks
Phil
-----Original Message-----
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: mardi 5 septembre 2017 16:39
To: users@nifi.apache.org
Subject: Re: Nifi kafka ser/deser problem
Phil,
What is the "Schema Write Strategy" set to on the AvroRecordSetWriter?
-Bryan
On Tue, Sep 5, 2017 at 10:23 AM, <ph...@orange.com> wrote:
> Hello Joe
>
> This json content
> {"type":"room","id":"room11","attributes":{"position":"47.100,3.246","
> surface":223,"norme":"NF"}}
>
> is the input of the PublishKafkaRecord_0_10 processor. This processor have among others the following properties set :
>
> Record Reader : JsonTreeReader --> pointing to the
> ConfluentSchemaRegistry Record Writer : AvroRecordSetWriter ->
> pointing to the confluentSchemaRegistry
>
>
> I trace the call to confluent registry and I see the get on the
> subject ( Confluent registry side) And I also see the binary
> content on a consumer shell console (kafka-console-consumer )
> ---------------
> room
> room1147.100,3.246▒NF
> -------------------------------
>
> Thanks
> Phil
>
> -----Original Message-----
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: mardi 5 septembre 2017 15:26
> To: users@nifi.apache.org
> Subject: Re: Nifi kafka ser/deser problem
>
> Hello
>
> Can you share the details of how you're serializing the data? We'll need to understand the configuration of the record writer of the publish kafka process to help discuss about reading it back (be that in nifi or otherwise).
>
> Since we support multiple modes of encoding the schema information this is the most likely source of difficulty.
>
> Thanks
> Joe
>
> On Tue, Sep 5, 2017 at 8:53 AM, <ph...@orange.com> wrote:
>> Hello, My SW environment à Nifi 1.4 compiled from sources , on Ubuntu
>> 14.04 and a running Confluent 3.3 Platform (running Confluent
>> Registry and kafkaStreams )
>>
>> I have a process group with a PublishKafkaRecord_0_10 that serialize
>> a record ( with 3 fields)
>>
>> And another process group with a ConsumeKafkaRecord_0_10 that
>> deserialise the record . that works nicely with reader/writer and with the Confluent
>> Registry storing the Avro MetaData.
>>
>> I do not have ser/deser problems because I think Nifi does both jobs
>> ……
>>
>>
>>
>> But I got problems of deserialization if I put some java Kafka client in
>> the game … ( try to consume from Confluent Kafka Streams )
>>
>> I got some deserialization exception :
>>
>> org.apache.kafka.common.errors.SerializationException: Error
>> deserializing Avro message for id -1
>>
>>
>>
>> Any trick to force another serialization ( nifi side) or
>> deserialization ( confluent side ) ?
>>
>>
>>
>> Best
>>
>> Phil
>>
>>
>>
>> _____________________________________________________________________
>> _ ___________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations
>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
>> exploites ou copies sans autorisation. Si vous avez recu ce message
>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi
>> que les pieces jointes. Les messages electroniques etant susceptibles
>> d'alteration, Orange decline toute responsabilite si ce message a ete
>> altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or
>> privileged information that may be protected by law; they should not
>> be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender
>> and delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have
>> been modified, changed or falsified.
>> Thank you.
>
> ______________________________________________________________________
> ___________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
> exploites ou copies sans autorisation. Si vous avez recu ce message
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or
> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
RE: Nifi kafka ser/deser problem
Posted by ph...@orange.com.
Hello Bryan,
Schema Write Strategy : Confluent Schema Registry Reference
Schema Access strategy : Use 'Schema Name' Property
Schema registry : ConfluentSchemaRegistry
Schema Name : topic7
Schema Text : ${avro.schema}
Compression Format : NONE
Thanks
Phil
-----Original Message-----
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: mardi 5 septembre 2017 16:39
To: users@nifi.apache.org
Subject: Re: Nifi kafka ser/deser problem
Phil,
What is the "Schema Write Strategy" set to on the AvroRecordSetWriter?
-Bryan
On Tue, Sep 5, 2017 at 10:23 AM, <ph...@orange.com> wrote:
> Hello Joe
>
> This json content
> {"type":"room","id":"room11","attributes":{"position":"47.100,3.246","
> surface":223,"norme":"NF"}}
>
> is the input of the PublishKafkaRecord_0_10 processor. This processor have among others the following properties set :
>
> Record Reader : JsonTreeReader --> pointing to the
> ConfluentSchemaRegistry Record Writer : AvroRecordSetWriter ->
> pointing to the confluentSchemaRegistry
>
>
> I trace the call to confluent registry and I see the get on the
> subject ( Confluent registry side) And I also see the binary
> content on a consumer shell console (kafka-console-consumer )
> ---------------
> room
> room1147.100,3.246▒NF
> -------------------------------
>
> Thanks
> Phil
>
> -----Original Message-----
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: mardi 5 septembre 2017 15:26
> To: users@nifi.apache.org
> Subject: Re: Nifi kafka ser/deser problem
>
> Hello
>
> Can you share the details of how you're serializing the data? We'll need to understand the configuration of the record writer of the publish kafka process to help discuss about reading it back (be that in nifi or otherwise).
>
> Since we support multiple modes of encoding the schema information this is the most likely source of difficulty.
>
> Thanks
> Joe
>
> On Tue, Sep 5, 2017 at 8:53 AM, <ph...@orange.com> wrote:
>> Hello, My SW environment à Nifi 1.4 compiled from sources , on Ubuntu
>> 14.04 and a running Confluent 3.3 Platform (running Confluent
>> Registry and kafkaStreams )
>>
>> I have a process group with a PublishKafkaRecord_0_10 that serialize
>> a record ( with 3 fields)
>>
>> And another process group with a ConsumeKafkaRecord_0_10 that
>> deserialise the record . that works nicely with reader/writer and with the Confluent
>> Registry storing the Avro MetaData.
>>
>> I do not have ser/deser problems because I think Nifi does both jobs
>> ……
>>
>>
>>
>> But I got problems of deserialization if I put some java Kafka client in
>> the game … ( try to consume from Confluent Kafka Streams )
>>
>> I got some deserialization exception :
>>
>> org.apache.kafka.common.errors.SerializationException: Error
>> deserializing Avro message for id -1
>>
>>
>>
>> Any trick to force another serialization ( nifi side) or
>> deserialization ( confluent side ) ?
>>
>>
>>
>> Best
>>
>> Phil
>>
>>
>>
>> _____________________________________________________________________
>> _ ___________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations
>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
>> exploites ou copies sans autorisation. Si vous avez recu ce message
>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi
>> que les pieces jointes. Les messages electroniques etant susceptibles
>> d'alteration, Orange decline toute responsabilite si ce message a ete
>> altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or
>> privileged information that may be protected by law; they should not
>> be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender
>> and delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have
>> been modified, changed or falsified.
>> Thank you.
>
> ______________________________________________________________________
> ___________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
> exploites ou copies sans autorisation. Si vous avez recu ce message
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or
> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
Re: Nifi kafka ser/deser problem
Posted by Bryan Bende <bb...@gmail.com>.
Phil,
What is the "Schema Write Strategy" set to on the AvroRecordSetWriter?
-Bryan
On Tue, Sep 5, 2017 at 10:23 AM, <ph...@orange.com> wrote:
> Hello Joe
>
> This json content {"type":"room","id":"room11","attributes":{"position":"47.100,3.246","surface":223,"norme":"NF"}}
>
> is the input of the PublishKafkaRecord_0_10 processor. This processor have among others the following properties set :
>
> Record Reader : JsonTreeReader --> pointing to the ConfluentSchemaRegistry
> Record Writer : AvroRecordSetWriter -> pointing to the confluentSchemaRegistry
>
>
> I trace the call to confluent registry and I see the get on the subject ( Confluent registry side)
> And I also see the binary content on a consumer shell console (kafka-console-consumer )
> ---------------
> room
> room1147.100,3.246▒NF
> -------------------------------
>
> Thanks
> Phil
>
> -----Original Message-----
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: mardi 5 septembre 2017 15:26
> To: users@nifi.apache.org
> Subject: Re: Nifi kafka ser/deser problem
>
> Hello
>
> Can you share the details of how you're serializing the data? We'll need to understand the configuration of the record writer of the publish kafka process to help discuss about reading it back (be that in nifi or otherwise).
>
> Since we support multiple modes of encoding the schema information this is the most likely source of difficulty.
>
> Thanks
> Joe
>
> On Tue, Sep 5, 2017 at 8:53 AM, <ph...@orange.com> wrote:
>> Hello, My SW environment à Nifi 1.4 compiled from sources , on Ubuntu
>> 14.04 and a running Confluent 3.3 Platform (running Confluent
>> Registry and kafkaStreams )
>>
>> I have a process group with a PublishKafkaRecord_0_10 that serialize a
>> record ( with 3 fields)
>>
>> And another process group with a ConsumeKafkaRecord_0_10 that
>> deserialise the record . that works nicely with reader/writer and with the Confluent
>> Registry storing the Avro MetaData.
>>
>> I do not have ser/deser problems because I think Nifi does both jobs
>> ……
>>
>>
>>
>> But I got problems of deserialization if I put some java Kafka client in
>> the game … ( try to consume from Confluent Kafka Streams )
>>
>> I got some deserialization exception :
>>
>> org.apache.kafka.common.errors.SerializationException: Error
>> deserializing Avro message for id -1
>>
>>
>>
>> Any trick to force another serialization ( nifi side) or
>> deserialization ( confluent side ) ?
>>
>>
>>
>> Best
>>
>> Phil
>>
>>
>>
>> ______________________________________________________________________
>> ___________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations
>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
>> exploites ou copies sans autorisation. Si vous avez recu ce message
>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi
>> que les pieces jointes. Les messages electroniques etant susceptibles
>> d'alteration, Orange decline toute responsabilite si ce message a ete
>> altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or
>> privileged information that may be protected by law; they should not
>> be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender and
>> delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have
>> been modified, changed or falsified.
>> Thank you.
>
> _________________________________________________________________________________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>
RE: Nifi kafka ser/deser problem
Posted by ph...@orange.com.
Hello Joe
This json content {"type":"room","id":"room11","attributes":{"position":"47.100,3.246","surface":223,"norme":"NF"}}
is the input of the PublishKafkaRecord_0_10 processor. This processor have among others the following properties set :
Record Reader : JsonTreeReader --> pointing to the ConfluentSchemaRegistry
Record Writer : AvroRecordSetWriter -> pointing to the confluentSchemaRegistry
I trace the call to confluent registry and I see the get on the subject ( Confluent registry side)
And I also see the binary content on a consumer shell console (kafka-console-consumer )
---------------
room
room1147.100,3.246▒NF
-------------------------------
Thanks
Phil
-----Original Message-----
From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: mardi 5 septembre 2017 15:26
To: users@nifi.apache.org
Subject: Re: Nifi kafka ser/deser problem
Hello
Can you share the details of how you're serializing the data? We'll need to understand the configuration of the record writer of the publish kafka process to help discuss about reading it back (be that in nifi or otherwise).
Since we support multiple modes of encoding the schema information this is the most likely source of difficulty.
Thanks
Joe
On Tue, Sep 5, 2017 at 8:53 AM, <ph...@orange.com> wrote:
> Hello, My SW environment à Nifi 1.4 compiled from sources , on Ubuntu
> 14.04 and a running Confluent 3.3 Platform (running Confluent
> Registry and kafkaStreams )
>
> I have a process group with a PublishKafkaRecord_0_10 that serialize a
> record ( with 3 fields)
>
> And another process group with a ConsumeKafkaRecord_0_10 that
> deserialise the record . that works nicely with reader/writer and with the Confluent
> Registry storing the Avro MetaData.
>
> I do not have ser/deser problems because I think Nifi does both jobs
> ……
>
>
>
> But I got problems of deserialization if I put some java Kafka client in
> the game … ( try to consume from Confluent Kafka Streams )
>
> I got some deserialization exception :
>
> org.apache.kafka.common.errors.SerializationException: Error
> deserializing Avro message for id -1
>
>
>
> Any trick to force another serialization ( nifi side) or
> deserialization ( confluent side ) ?
>
>
>
> Best
>
> Phil
>
>
>
> ______________________________________________________________________
> ___________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
> exploites ou copies sans autorisation. Si vous avez recu ce message
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi
> que les pieces jointes. Les messages electroniques etant susceptibles
> d'alteration, Orange decline toute responsabilite si ce message a ete
> altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or
> privileged information that may be protected by law; they should not
> be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and
> delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have
> been modified, changed or falsified.
> Thank you.
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.