You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sungwook Yoon <sy...@maprtech.com> on 2016/06/23 18:46:04 UTC
Failed to deserialize data to Avro:
Hi,
I am testing kafka connect and got this error,
Exception in thread "WorkerSinkTask-local-file-sink-0"
org.apache.kafka.connect.errors.DataException: Failed to deserialize data
to Avro:
at
io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:109)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:266)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:175)
at
org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
at
org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
at
org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)
Caused by: org.apache.kafka.common.errors.SerializationException: Error
deserializing Avro message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown
magic byte!
I am using
Kafka Connect 2.0.0
Kafka 0.9
with Schema Registry Running for Avro Schema Serving.
Why am I getting this error?
I believe the schema registry server is running fine, since I checked its
running through other means.
Any insight would be helpful.
Thanks,
Sungwook
RE: Failed to deserialize data to Avro:
Posted by "Tauzell, Dave" <Da...@surescripts.com>.
That should work then. I would take some messages off the queue and verify that they have the correct magic byte (byte 0).
-Dave
Dave Tauzell | Senior Software Engineer | Surescripts
O: 651.855.3042 | www.surescripts.com | Dave.Tauzell@surescripts.com
Connect with us: Twitter I LinkedIn I Facebook I YouTube
-----Original Message-----
From: Sungwook Yoon [mailto:syoon@maprtech.com]
Sent: Thursday, June 23, 2016 2:32 PM
To: users@kafka.apache.org
Subject: Re: Failed to deserialize data to Avro:
So, I am just consuming from already existing Kafka queue and topics.
According to our internal documentation, when we put event data into Kafka queue, each message has
1 magic byte
4 bytes of Schema ID
Then Avro serialized data
And we have our Schema Registry server running.
Sungwook
On Thu, Jun 23, 2016 at 12:25 PM, Tauzell, Dave < Dave.Tauzell@surescripts.com> wrote:
> How are you putting data onto the Topic? The HdfsSink expects that
> you used the KafkaAvroSerializer (
> http://docs.confluent.io/1.0/schema-registry/docs/serializer-formatter
> .html) which prepends a null byte and schema registry id to the front
> of the serialized avro data. If you just put avro onto the topic you
> will see this error.
>
> -Dave
>
> Dave Tauzell | Senior Software Engineer | Surescripts
> O: 651.855.3042 | www.surescripts.com | Dave.Tauzell@surescripts.com
> Connect with us: Twitter I LinkedIn I Facebook I YouTube
>
>
> -----Original Message-----
> From: Sungwook Yoon [mailto:syoon@maprtech.com]
> Sent: Thursday, June 23, 2016 1:46 PM
> To: users@kafka.apache.org
> Subject: Failed to deserialize data to Avro:
>
> Hi,
>
> I am testing kafka connect and got this error,
>
> Exception in thread "WorkerSinkTask-local-file-sink-0"
> org.apache.kafka.connect.errors.DataException: Failed to deserialize
> data to Avro:
> at
>
> io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:109)
> at
>
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:266)
> at
>
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:175)
> at
>
> org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
> at
>
> org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
> at
>
> org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThrea
> d.java:82) Caused by:
> org.apache.kafka.common.errors.SerializationException: Error
> deserializing Avro message for id -1 Caused by:
> org.apache.kafka.common.errors.SerializationException: Unknown magic byte!
>
>
> I am using
> Kafka Connect 2.0.0
> Kafka 0.9
> with Schema Registry Running for Avro Schema Serving.
>
> Why am I getting this error?
> I believe the schema registry server is running fine, since I checked
> its running through other means.
>
>
> Any insight would be helpful.
>
> Thanks,
>
> Sungwook
> This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of
> the individual or entity to whom they are addressed. If you have
> received this e-mail in error, please notify the sender by reply
> e-mail immediately and destroy all copies of the e-mail and any attachments.
>
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
Re: Failed to deserialize data to Avro:
Posted by Sungwook Yoon <sy...@maprtech.com>.
So, I am just consuming from already existing Kafka queue and topics.
According to our internal documentation, when we put event data into Kafka
queue, each message has
1 magic byte
4 bytes of Schema ID
Then Avro serialized data
And we have our Schema Registry server running.
Sungwook
On Thu, Jun 23, 2016 at 12:25 PM, Tauzell, Dave <
Dave.Tauzell@surescripts.com> wrote:
> How are you putting data onto the Topic? The HdfsSink expects that you
> used the KafkaAvroSerializer (
> http://docs.confluent.io/1.0/schema-registry/docs/serializer-formatter.html)
> which prepends a null byte and schema registry id to the front of the
> serialized avro data. If you just put avro onto the topic you will see
> this error.
>
> -Dave
>
> Dave Tauzell | Senior Software Engineer | Surescripts
> O: 651.855.3042 | www.surescripts.com | Dave.Tauzell@surescripts.com
> Connect with us: Twitter I LinkedIn I Facebook I YouTube
>
>
> -----Original Message-----
> From: Sungwook Yoon [mailto:syoon@maprtech.com]
> Sent: Thursday, June 23, 2016 1:46 PM
> To: users@kafka.apache.org
> Subject: Failed to deserialize data to Avro:
>
> Hi,
>
> I am testing kafka connect and got this error,
>
> Exception in thread "WorkerSinkTask-local-file-sink-0"
> org.apache.kafka.connect.errors.DataException: Failed to deserialize data
> to Avro:
> at
>
> io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:109)
> at
>
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:266)
> at
>
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:175)
> at
>
> org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
> at
>
> org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
> at
>
> org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)
> Caused by: org.apache.kafka.common.errors.SerializationException: Error
> deserializing Avro message for id -1 Caused by:
> org.apache.kafka.common.errors.SerializationException: Unknown magic byte!
>
>
> I am using
> Kafka Connect 2.0.0
> Kafka 0.9
> with Schema Registry Running for Avro Schema Serving.
>
> Why am I getting this error?
> I believe the schema registry server is running fine, since I checked its
> running through other means.
>
>
> Any insight would be helpful.
>
> Thanks,
>
> Sungwook
> This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of the
> individual or entity to whom they are addressed. If you have received this
> e-mail in error, please notify the sender by reply e-mail immediately and
> destroy all copies of the e-mail and any attachments.
>
RE: Failed to deserialize data to Avro:
Posted by "Tauzell, Dave" <Da...@surescripts.com>.
How are you putting data onto the Topic? The HdfsSink expects that you used the KafkaAvroSerializer (http://docs.confluent.io/1.0/schema-registry/docs/serializer-formatter.html) which prepends a null byte and schema registry id to the front of the serialized avro data. If you just put avro onto the topic you will see this error.
-Dave
Dave Tauzell | Senior Software Engineer | Surescripts
O: 651.855.3042 | www.surescripts.com | Dave.Tauzell@surescripts.com
Connect with us: Twitter I LinkedIn I Facebook I YouTube
-----Original Message-----
From: Sungwook Yoon [mailto:syoon@maprtech.com]
Sent: Thursday, June 23, 2016 1:46 PM
To: users@kafka.apache.org
Subject: Failed to deserialize data to Avro:
Hi,
I am testing kafka connect and got this error,
Exception in thread "WorkerSinkTask-local-file-sink-0"
org.apache.kafka.connect.errors.DataException: Failed to deserialize data to Avro:
at
io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:109)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:266)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:175)
at
org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
at
org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
at
org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)
Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1 Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!
I am using
Kafka Connect 2.0.0
Kafka 0.9
with Schema Registry Running for Avro Schema Serving.
Why am I getting this error?
I believe the schema registry server is running fine, since I checked its running through other means.
Any insight would be helpful.
Thanks,
Sungwook
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.