You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Philippe Derome <ph...@gmail.com> on 2016/06/16 18:33:06 UTC
avro-console-producer issue for Kafka Stream example PageViewRegionLambdaExample
I am able to produce Avro from producer for pageview.avsc but not for
userprofile.avsc. Here are two samples below, first one leaves out Avro
optional experience, second one specifies it.
Note: I do not register using REST curl calls schemas to schema-registry as
I assume that kafka-avro-console-producer takes care of that.
First case has Avro disagreeing that experience can be optional with an
exception:
AvroTypeException: Expected field name not found: experience
Second case generates another AvroTypeException complaining about the union
type:
Caused by: org.apache.avro.AvroTypeException: Expected start-union. Got
VALUE_STRING
./bin/kafka-avro-console-producer --broker-list localhost:9092 --topic
UserProfile --property value.schema='{
"type": "record",
"name": "UserProfile",
"fields": [
{"name": "experience", "type": ["string", "null"]},
{"name": "region", "type": "string"}
]
}'
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/Applications/confluent-3.0.0/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/Applications/confluent-3.0.0/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/Applications/confluent-3.0.0/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{"region":"US"}
org.apache.kafka.common.errors.SerializationException: Error deserializing
json {"region":"US"} to Avro of schema
{"type":"record","name":"UserProfile","fields":[{"name":"experience","type":["string","null"]},{"name":"region","type":"string"}]}
Caused by: org.apache.avro.AvroTypeException: Expected field name not
found: experience
at org.apache.avro.io.JsonDecoder.doAction(JsonDecoder.java:477)
at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.JsonDecoder.advance(JsonDecoder.java:139)
at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:429)
at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
at
org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
at
io.confluent.kafka.formatter.AvroMessageReader.jsonToAvro(AvroMessageReader.java:189)
at
io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:157)
at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:55)
at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
./bin/kafka-avro-console-producer --broker-list localhost:9092 --topic
UserProfile --property value.schema='{
"type": "record",
"name": "UserProfile",
"fields": [
{"name": "experience", "type": ["string", "null"]},
{"name": "region", "type": "string"}
]
}'
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/Applications/confluent-3.0.0/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/Applications/confluent-3.0.0/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/Applications/confluent-3.0.0/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{"region":"US","experience":"scala"}
org.apache.kafka.common.errors.SerializationException: Error deserializing
json {"region":"US","experience":"scala"} to Avro of schema
{"type":"record","name":"UserProfile","fields":[{"name":"experience","type":["string","null"]},{"name":"region","type":"string"}]}
Caused by: org.apache.avro.AvroTypeException: Expected start-union. Got
VALUE_STRING
at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:697)
at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:441)
at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
at
org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
at
io.confluent.kafka.formatter.AvroMessageReader.jsonToAvro(AvroMessageReader.java:189)
at
io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:157)
at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:55)
at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
Re: avro-console-producer issue for Kafka Stream example PageViewRegionLambdaExample
Posted by Philippe Derome <ph...@gmail.com>.
Sorry, this thread belongs to confluent not Kafka.apache.org as it pertains
to Confluent's examples.
On Thu, Jun 16, 2016 at 2:33 PM, Philippe Derome <ph...@gmail.com> wrote:
> I am able to produce Avro from producer for pageview.avsc but not for
> userprofile.avsc. Here are two samples below, first one leaves out Avro
> optional experience, second one specifies it.
>
> Note: I do not register using REST curl calls schemas to schema-registry
> as I assume that kafka-avro-console-producer takes care of that.
>
> First case has Avro disagreeing that experience can be optional with an
> exception:
>
> AvroTypeException: Expected field name not found: experience
>
> Second case generates another AvroTypeException complaining about the
> union type:
>
> Caused by: org.apache.avro.AvroTypeException: Expected start-union. Got
> VALUE_STRING
>
> ./bin/kafka-avro-console-producer --broker-list localhost:9092 --topic
> UserProfile --property value.schema='{
>
> "type": "record",
>
> "name": "UserProfile",
>
> "fields": [
>
> {"name": "experience", "type": ["string", "null"]},
>
> {"name": "region", "type": "string"}
>
> ]
>
> }'
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in
> [jar:file:/Applications/confluent-3.0.0/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/Applications/confluent-3.0.0/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/Applications/confluent-3.0.0/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> {"region":"US"}
>
> org.apache.kafka.common.errors.SerializationException: Error deserializing
> json {"region":"US"} to Avro of schema
> {"type":"record","name":"UserProfile","fields":[{"name":"experience","type":["string","null"]},{"name":"region","type":"string"}]}
>
> Caused by: org.apache.avro.AvroTypeException: Expected field name not
> found: experience
>
> at org.apache.avro.io.JsonDecoder.doAction(JsonDecoder.java:477)
>
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>
> at org.apache.avro.io.JsonDecoder.advance(JsonDecoder.java:139)
>
> at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:429)
>
> at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
>
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>
> at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
>
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
>
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
>
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
>
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
>
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
>
> at
> io.confluent.kafka.formatter.AvroMessageReader.jsonToAvro(AvroMessageReader.java:189)
>
> at
> io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:157)
>
> at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:55)
>
> at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
>
>
> ./bin/kafka-avro-console-producer --broker-list localhost:9092 --topic
> UserProfile --property value.schema='{
>
> "type": "record",
>
> "name": "UserProfile",
>
> "fields": [
>
> {"name": "experience", "type": ["string", "null"]},
>
> {"name": "region", "type": "string"}
>
> ]
>
> }'
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in
> [jar:file:/Applications/confluent-3.0.0/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/Applications/confluent-3.0.0/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/Applications/confluent-3.0.0/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> {"region":"US","experience":"scala"}
>
> org.apache.kafka.common.errors.SerializationException: Error deserializing
> json {"region":"US","experience":"scala"} to Avro of schema
> {"type":"record","name":"UserProfile","fields":[{"name":"experience","type":["string","null"]},{"name":"region","type":"string"}]}
>
> Caused by: org.apache.avro.AvroTypeException: Expected start-union. Got
> VALUE_STRING
>
> at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:697)
>
> at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:441)
>
> at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
>
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>
> at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
>
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
>
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
>
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
>
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
>
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
>
> at
> io.confluent.kafka.formatter.AvroMessageReader.jsonToAvro(AvroMessageReader.java:189)
>
> at
> io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:157)
>
> at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:55)
>
> at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
>
Re: avro-console-producer issue for Kafka Stream example PageViewRegionLambdaExample
Posted by Guozhang Wang <wa...@gmail.com>.
Here is my understanding about Avro "union type (or optional field if one
of its union value is null)":
1. Even for such fields, you need to specify them as "null" in your data,
such as {"region" : "US", "experience" : null}.
2. And if you do have a value for these union fields, you need to specify
its type, such as {"region" : "US", "experience" : {"string": "scala"}}.
Guozhang
On Thu, Jun 16, 2016 at 11:33 AM, Philippe Derome <ph...@gmail.com>
wrote:
> I am able to produce Avro from producer for pageview.avsc but not for
> userprofile.avsc. Here are two samples below, first one leaves out Avro
> optional experience, second one specifies it.
>
> Note: I do not register using REST curl calls schemas to schema-registry as
> I assume that kafka-avro-console-producer takes care of that.
>
> First case has Avro disagreeing that experience can be optional with an
> exception:
>
> AvroTypeException: Expected field name not found: experience
>
> Second case generates another AvroTypeException complaining about the union
> type:
>
> Caused by: org.apache.avro.AvroTypeException: Expected start-union. Got
> VALUE_STRING
>
> ./bin/kafka-avro-console-producer --broker-list localhost:9092 --topic
> UserProfile --property value.schema='{
>
> "type": "record",
>
> "name": "UserProfile",
>
> "fields": [
>
> {"name": "experience", "type": ["string", "null"]},
>
> {"name": "region", "type": "string"}
>
> ]
>
> }'
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in
>
> [jar:file:/Applications/confluent-3.0.0/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
>
> [jar:file:/Applications/confluent-3.0.0/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
>
> [jar:file:/Applications/confluent-3.0.0/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> {"region":"US"}
>
> org.apache.kafka.common.errors.SerializationException: Error deserializing
> json {"region":"US"} to Avro of schema
>
> {"type":"record","name":"UserProfile","fields":[{"name":"experience","type":["string","null"]},{"name":"region","type":"string"}]}
>
> Caused by: org.apache.avro.AvroTypeException: Expected field name not
> found: experience
>
> at org.apache.avro.io.JsonDecoder.doAction(JsonDecoder.java:477)
>
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>
> at org.apache.avro.io.JsonDecoder.advance(JsonDecoder.java:139)
>
> at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:429)
>
> at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
>
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>
> at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
>
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
>
> at
>
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
>
> at
>
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
>
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
>
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
>
> at
>
> io.confluent.kafka.formatter.AvroMessageReader.jsonToAvro(AvroMessageReader.java:189)
>
> at
>
> io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:157)
>
> at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:55)
>
> at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
>
>
> ./bin/kafka-avro-console-producer --broker-list localhost:9092 --topic
> UserProfile --property value.schema='{
>
> "type": "record",
>
> "name": "UserProfile",
>
> "fields": [
>
> {"name": "experience", "type": ["string", "null"]},
>
> {"name": "region", "type": "string"}
>
> ]
>
> }'
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in
>
> [jar:file:/Applications/confluent-3.0.0/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
>
> [jar:file:/Applications/confluent-3.0.0/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
>
> [jar:file:/Applications/confluent-3.0.0/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> {"region":"US","experience":"scala"}
>
> org.apache.kafka.common.errors.SerializationException: Error deserializing
> json {"region":"US","experience":"scala"} to Avro of schema
>
> {"type":"record","name":"UserProfile","fields":[{"name":"experience","type":["string","null"]},{"name":"region","type":"string"}]}
>
> Caused by: org.apache.avro.AvroTypeException: Expected start-union. Got
> VALUE_STRING
>
> at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:697)
>
> at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:441)
>
> at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
>
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>
> at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
>
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
>
> at
>
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
>
> at
>
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
>
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
>
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
>
> at
>
> io.confluent.kafka.formatter.AvroMessageReader.jsonToAvro(AvroMessageReader.java:189)
>
> at
>
> io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:157)
>
> at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:55)
>
> at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
>
--
-- Guozhang