You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/02/04 11:09:49 UTC
[GitHub] [pulsar] crazylab opened a new issue #6213: [Client] JSONSchema not
able to encode message in case class (scala)
crazylab opened a new issue #6213: [Client] JSONSchema not able to encode message in case class (scala)
URL: https://github.com/apache/pulsar/issues/6213
Client Library version: 2.5.0
**Describe the bug**
The producer is not able to produce JSON data with the content of the case class. Only empty values are getting passed to the broker. This is primarily because the JSON encoder not able to encode the message properly.
**Code snippet**
```scala
case class Student(name: String, roll: Int)
val client: PulsarClient = PulsarClient.builder()
.serviceUrl("pulsar://localhost:6650")
.build();
val producer = client.newProducer(JSONSchema.of(classOf[Student]))
.topic("foobar")
.sendTimeout(3, TimeUnit.SECONDS).create
def produce(msg: Student) = {
producer.send(msg)
}
```
**Expected behavior**
Should encode the data available in case class and send it to the broker for the pulsar consumer.
**Desktop (please complete the following information):**
- Mac: [e.g. iOS]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] Nainwake edited a comment on issue #6213: [Client]
JSONSchema not able to encode message in case class (scala)
Posted by GitBox <gi...@apache.org>.
Nainwake edited a comment on issue #6213: [Client] JSONSchema not able to encode message in case class (scala)
URL: https://github.com/apache/pulsar/issues/6213#issuecomment-613073498
Hey @yjshen thanks for the reply.
Indeed it works well for producer and consumer.
See below a quick and dirty case class to be able to produce/consume without troubles :
`
case class CaseMessage(
@BeanProperty key: String,
@BeanProperty event_time: Long,
@BeanProperty value: String
)
{
def this() = this("", 0, "")
}
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] yjshen commented on issue #6213: [Client] JSONSchema not
able to encode message in case class (scala)
Posted by GitBox <gi...@apache.org>.
yjshen commented on issue #6213: [Client] JSONSchema not able to encode message in case class (scala)
URL: https://github.com/apache/pulsar/issues/6213#issuecomment-585208764
To use Scala case class as POJO, you should write like this:
```scala
case class Foo(@BeanProperty i: Int, @BeanProperty f: Float, @BeanProperty bar: Bar)
case class Bar(@BeanProperty b: Boolean, @BeanProperty s: String)
case class F1(@BeanProperty baz: Baz)
case class Baz(
@BeanProperty f: Float,
@BeanProperty d: Double,
@BeanProperty mp: util.Map[String, Bar],
@BeanProperty arr: Array[Bar])
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #6213: [Client] JSONSchema not
able to encode message in case class (scala)
Posted by GitBox <gi...@apache.org>.
jiazhai commented on issue #6213: [Client] JSONSchema not able to encode message in case class (scala)
URL: https://github.com/apache/pulsar/issues/6213#issuecomment-585116566
@yjshen Would you please help take a look when you have time?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] yjshen edited a comment on issue #6213: [Client]
JSONSchema not able to encode message in case class (scala)
Posted by GitBox <gi...@apache.org>.
yjshen edited a comment on issue #6213: [Client] JSONSchema not able to encode message in case class (scala)
URL: https://github.com/apache/pulsar/issues/6213#issuecomment-612549279
How about adding a no-arg constructor in the case class? Does it works?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] crazylab commented on issue #6213: [Client] JSONSchema not
able to encode message in case class (scala)
Posted by GitBox <gi...@apache.org>.
crazylab commented on issue #6213: [Client] JSONSchema not able to encode message in case class (scala)
URL: https://github.com/apache/pulsar/issues/6213#issuecomment-585220007
@yjshen Tried as per your suggestion. Now the message is getting produced correctly. But the consumer is failing while deserialising the message.
```scala
case class Student(@BeanProperty name: String, @BeanProperty roll: Int)
val client: PulsarClient = PulsarClient.builder()
.serviceUrl("pulsar://localhost:6650")
.build();
client.newReader()
val consumer = client.newConsumer(JSONSchema.of(classOf[Student]))
.topic(Const.TOPIC_NAME)
.subscriptionName("s1")
.subscribe()
def consume(): Message[Student] = {
consumer.receive()
}
```
```text
Exception in thread "main" org.apache.pulsar.client.api.SchemaSerializationException: org.apache.pulsar.shade.com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `com.Student` (no Creators, like default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
at [Source: (byte[])"{"name":"A","roll":1}"; line: 1, column: 2]
at org.apache.pulsar.client.impl.schema.reader.JsonReader.read(JsonReader.java:41)
at org.apache.pulsar.client.impl.schema.StructSchema.decode(StructSchema.java:88)
at org.apache.pulsar.client.impl.MessageImpl.getValue(MessageImpl.java:273)
at com.consumer.SomethingElse$.main(Consumer.scala:42)
at com.consumer.SomethingElse.main(Consumer.scala)
Caused by: org.apache.pulsar.shade.com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `com.Student` (no Creators, like default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
at [Source: (byte[])"{"name":"A","roll":1}"; line: 1, column: 2]
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1452)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1028)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1297)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4014)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3092)
at org.apache.pulsar.client.impl.schema.reader.JsonReader.read(JsonReader.java:39)
... 4 more
```
A different error is coming when using `AvroSchema`
```text
Exception in thread "main" java.lang.RuntimeException: java.lang.NoSuchMethodException: com.Student.<init>()
at org.apache.pulsar.shade.org.apache.avro.specific.SpecificData.newInstance(SpecificData.java:353)
at org.apache.pulsar.shade.org.apache.avro.specific.SpecificData.newRecord(SpecificData.java:369)
at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData.newRecord(ReflectData.java:901)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:212)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
at org.apache.pulsar.client.impl.schema.reader.AvroReader.read(AvroReader.java:52)
at org.apache.pulsar.client.impl.schema.StructSchema.decode(StructSchema.java:94)
at org.apache.pulsar.client.impl.MessageImpl.getValue(MessageImpl.java:270)
at com.consumer.SomethingElse$.main(Consumer.scala:42)
at com.consumer.SomethingElse.main(Consumer.scala)
Caused by: java.lang.NoSuchMethodException: com.Student.<init>()
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getDeclaredConstructor(Class.java:2178)
at org.apache.pulsar.shade.org.apache.avro.specific.SpecificData.newInstance(SpecificData.java:347)
... 11 more
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] Nainwake commented on issue #6213: [Client] JSONSchema not
able to encode message in case class (scala)
Posted by GitBox <gi...@apache.org>.
Nainwake commented on issue #6213: [Client] JSONSchema not able to encode message in case class (scala)
URL: https://github.com/apache/pulsar/issues/6213#issuecomment-613073498
Hey @yjshen thanks for the reply.
Indeed it works well for producer and consumer.
See below a quick and dirty case class to be able to produce/consume without troubles :
`case class CaseMessage(
@BeanProperty key: String,
@BeanProperty event_time: Long,
@BeanProperty value: String
)
{
def this() = this("", 0, "")
}`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] crazylab commented on issue #6213: [Client] JSONSchema not
able to encode message in case class (scala)
Posted by GitBox <gi...@apache.org>.
crazylab commented on issue #6213: [Client] JSONSchema not able to encode message in case class (scala)
URL: https://github.com/apache/pulsar/issues/6213#issuecomment-585169092
Hi, figured out that this issue is basically happening because Pulsar is not able to serialise the Scala `case class` properly. Tried replacing the scala case class with Java POJO and it has started working.
We have also observed similar behaviour with `AvroSchema`. In that case though it is able to serialise the case class but unable to deserialise.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] yjshen commented on issue #6213: [Client] JSONSchema not
able to encode message in case class (scala)
Posted by GitBox <gi...@apache.org>.
yjshen commented on issue #6213: [Client] JSONSchema not able to encode message in case class (scala)
URL: https://github.com/apache/pulsar/issues/6213#issuecomment-612549279
How about adding a no-arg constructor in the case class?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] Nainwake commented on issue #6213: [Client] JSONSchema not
able to encode message in case class (scala)
Posted by GitBox <gi...@apache.org>.
Nainwake commented on issue #6213: [Client] JSONSchema not able to encode message in case class (scala)
URL: https://github.com/apache/pulsar/issues/6213#issuecomment-612518614
Any updates on the consumer with scala case class ? Problem stil exists in 2.5.0.
pulsar4s seems to works from what i see.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] yjshen edited a comment on issue #6213: [Client]
JSONSchema not able to encode message in case class (scala)
Posted by GitBox <gi...@apache.org>.
yjshen edited a comment on issue #6213: [Client] JSONSchema not able to encode message in case class (scala)
URL: https://github.com/apache/pulsar/issues/6213#issuecomment-612549279
How about adding a no-arg constructor in the case class? Does it work?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services