[GitHub] [pulsar] KannarFr opened a new issue #5006: AvroSchema in pulsar function

[GitBox <gi...@apache.org>]

KannarFr opened a new issue #5006: AvroSchema in pulsar function
URL: https://github.com/apache/pulsar/issues/5006
 
 
   **Describe the bug**
   I'm using pulsar4s (pulsar scala client) and `pulsar-client consume` to debug.
   
   When I produce and consume using pulsar4s-avro, I've got the following message using `pulsar-client consume`:
   
   ```
   ----- got message -----
   Objavro.schema�
                  {"type":"record","name":"ApplicationLog","namespace":"models.logs.application_log_module","fields":[{"name":"syslog_pri","type":"string"},{"name":"syslog_facility","type":"string"},{"name":"syslog_facility_code","type":"int"},{"name":"syslog_severity_code","type":"int"},{"name":"syslog_severity","type":"string"},{"name":"version","type":"string"},{"name":"host","type":"string"},{"name":"zone","type":"string"},{"name":"tags","type":{"type":"array","items":"string"}},{"name":"timestamp","type":"string"},{"name":"ownerId","type":"string"},{"name":"data","type":"string"},{"name":"deploymentId","type":"string"},{"name":"program","type":"string"},{"name":"instanceId","type":"string"},{"name":"appId","type":"string"}]}avro.codenull
                                                                                                                                                                                                                                                                                                                                      >:k���HRC��30yoyoversiohoszontagtimeowneridatadeployid
                                                                                                                                                                                                                                                                                                                                                                                            prograinstanceid
   appId
        >:k���HRC�
   ```
   This one is well parsed to the wanted type using pulsar4s subscription.
   
   Now, if I produce using pulsar-function Context like:
   
   ```
   ReflectData REFLECT = new ReflectData();
   Schema schema = REFLECT.getSchema(ApplicationLog.class);
   AvroSchema<ApplicationLog> pulsarSchema = AvroSchema.of(SchemaDefinition.ApplicationLogbuilder().withJsonDef(schema.toString()).withAlwaysAllowNull(false).build());
   ```
   
   I used this native avro method to ensure that I'm using exactly the same schema that works with pulsar4s produce. So both test have exactly the same `pulsar-admin schemas get` result.
   
   ```
   context.newOutputMessage(
       topic,
       pulsarSchema
   ).value(log).sendAsync();
   ```
   
   Using `pulsar-client consume` I've got:
   
   ```
   ----- got message -----
   30
     daemon
           informational1pbinaries-clevercloud-customers.services.clever-cloud.copar0tcp
                                                                                        appLogcustomers02019-08-20T14:32:37.058ZRorga_858600a8-74f4-4d75-a8a3-f5b868be093c02019-08-20T14:32:36.690Z^deployment_1760e083-9f7a-433f-bddf-8cac43a4768adeploy-node.shH6607e717-8dad-4551-bdab-d021792715acPapp_3b65d74f-66f2-4218-bc91-6bce8d1e486a
   ```
   
   And when I try to subscribe using pulsar4s, I've got `InvalidAvroMagicException: Not an Avro data file` which normal because the avro's magic length is not correct.
   
   I don't know if it's a bug or if I just have a bad usage during pulsar-function producing.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services