[GitHub] Fokko commented on issue #7001: Which parser to use for Avro encoded messages from Divolte?

[GitBox <gi...@apache.org>]

Fokko commented on issue #7001: Which parser to use for Avro encoded messages from Divolte?
URL: https://github.com/apache/incubator-druid/issues/7001#issuecomment-460552674
 
 
   Hi @kad3nce, thanks for tagging me in this discussion.
   
   The tutorial is already a bit old, I might update it when I find the time to do it :-)
   
   When pushing messages to Kafka, it won't push the schema every time, because this will become expensive as you repeat the schema quite a lot, therefore we need to provide the schema to Druid.
   Divolte supports to customize your own schema and map the incoming request to the Avro fields.
   
   -  Firstly, I would configure the Divolte hdfs sink and point it to the local file system. Periodically the Avro files should end up there. This is to verify if Divolte is happy and able to write the Avro messages. It might be that the config isn't picked up for some reason. Then you can also inspect the data using `avro-tools`.
   - Secondly, I noticed that you set the format of the timestamp to `iso`. Divolte provides a long timestamp by default.
   
   I wouldn't be surprised that there would be an encoding issue. I don't see any places where we do encoding of the messages. If you have some of the base64 encoded messages, I can start writing a unit test.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org