[GitHub] [incubator-hudi] pratyakshsharma commented on pull request #1565: [HUDI-73]: implemented vanilla AvroKafkaSource

[GitBox <gi...@apache.org>]

pratyakshsharma commented on pull request #1565:
URL: https://github.com/apache/incubator-hudi/pull/1565#issuecomment-629811530


   So handling schema evolutions without schema-registry is going to be really tricky. I tried googling around this stuff, and found the below 2 links. These might be useful in what we want to achieve - 
   
   1. https://stackoverflow.com/questions/37290303/producing-and-consuming-avro-messages-from-kafka-without-confluent-components
   2. https://github.com/farmdawgnation/registryless-avro-converter
   
   Particularly the second repository aims at serializing and deserializing avro data without schema-registry using Confluent and Avro libraries. At a high level, it looks like they are also not handling schema evolution in their code. I would need some time to go through it in depth though. 
   Also if you see the description of jira (https://issues.apache.org/jira/browse/HUDI-73), it mentions integration of AvroKafkaSource with FilebasedSchemaProvider (which is what is done in this PR :) ). If we really want to integrate it with FilebasedSchemaProvider, then I do not think it is feasible to handle schema evolution, since as a user, one cannot keep on changing schema files on every evolution. Thoughts? @vinothchandar 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org