[GitHub] [incubator-pinot] jackjlli commented on pull request #5984: Add row based schema validation code to detect schema mismatch

[GitBox <gi...@apache.org>]

jackjlli commented on pull request #5984:
URL: https://github.com/apache/incubator-pinot/pull/5984#issuecomment-689274482


   Most of the cases can be covered by validating pinot schema and avro schema. One tricky thing is that when all the fields are required to be fetched, we convert the avro generic record to string first, then parse it as a json:
   https://github.com/apache/incubator-pinot/blob/d54b04a2562f86dfb3adaa02ff400951d8108738/pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroRecordExtractor.java#L49
   ```
     /**
      * Converts from a GenericRecord to a json map
      */
     public static Map<String, Object> genericRecordToJson(GenericRecord genericRecord) {
       try {
         String jsonString = genericRecord.toString();
         return DEFAULT_MAPPER.readValue(jsonString, new TypeReference<Map<String, Object>>() {
         });
       } catch (IOException e) {
         throw new IllegalStateException("Caught exception when converting generic record " + genericRecord + " to JSON");
       }
     }
   ```
   The data type of the value from the k-v pair might get changed. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org