[GitHub] [pinot] kishoreg commented on issue #9045: Handle poison pills in message Confluent Schema Registry Avro decoder

[GitBox <gi...@apache.org>]

kishoreg commented on issue #9045:
URL: https://github.com/apache/pinot/issues/9045#issuecomment-1183324160

   sorry for the delay in pitching in. I think the right solution might be to support this at two levels
   - Decoder (convert raw payload to GenericRow) 
      * Decoder should internally handle exception, retry etc but probably not skip a record and return null instead or generic row with skip=true or something like that and adhere to the contract.. return the new offset.
   - Pinot  (convert generic row to pinot data types)
    * this should be common behavior across all decoders, if the conversion fails and if the ignoreBadRows is set to true, we should be able to continue with the next record else log the exception and stop the consumption
   
   I also dont want to block @ddcprg. If you want to just handle the first part as part of the decoder, should be fine. please make sure it still conforms to the interface contract. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org