[GitHub] [pinot] jadami10 commented on issue #9014: Allowing empty segments for all stream consumers

[GitBox <gi...@apache.org>]

jadami10 commented on issue #9014:
URL: https://github.com/apache/pinot/issues/9014#issuecomment-1175179709

   The problem is that there is data published, and it's quite high volume. We are just filtering it all out in our decoder. So in this case, the server never actually commits the segment to advance the offsets. So a restart will cause it to go all the way back to the last time it saw a message it didn't filter out which can be hours or days.
   
   Our table config is has a bunch of custom keys for our internal stream ingestion plugin, so i'm not sure it would be helpful.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org