[GitHub] [druid] gianm commented on pull request #14281: Adding SegmentMetadataEvent and publishing them via KafkaEmitter

["gianm (via GitHub)" <gi...@apache.org>]

gianm commented on PR #14281:
URL: https://github.com/apache/druid/pull/14281#issuecomment-1550742340

   > That's a great point @gianm. Do you have a recommendation for a better place to emit this segment metadata event instead of this place to prevent this?
   
   I think for it to be "perfect" the best way to do it would be to emit in the place you emit here, but also have some other process that detects missed emits somehow and fixes them up by redoing the missed emits. This would be a lot more complex of an implementation, however. So I'd only recommend doing that if it seems worth it.
   
   To figure that out, I would consider the requirements here. What kind of things are likely to consume the emitted payloads? Could they tolerate either of the following conditions?
   
   - missed emits (segments that are published, but never emitted)
   - bogus emits (segments that are never published, but were emitted anyway)
   
   If one or both of these can be tolerated, the implementation becomes a lot simpler.
   
   If I understand correctly— the one you have in this PR is the "missed emit" scenario. It won't generate bogus emits, but it can potentially miss some.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org