[GitHub] [druid] jihoonson commented on pull request #11025: Add an option for ingestion task to drop (mark unused) all existing segments that are contained by interval in the ingestionSpec

[GitBox <gi...@apache.org>]

jihoonson commented on pull request #11025:
URL: https://github.com/apache/druid/pull/11025#issuecomment-812223044


   Probably #10676 doesn't help much for this problem because this PR will mark old segments as unused as a part of publishing segments.
   
   > Hmm..if as part of the ingestion task we insert metadata-only (fake) empty segment and trick the coordinator in thinking that it is already loaded (so that we don't try to load it). Then the coordinator will only drop the old segments when the other new (real) segments are loaded. I guess we also have to make the query path skip these fake segments. Do you think this is about the right track? or is it over-complicating the solution?
   
   Publishing empty segments sounds like an easy fix. I think it will be less complicated than you described. We don't have to trick the coordinator, but should make it to skip loading empty segments. The coordinator will mark the segment overshadowed by empty ones as unused no matter whether they are loaded in historicals. The broker will not be aware of empty segments since historicals won't load nor announce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org