[GitHub] [incubator-druid] glasser edited a comment on issue #7180: [PROPOSAL] Cluster names to prevent multiple Druid clusters from interfering with each other

[GitBox <gi...@apache.org>]

glasser edited a comment on issue #7180: [PROPOSAL] Cluster names to prevent multiple Druid clusters from interfering with each other
URL: https://github.com/apache/incubator-druid/issues/7180#issuecomment-469078855
 
 
   > Brokers and coordinators could ignore all segments from historicals that aren't in the druid_segments table. This potentially solves other hypothetical failure modes (running a historical on a machine with an existing segment cache which had been turned off for a long time, and some segments on it had been deleted already?) but requires more machines to talk to the metadata db more often.
   
   Here's a simpler version of the above that removes the requirement to define cluster names (though that does remove my web console benefit too :) ) and solves the exact problem in a pretty targeted way.
   
   When historicals (and real-time nodes I guess, though I don't know much about them) run `SegmentLoadDropHandler.loadLocalCache` on startup, they can check each segment that they find against the `druid_segments` table, and ignore any segments that aren't there.  (I don't know if this should crash the server or just log and ignore.)
   
   This is different from my previous proposal because it's just a single startup-time check, not an ongoing process, and it's done by historicals, not coordinators and brokers.  Instead of "filter out announced segments that aren't in the DB", it's "don't announce segments that aren't in the DB".
   
   I don't know the most appropriate way to do this check is. Do historicals have access to MetadataSegmentManager or a similar class?  I think not and they probably shouldn't just for this one. Or do they have access to something like CoordinatorClient to talk to coordinators? (Do historicals ever talk to anything other than ZooKeeper and deep storage?)
   
   (I also know ~nothing about realtime nodes other than that it looks like `SegmentLoadDropHandler` is also part of it. If it's difficult to implement this feature for realtime nodes, I could consider only doing it for historicals.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org