[GitHub] jihoonson commented on issue #6834: [Proposal] Add published segment cache in broker

[GitBox <gi...@apache.org>]

jihoonson commented on issue #6834: [Proposal] Add published segment cache in broker
URL: https://github.com/apache/incubator-druid/issues/6834#issuecomment-454274573
 
 
   @surekhasaharan thanks for raising this proposal! It would be really useful for #6319. It looks good to me overall, but I have some questions for details.
   
   For phase 1,
   - How does the broker keep available segments and published segments in memory? Are they stored separately or merged into a single data structure?
   
   For phase 2,
   - Killing segments is done by the kill tasks. How to notify the coordinator on killing segments? [I suggested something related to this](https://github.com/apache/incubator-druid/issues/5304#issuecomment-376661571). Probably we can add `TriggerAction` which can be triggered by any metadata store change.
   - Should we really need to modify dataSegment? I think it's probably fine to add a new type like `DataSegmentWithCreatedDate` unless `createdDate` needs to be used in many places in addition to the coordinator cache.
   - If it's the coordinator API, it should start with `/druid/coordinator/v1`. Would you add the full URL for the new API?
   - What is the return type of the new API? It should also be able to say "there's no delta for the given timestamp. I'm sending the full set of segments".
   
   For both phase,
   - [Each broker can be configured to watch some particular tiers or dataSources](http://druid.io/docs/latest/configuration/index.html#segment-discovery). If these configurations are set, brokers don't have to retrieve all segments. How do the old & new APIs handle this?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org