[GitHub] [incubator-druid] gianm commented on issue #6834: [Proposal] Add published segment cache in broker

[GitBox <gi...@apache.org>]

gianm commented on issue #6834: [Proposal] Add published segment cache in broker
URL: https://github.com/apache/incubator-druid/issues/6834#issuecomment-478621453
 
 
   > Brokers delegate Druid SQL queries against sys.segments to Coordinator. So brokers don't have to deal with published (used) segments in their memory at all.
   
   We had talked a bit about this idea offline last week and I believe generally thought it was a good idea in isolation. In particular, we had thought about moving the entire `sys` schema implementation to the Coordinator and having the Broker send any SQL queries on `sys` over there. Users could also query `sys` tables on the Coordinator directly if they wanted.
   
   However, there were two counterpoints raised:
   
   - It is a somewhat common request from our users to add an option to the Broker to either fail a query, or provide a warning, if that query is returning partial results because the underlying segments are partially unavailable. Right now, if some segments are not available, the Broker's query runner does not know (it has zero information about unavailable segments). So it will simply return partial results, without informing the caller. Definitely you could imagine life being better if the caller knew about this. For the feature to work properly the Broker needs to know about segments that are published but unavailable, which is the same
   - https://github.com/apache/incubator-druid/issues/6319 contemplates a design for finer-grained locking and finer-grained atomic overwriting that would be useful for a compaction feature. Its design, as currently contemplated, requires the Broker to retain information about segments that are published but unavailable.
   
   > Brokers access the metadata store to update their view of published (used) segments directly, without Coordinator as a proxy.
   
   This one was considered as well, the main counterpoint raised was that currently, only the Coordinator and Overlord have a direct dependency on the metadata store, and there was a desire to keep it that way. In turn there were two reasons for that:
   
   - A desire to minimize the number of services that depend on the metadata store, both for architectural complexity and load reasons.
   - A thought that someday, in the future, we might want to bring the metadata store 'in house' and have it be something the Coordinator does on its own without an external dependency. This could be through implementing Raft or some other consensus protocol on the Coordinators, and using that to allow the Coordinators to take on the functionality of both ZooKeeper and the metadata store. The motivation would be to simplify deployment and reduce the number of mandatory external dependencies. In this case, any other server type that needs to interact with metadata would have to do so through Coordinator APIs. So we might as well stick to that now to make a potential future change easier.
   
   > For memory savings, adding an extra flags to DataSegment object: isAvailable, isPublished, and using a single unified ConcurrentHashMap view for the purposes of both BrokerServerView and MetadataSegmentView.
   
   This was not discussed. I haven't thought through the implications fully but I would worry that it means either DataSegment must no longer be immutable (because isAvailable, etc can change) or that it remains immutable but can no longer be interned effectively (because there's no longer a single DataSegment object per SegmentId). So I would look at those things first when evaluating this idea.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org