Re: [I] Multi-cluster Stream (Kafka/Kinesis) Druid Ingest Proposal (druid)

["gianm (via GitHub)" <gi...@apache.org>]

gianm commented on issue #18008:
URL: https://github.com/apache/druid/issues/18008#issuecomment-2937775525

   IMO, conceptually, there should be no problem having multiple supervisors per datasource. Nothing about the design of supervisors should limit that. Certain APIs do assume the supervisor ID and datasource name are the same, but this limitation could be unraveled. I think this is preferable to a composite supervisor, so each one could be managed separately.
   
   Identifying supervisors needs to be reworked a bit. IMO, each supervisor should still be associated with a specific datasource (i.e. there should be a `getDataSource` method on the supervisor). To differentiate the supervisors I think we should have a "supervisor name" that is logically namespaced within the datasource. The unique key for a supervisor would be the pair of `dataSource` + `supervisorName`. In particular, two supervisors for different datasources should be able to have the same name.
   
   IMO, we should avoid having an "ID" for supervisors that is a single string. This creates possibilities for difficult-to-resolve conflicts. For example: just from looking at the string, it's ambiguous if `foo_bar_baz` is the supervisor `baz` for datasource `foo_bar`, or the supervisor `bar_baz` for datasource `foo`. This suggests we should change the `getId` method on `SupervisorSpec` to return a `SupervisorId` (which itself contains two `String`, one for datasource and one for name) rather than returning a `String`. For compatibility with currently-existing supervisors, we should allow `String` to be read into `SupervisorId`, and create one where the string is used as the datasource name, and the supervisor name is set according to some convention (e.g. set it to the same as datasource). We'd need to include a database migration to make this work.
   
   A database migration may also be needed to store multiple metadata rows (one per supervisor, not one per datasource).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org