[GitHub] [druid] paul-rogers commented on issue #9926: A Proposal about Extension Center

[GitBox <gi...@apache.org>]

paul-rogers commented on issue #9926:
URL: https://github.com/apache/druid/issues/9926#issuecomment-970649385


   @FrankChen021, we can look to see how other Apache tools accomplish the same goals. At the very lowest level, Presto/Trino uses the [Java Service Loader](https://docs.oracle.com/javase/9/docs/api/java/util/ServiceLoader.html) as a standard way to load extensions, perhaps in their own class loader to minimize conflicts.
   
   Extensions need access to configuration, as described above. Provide this via a simple Druid-defined API to hide the implementation details. (Seems we currently use Guice to wire up configuration info, but this would be awkward for extensions.)
   
   Another question about configuration is whether it is static (loaded from config files on process launch), or dynamic (able to change in a running system.) Netflix has a [mechanism](https://github.com/Netflix/archaius) to handle dynamic configuration. There are probably others.
   
   How are extensions configured? Do they need their own config, or can the config be part of the overall Druid config? Drill uses the latter: the config system is robust enough to allow third-party config to live in either the main Druid config files, in an extension-provided config file (for defaults), or in the user customized config file.
   
   Yet another question is how to distribute extensions. Must they be part of the Druid install so that, say, one has to build a new Docker container for each? Or, can they be dynamically distributed the way that Spark pushes job jars to worker nodes? If part of the container, can we configure them so that they are outside of the Druid product directory so one can easily replace an extension, or Druid, without having to rebuild the combined Druid + extensions combination? Drill, for example, provides a "user config" directory that holds config and extension jars so one can upgrade Drill by deleting the old version and installing a new one without mixing in user files.
   
   The title "Extension Center" sounds like a web service: an app store. This suggests that extensions can be added to (or removed from, or upgraded on) a running Druid cluster. They would be installed, then configured, then enabled. To upgrade, we'd disable them, upgrade them, perhaps adjust the config, and enable again.
   
   Maybe start with cleaning up the basic mechanisms, then, when that tidied up, think about the dynamic aspects.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org