[GitHub] jfim opened a new pull request #3506: Add configuration merging tool

[GitBox <gi...@apache.org>]

jfim opened a new pull request #3506: Add configuration merging tool
URL: https://github.com/apache/incubator-pinot/pull/3506
 
 
   Add a configuration merging tool to aid in migration between old-style
   configurations and the new style configuration. Configurations can be
   migrated by backing up tables using the config backup tool, then merging
   them.
   
   The merging is only necessary if configurations for multiple
   configuration profiles are desired. This happens when multiple Pinot
   clusters are managed using configuration profiles and a single source of
   truth for configurations is desired.
   
   For example, given multiple Pinot clusters in multiple data centers, one
   might have two staging clusters and two production clusters. Since the
   configuration backup tool only backs up the configuration for a single
   Pinot cluster, without merging there would be one set of configurations
   for each cluster. With config merging, one would end up with identical
   keys merged across configurations (eg. table.name is the same across all
   clusters) and different keys across configurations being merged (eg.
   table.replication___staging=1 and table.replication___prod=3).
   
   An optimization process is applied to minimize the number of generated
   keys in the case where there are heterogenous combinations of clusters
   with multiple profiles applied to them. For example, the combination
   table.replication___staging=1 and table.replication___prod=3 is
   preferred over table.replication___staging-us-west1=1,
   table.replication___staging-us-east1=1,
   table.replication___prod-us-west1=3, table.replication___us-east1=3 even
   though both combinations expand to the same data.
   
   Furthermore, a substitution process is applied to replace values by
   configuration values read from the existing profiles. For example, if a
   profile exists for "prod-us-east1" that contains
   "kafka.broker=kafka-us-east1:2181", configuration keys that have the
   value "kafka-us-east1:2181" and the "prod-us-east1" configuration
   profile enabled will get replaced with "${kafka.broker}". In the case
   where multiple replacements are possible, a minimization process is
   applied to generate the smallest set of generated keys, similar to the
   configuration optimization for keys that differ across clusters.
   
   Thus, the migration process is:
   1. Write the per profile configuration files (eg. profile/staging.conf,
      profile/staging-us-east1.conf", ...)
   2. Back up configuration files using the table config backup tool in
      directories that contain the profile tags to be applied for that
      cluster. For example, data from the staging-us-west1 cluster would be
      backed up to configBackup/staging/staging-us-west1 so that
      configuration files in that directory are understood to have the
      staging and staging-us-west1 configuration profiles.
   3. Invoke the configuration merger with the configBackup directory as
      input.
   4. Review the generated merged files for stylistic preferences, as
      desired.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org
For additional commands, e-mail: dev-help@pinot.apache.org