Re: [I] [CORE] Configuration enhancements [incubator-gluten]

["yikf (via GitHub)" <gi...@apache.org>]

yikf commented on issue #6970:
URL: https://github.com/apache/incubator-gluten/issues/6970#issuecomment-2511452471

   Hi all, we will continue with this work, and we plan to discuss the following points.
   
   ## Project engineering
   Split the Gluten config and from a module perspective, we assigned the gluten, velox, and ch-related configurations to their respective modules. At the level of a single module, we classified the configurations into two classes, static and non-static.
   
   ## Configuration Specifications
   Standardize configuration naming, and prefix all configuration entries with `spark.gluten`.  As described in https://github.com/apache/incubator-gluten/issues/6970#issuecomment-2304090991. 
   
   Now we are leaning towards Option 2, and if there are no other objections, we will standardize based on Option 2. If there are other ideas, we can initiate an email vote.
   
   Option 1
   1. _**common config:**_ spark.gluten.sql.columnar.xxx => spark.gluten.xxx
   2. **_backend config:_** spark.gluten.sql.columnar.backend.velox.xxx => spark.gluten.backend.velox.xxx
   
   Option 2
   1. _**common config:**_ spark.gluten.sql.columnar.xxx => spark.gluten.common.xxx
   2. **_backend config:_** spark.gluten.sql.columnar.backend.velox.xxx => spark.gluten.velox..xxx
   
   ## Backward compatibility
   When configuring key modifications, we need to be compatible with the old keys and output deprecation information, gradually phasing them out in subsequent versions.
   
   ## Summary & Implementation
   To achieve greater configuration flexibility, I initially wanted to reimplement ConfigEntry. The advantage of this approach is that we can customize attributes, such as "backend" to identify more customized information. However, Gluten's configuration comes from Spark, which makes it seem unlikely that we can reimplement ConfigEntry. For example, we cannot identify static configuration if we do so. If we directly base our ConfigEntry on Spark, it may not be flexible enough, such as not being able to record expiration information.
   
   Any thoughs? @zhztheplayer @Yohahaha @zml1206 @jackylee-ch 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org