[jira] [Commented] (HUDI-89) Clean up placement, naming, defaults of HoodieWriteConfig

["sivabalan narayanan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-89?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17205751#comment-17205751 ] 

sivabalan narayanan commented on HUDI-89:
-----------------------------------------

sorry, I was busy for the last few weeks. Here is my understanding. I don't have full context around moving configs to right classes. I need sometime to look into CompactionConfig, StorageConfig, HoodieCleanConfig. But wrt revamping Configs in general here is the idea. 

As of now, config management is naive. Let's say we want to add a new config, we add a key string to HoodieWriteConfig, and then add a default, expose getter and setter with builder pattern. Call into setting up defaults for properties not set. and then build the HoodieWriteConfig. We wish to introduce a class called ConfigOption (source: [[1]|[https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/ConfigOption.java |https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/ConfigOption.java)]], [[2]|[https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/Configuration.java |https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/ConfigOption.java)]] and [[3]|[https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/ConfigOptions.java|https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/ConfigOption.java)]] [)|https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/ConfigOption.java)] We are not looking for a full fledged ConfigOption(which include fallback keys and and stuff), but just key, value, defaultValue and description for now. We can iteratively add more features. For eg: there was some prep work done earlier on this regards [here|https://github.com/apache/hudi/pull/1094/files].

By this, we can bind a key, default value, description for every config value together. With this, the default value is maintained along w/ the actual config in ConfigOption and so get() should return the actual value if set. if not will return the default value if set. Also, description will come in handy when we want to generate release docs. 

And btw, with this change, we also want to rename HoodieWriteConfig to HoodieClientConfig. 

[~vinoth]: I understand we don't want to do a complete overhaul which involves changes to the way users set the properties. So, may I know how do we go about populating ConfigOptions from a map of properties or from a property file. In other words, how do we intercept the value type from the property. Or am I missing something on what changes we need to make. 

 

> Clean up placement, naming, defaults of HoodieWriteConfig
> ---------------------------------------------------------
>
>                 Key: HUDI-89
>                 URL: https://issues.apache.org/jira/browse/HUDI-89
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: Code Cleanup, Usability, Writer Core
>            Reporter: Vinoth Chandar
>            Assignee: Vinoth Chandar
>            Priority: Major
>
> # Rename HoodieWriteConfig to HoodieClientConfig 
>  # Move bunch of configs from  CompactionConfig to StorageConfig 
>  # Introduce new HoodieCleanConfig
>  # Should we consider lombok or something to automate the defaults/getters/setters
>  # Consistent name of properties/defaults 
>  # Enforce bounds more strictly 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)