[jira] [Updated] (GOBBLIN-1179) Add a typed config to replace properties

["Alex Li (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/GOBBLIN-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alex Li updated GOBBLIN-1179:
-----------------------------
    Description: 
Add a typed config to replace *_properties.get(“ini.file.userName”)_* with *config.userName*

The gobblin config file is an ini file. Java loads the ini file to a properties instance. The way to use the config information is to get from the properties.
|workUnitState.getPropAsBoolean(BULK_API_USE_QUERY_ALL)|
|workUnitState.getPropAsInt(FETCH_RETRY_LIMIT_KEY, DEFAULT_FETCH_RETRY_LIMIT)|
|Math.max(MIN_SIZE,Math.min(MAX_SIZE, workUnitState.getPropAsInt(PARTITION_SIZE, DEFAULT_SIZE))); 
 // partition size must be >= min and <= max, otherwise use default|

Problems
 # No consistent key naming model

 * A long dot-separated key string is used, easy to run into typos. The config code is pretty verbose: We use *properties.getProp(key, default)*
 * Key collision if the same type is used in multiple places, e.g kafka.brokers

 # No ownership management

 * in gobblin connector package: We have multiple constant static classes. GobblinKeys, QueryBaseKeys, GaapKeys, and SalesforceConnectorKeys.We can even directly read config values by state.getProp(*“my.key”*) without creating any constant key.

 # No static validation

 * Required & default value
 * Type check
 * Date range
 * Enum

 # No dependency check

 * If users set to *useGaap=true*, there must be *gaap.url* and *gaap.credential*. And this needs to be verified at both runtime and compile time.

Doc:

[https://docs.google.com/document/d/1ZqRN4PHq2LXFMfMwpbe9UHfgOO-xIRUKdwzZ24b-F00/edit#heading=h.g5ljsztd5qwd]

 

  was:
Add a typed config to replace *_properties.get(“ini.file.userName”)_* with *config.userName*

The gobblin config file is an ini file. Java loads the ini file to a properties instance. The way to use the config information is to get from the properties.
|workUnitState.getPropAsBoolean(BULK_API_USE_QUERY_ALL)|
|workUnitState.getPropAsInt(FETCH_RETRY_LIMIT_KEY, DEFAULT_FETCH_RETRY_LIMIT)|
|Math.max(MIN_SIZE,Math.min(MAX_SIZE, workUnitState.getPropAsInt(PARTITION_SIZE, DEFAULT_SIZE))); 
// partition size must be >= min and <= max, otherwise use default|

Problems
 # No consistent key naming model
 * A long dot-separated key string is used, easy to run into typos. The config code is pretty verbose: We use *properties.getProp(key, default)*
 * Key collision if the same type is used in multiple places, e.g kafka.brokers


 # No ownership management
 * in gobblin connector package: We have multiple constant static classes. GobblinKeys, QueryBaseKeys, GaapKeys, and SalesforceConnectorKeys.We can even directly read config values by state.getProp(*“my.key”*) without creating any constant key.


 # No static validation
 * Required & default value
 * Type check
 * Date range
 * Enum


 # No dependency check
 * If users set to *useGaap=true*, there must be *gaap.url* and *gaap.credential*. And this needs to be verified at both runtime and compile time.

 


> Add a typed config to replace properties
> ----------------------------------------
>
>                 Key: GOBBLIN-1179
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1179
>             Project: Apache Gobblin
>          Issue Type: Task
>            Reporter: Alex Li
>            Priority: Major
>             Fix For: 0.15.0
>
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Add a typed config to replace *_properties.get(“ini.file.userName”)_* with *config.userName*
> The gobblin config file is an ini file. Java loads the ini file to a properties instance. The way to use the config information is to get from the properties.
> |workUnitState.getPropAsBoolean(BULK_API_USE_QUERY_ALL)|
> |workUnitState.getPropAsInt(FETCH_RETRY_LIMIT_KEY, DEFAULT_FETCH_RETRY_LIMIT)|
> |Math.max(MIN_SIZE,Math.min(MAX_SIZE, workUnitState.getPropAsInt(PARTITION_SIZE, DEFAULT_SIZE))); 
>  // partition size must be >= min and <= max, otherwise use default|
> Problems
>  # No consistent key naming model
>  * A long dot-separated key string is used, easy to run into typos. The config code is pretty verbose: We use *properties.getProp(key, default)*
>  * Key collision if the same type is used in multiple places, e.g kafka.brokers
>  # No ownership management
>  * in gobblin connector package: We have multiple constant static classes. GobblinKeys, QueryBaseKeys, GaapKeys, and SalesforceConnectorKeys.We can even directly read config values by state.getProp(*“my.key”*) without creating any constant key.
>  # No static validation
>  * Required & default value
>  * Type check
>  * Date range
>  * Enum
>  # No dependency check
>  * If users set to *useGaap=true*, there must be *gaap.url* and *gaap.credential*. And this needs to be verified at both runtime and compile time.
> Doc:
> [https://docs.google.com/document/d/1ZqRN4PHq2LXFMfMwpbe9UHfgOO-xIRUKdwzZ24b-F00/edit#heading=h.g5ljsztd5qwd]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)