[jira] [Commented] (SAMZA-40) Refactor Samza configuration

["David Chen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-40?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290972#comment-14290972 ] 

David Chen commented on SAMZA-40:
---------------------------------

{quote}
Should support global defaults for a config property. Right now, we do config.getFoo.getOrElse() everywhere.
Should validate config up front, rather than thrown runtime exceptions randomly throughout the code.
We are mixing wiring and configuration together. How do other systems handle this?
We have fragmented configuration (anybody can define configuration). How do other systems handle this?
How to handle undefined configuration? How to make this interoperable with both Java and Scala (i.e. should we support Option in Scala)?
{quote}

Using a serialization format such as Protocol Buffers, Thrift, or Avro IDL for representing configs would address a number of these concerns. By using the classes generated from the IDLs, the configs are well-defined and enforce some separation between configuration and wiring. We also get some validation built in (via types and required/optional) and defining default values for fields. Further, we also get serialization for free in case we need to send configs across the network. Finally, this provides schema evolution for when fields are added or changed.

> Refactor Samza configuration
> ----------------------------
>
>                 Key: SAMZA-40
>                 URL: https://issues.apache.org/jira/browse/SAMZA-40
>             Project: Samza
>          Issue Type: Bug
>          Components: container
>    Affects Versions: 0.6.0
>            Reporter: Chris Riccomini
>              Labels: project
>
> Samza's configuration system has several problems that we need to resolved.
> * Want to auto-generate documentation based off of configuration.
> * Should support global defaults for a config property. Right now, we do config.getFoo.getOrElse() everywhere.
> * Should validate config up front, rather than thrown runtime exceptions randomly throughout the code.
> * We are mixing wiring and configuration together. How do other systems handle this?
> * We have fragmented configuration (anybody can define configuration). How do other systems handle this?
> * How to handle undefined configuration? How to make this interoperable with both Java and Scala (i.e. should we support Option in Scala)?
> * Should remain immutable.
> * Should remove implicits. It's just confusing.
> * Do we want to support complex types (list, map) for values, not just String?
> We need a design proposal for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)