[jira] [Commented] (SAMZA-144) Add stream-level overrides for KafkaSystemFactory config

["Yan Fang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14015855#comment-14015855 ] 

Yan Fang commented on SAMZA-144:
--------------------------------

I am having a look at this issue and realizing that, when we support stream-level consumer and porducer override, then, if the user specifies two different Kafka brokers for two topics (streams), how do we define which broker to use for the checkpoint? An easy solusion is, the user MUST specify
{code}
 systems.my-kafka-system.producer.*
{code}
even thought they do stream-level consumer/producer override. Otherwise, we may need to add some logic in checkpoint manager to define which Kafka broker to use.

What do you think? Thank you.

> Add stream-level overrides for KafkaSystemFactory config
> --------------------------------------------------------
>
>                 Key: SAMZA-144
>                 URL: https://issues.apache.org/jira/browse/SAMZA-144
>             Project: Samza
>          Issue Type: Bug
>          Components: kafka
>    Affects Versions: 0.6.0
>            Reporter: Chris Riccomini
>
> Currently, we allow Kafka systems to be defined with:
> {noformat}
> systems.my-kafka-system.consumer.*
> {noformat}
> And:
> {noformat}
> systems.my-kafka-system.producer.*
> {noformat}
> We don't support stream-level configurations, the way we do with StreamConfig configuration:
> {code}
>   val STREAM_PREFIX = "systems.%s.streams.%s."
>   val MSG_SERDE = STREAM_PREFIX + "samza.msg.serde"
>   val KEY_SERDE = STREAM_PREFIX + "samza.key.serde"
>   val CONSUMER_RESET_OFFSET = STREAM_PREFIX + "samza.reset.offset"
> {code}
> We should add stream-level configuration override support for Kafka consumer and producers with:
> {noformat}
> systems.my-kafka-system.streams.my-kafka-topic.consumer.*
> systems.my-kafka-system.streams.my-kafka-topic.producer.*
> {noformat}
> KafkaConfig and KafkaSystemFactory should be updated to handle these.



--
This message was sent by Atlassian JIRA
(v6.2#6252)