You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by QiShu <sh...@eefung.com> on 2017/12/14 12:35:59 UTC

Question about auto created Kafka topics

Hi,

 

       Coordinator and checkpoint topic will be created automatically by Samza when a task started, but these 2 topics only have one replica, how can I set the num of replicas?

 

       Thanks.

 

————————
QiShu

Re: Question about auto created Kafka topics

Posted by QiShu <sh...@eefung.com>.
Hi Jacob Maes,
	Thanks for your help!
————————
Qi Shu 

 17/12/15 00:14，“Jacob Maes”<ja...@gmail.com>:

    Samza has configurations for the replication factors used to create various
    topics. Here is how to set each one:
    
    *Checkpoint*
    task.checkpoint.replication.factor ->
    systems.<kafka-system>.default.stream.replication.factor
    -> 3
    
    Which means, if "task.checkpoint.replication.factor" is configured, it is
    used. Else if systems.<kafka-system>.default.stream.replication.factor is
    configured, it is used. Else use a default of 3.
    
    *Coordinator*
    job.coordinator.replication.factor ->
    systems.<kafka-system>.default.stream.replication.factor
    -> 3
    
    *Changelog*
    stores.<store-name>.changelog.replication.factor ->
    stores.default.changelog.replication.factor
    -> systems.<kafka-system>.default.stream.replication.factor -> 2
    
    *Intermediate streams *(for partitionBy operator in the high-level API)
    systems.<kafka-system>.default.stream.replication.factor -> 3
    
    None of them default to 1 replica, so I'm not sure why you're seeing only
    1. Does your job have any of the above properties set?
    
    What version of Samza is being used?
    
    -Jake
    
    On Thu, Dec 14, 2017 at 4:35 AM, QiShu <sh...@eefung.com> wrote:
    
    > Hi,
    >
    >
    >
    >        Coordinator and checkpoint topic will be created automatically by
    > Samza when a task started, but these 2 topics only have one replica, how
    > can I set the num of replicas?
    >
    >
    >
    >        Thanks.
    >
    >
    >
    > ————————
    > QiShu
    >
    >
    >
    >

Re: Question about auto created Kafka topics

Posted by Jacob Maes <ja...@gmail.com>.
Samza has configurations for the replication factors used to create various
topics. Here is how to set each one:

*Checkpoint*
task.checkpoint.replication.factor ->
systems.<kafka-system>.default.stream.replication.factor
-> 3

Which means, if "task.checkpoint.replication.factor" is configured, it is
used. Else if systems.<kafka-system>.default.stream.replication.factor is
configured, it is used. Else use a default of 3.

*Coordinator*
job.coordinator.replication.factor ->
systems.<kafka-system>.default.stream.replication.factor
-> 3

*Changelog*
stores.<store-name>.changelog.replication.factor ->
stores.default.changelog.replication.factor
-> systems.<kafka-system>.default.stream.replication.factor -> 2

*Intermediate streams *(for partitionBy operator in the high-level API)
systems.<kafka-system>.default.stream.replication.factor -> 3

None of them default to 1 replica, so I'm not sure why you're seeing only
1. Does your job have any of the above properties set?

What version of Samza is being used?

-Jake

On Thu, Dec 14, 2017 at 4:35 AM, QiShu <sh...@eefung.com> wrote:

> Hi,
>
>
>
>        Coordinator and checkpoint topic will be created automatically by
> Samza when a task started, but these 2 topics only have one replica, how
> can I set the num of replicas?
>
>
>
>        Thanks.
>
>
>
> ————————
> QiShu
>
>
>
>