sync and async producers

[Anh Thu Vu <vu...@gmail.com>]

Hi all,

Just 3 simple questions:
1) The settings of kafka producer (i.e. the batchsize and type - either
sync or async) are system -wise or per job?
In another word, if I run 2 jobs on the same system (same kafka broker
list, zookeeper cluster), one with
systems.kafka.producer.producer.type=sync
and one with
systems.kafka.producer.producer.type=async
does it mean the 2 producers for the 2 jobs will be of different type? or
will they overwrite each other somehow?

2) Just in case the settings of the 2 jobs overwrite each other: how can I
have different producer types on the same cluster.

3) Is it possible to have different producer type and batchsize for
different kafka topics (in the same job)?

Cheers,
Casey

RE: sync and async producers

[Garry Turkington <g....@improvedigital.com>]

Casey,

For distinct jobs with their own  config files there should be no unexpected interactions; the definition of a system in each job should be discrete.

Which is also your answer to the other part. In general if you need to address some aspect of a system in different ways within a job where there isn't task-level configurations then you  can define for example systems.kafka1, systems.kafka2 and so on. In other words view the specified system name as not only a way of referring to a system construct but also a way of segmenting the configuration namespace if that helps.

There may be some configurations in the depths of Kafka or other systems that misbehave so YMMV but this is the general abstraction.

Garry

-----Original Message-----
From: Anh Thu Vu [mailto:vuanhthu888@gmail.com] 
Sent: 07 April 2014 16:45
To: dev@samza.incubator.apache.org
Subject: sync and async producers

Hi all,

Just 3 simple questions:
1) The settings of kafka producer (i.e. the batchsize and type - either sync or async) are system -wise or per job?
In another word, if I run 2 jobs on the same system (same kafka broker list, zookeeper cluster), one with systems.kafka.producer.producer.type=sync
and one with
systems.kafka.producer.producer.type=async
does it mean the 2 producers for the 2 jobs will be of different type? or will they overwrite each other somehow?

2) Just in case the settings of the 2 jobs overwrite each other: how can I have different producer types on the same cluster.

3) Is it possible to have different producer type and batchsize for different kafka topics (in the same job)?

Cheers,
Casey

-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2014.0.4355 / Virus Database: 3722/7301 - Release Date: 04/04/14