Re: Kafka Mirroring Issue

[Ashutosh Prasad Panigrahi <as...@gmail.com>]

Hi Francois,

Sorry in getting back to you a bit late.

I am still not able to trace the problem.
No i don't have a partition key and its not respecting "
topic.metadata.refresh.interval.ms"  , even if i set this to 0, data is
only being produced to only one partition.

Ashutosh


On Mon, Aug 25, 2014 at 10:59 PM, François Langelier <f....@gmail.com>
wrote:

> Do you have a partition key?
>
> IIRC, the DefaultPartitioner works differently if you use a partition key
> or not.
>
> If you do have a partition key, it use that algorithm:
>     Utils.abs(key.hashCode) % numPartitions
> (So if your key is the same for all your topic you will always publish in
> the same partition)
>
> And if you don't, it will change the partition every "
> topic.metadata.refresh.interval.ms"
>
>
>
> François Langelier
> Étudiant en génie Logiciel - École de Technologie Supérieure
> <http://www.etsmtl.ca/>
> Capitaine Club Capra <http://capra.etsmtl.ca/>
> VP-Communication - CS Games <http://csgames.org> 2014
> Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
>
> Comité Organisateur Olympiades ÉTS 2012
> Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
>
>
> On Mon, Aug 25, 2014 at 12:04 PM, Ashutosh Prasad Panigrahi <
> ashutosh1986@gmail.com> wrote:
>
>>  I am using the default partitioner.
>>
>> Below are the details,
>>
>> source and target have auto.create.topics.enabled=true and default number
>> of partitions is 2.
>>
>>  bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
>> mirror-maker/consumer.
>> properties --producer.config mirror-maker/producer.properties
>> --whitelist=".*" *--num.streams 2  --num.producers 2*
>>
>>
>> the consumer.properties for mirror maker looks like this
>> group.id=test-group
>> zookeeper.connect=<ZK_IP_LIST>
>>
>> and producer.properties  for mirror maker looks like this
>>
>> metadata.broker.list=<KAFKA_IP_LIST>
>> producer.type=async
>> topic.metadata.refresh.interval.ms=60000
>>
>>
>>
>> Ashutosh
>>
>>
>> On Mon, Aug 25, 2014 at 6:38 PM, François Langelier <
>> f.langelier@gmail.com> wrote:
>>
>>> What is your partitioning function?
>>>
>>>
>>>
>>> François Langelier
>>> Étudiant en génie Logiciel - École de Technologie Supérieure
>>> <http://www.etsmtl.ca/>
>>> Capitaine Club Capra <http://capra.etsmtl.ca/>
>>> VP-Communication - CS Games <http://csgames.org> 2014
>>> Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
>>> Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
>>>
>>> Comité Organisateur Olympiades ÉTS 2012
>>> Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
>>>
>>>
>>> On Mon, Aug 25, 2014 at 6:18 AM, Ashutosh Prasad Panigrahi <
>>> ashutosh1986@gmail.com> wrote:
>>>
>>> > Hi Guozhang,
>>> >
>>> > I repeated my experiment with topic.metadata.refresh.interval.ms set
>>> to 60
>>> > seconds, and executed the mirror for more than 24 hours.
>>> >
>>> > I still see that it only writes to Partition "1" and totally ignores
>>> > partition "0".
>>> >
>>> > Can you please provide me some pointers to debug this.
>>> >
>>> > Ashutosh
>>> >
>>> >
>>> > On Thu, Aug 21, 2014 at 8:52 PM, Guozhang Wang <wa...@gmail.com>
>>> wrote:
>>> >
>>> > > Hi Ashutosh,
>>> > >
>>> > > Maybe this faq can help you?
>>> > >
>>> > >
>>> > >
>>> >
>>> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified
>>> > > ?
>>> > >
>>> > > Guozhang
>>> > >
>>> > >
>>> > > On Thu, Aug 21, 2014 at 12:09 AM, Ashutosh Prasad Panigrahi <
>>> > > ashutosh1986@gmail.com> wrote:
>>> > >
>>> > > > Hello All,
>>> > > >
>>> > > > When i start the mirroring process, events are written only to
>>> > partition
>>> > > > "1" on the target cluster and no data is written to partition 0.
>>> > > >
>>> > > > I am stuck on this and wondering what am i doing wrong .
>>> > > >
>>> > > > My production setup looks something like this
>>> > > > 1. Datacenter1 :        3 kafka nodes/3 zookeeper nodes
>>> > > > 2. Datacenter 2 :       3 kafka nodes/3 zookeeper nodes
>>> > > >
>>> > > > All the kafka nodes are configured with
>>> auto.create.topics.enable=true
>>> > > and
>>> > > > the default partition is 2 .
>>> > > >
>>> > > > I start the kafka mirroring tool with the following command
>>> > > >
>>> > > >  bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
>>> > > > mirror-maker/consumer.properties --producer.config
>>> > > > mirror-maker/producer.properties --whitelist=".*" *--num.streams 2
>>> > > > --num.producers 2*
>>> > > >
>>> > > >
>>> > > > the consumer.properties for mirror-config looks like this
>>> > > > group.id=test-group
>>> > > > zookeeper.connect=<ZK_IP_LIST>
>>> > > >
>>> > > > and producer.properties  for mirror-config looks like this
>>> > > >
>>> > > > metadata.broker.list=<KAFKA_IP_LIST>
>>> > > > producer.type=async
>>> > > > topic.metadata.refresh.interval.ms=-1
>>> > > >
>>> > > >
>>> > > >
>>> > > > Regards
>>> > > > --
>>> > > > Ashutosh
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > -- Guozhang
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Ashutosh Prasad Panigrahi
>>> >
>>> > All you who sleep tonight far from the ones you love
>>> > No hands to the left or right and emptiness above
>>> > Know that you aren't alone
>>> > the whole world shares your tears
>>> > some for two nights or one
>>> > and some for all their years.......
>>> >
>>>
>>
>>
>>
>> --
>> Ashutosh Prasad Panigrahi
>>
>> All you who sleep tonight far from the ones you love
>> No hands to the left or right and emptiness above
>> Know that you aren't alone
>> the whole world shares your tears
>> some for two nights or one
>> and some for all their years.......
>>
>
>


-- 
Ashutosh Prasad Panigrahi

All you who sleep tonight far from the ones you love
No hands to the left or right and emptiness above
Know that you aren't alone
the whole world shares your tears
some for two nights or one
and some for all their years.......