some questions about kafka

[田晓亮 <xi...@gmail.com>]

Here is my environment:
1 kafka server with 5 partition,3 zookeeper server,1 producer,1 consumer

There is kafka log.
topic-0
....
topic-4


Producer:
when I produce message I just let zookeeper to decide which partition
should be chosen,Like that:
*producer.send(new ProducerData<String, String>("topic", messageStr)); *



Consumer:
In consumer,I just use one partition to consume message from kafka
,eventually I get every message from kafka

*Map<String, Integer> topicCountMap = new ConcurrentHashMap<String,
Integer>();*
* topicCountMap.put(topic, new Integer(1));*
*
*
*Map<String, List<KafkaMessageStream<Message>>> consumerMap = consumer*
* .createMessageStreams(topicCountMap);*




Then I start another same consumer to consume message from kafka,eventually
I get every message from kafka too.
Here is the new environment:
1 kafka server with 5 partition in each of them,3 zookeeper server,1
producer,2 consumer


But after I start another kafka server.

Here is the new environment:
2 kafka server with 5 partition in each of them,3 zookeeper server,1
producer,2 consumer

There is no kafka log in kafka server.

And eventually I just get about half of  messages from kafka.


I think maybe kafka didn't join to cluster? but actually the config is
right:
zk.connect=184.72.143.58:2181,23.20.230.102:2181,23.22.146.46:2181

then why there is no kafka log in new kafka server

and what is the relation ship between the stream number in code and
partition number in kafka.

even if  there is 5 partition in kafka,I can still just  use 1 stream to
consume  5 partition's messages,why?

if I use 5 stream to consume 5 partion,what will happend.

I'm confused about those conceptions.

Please help me ,thanks.

Re: some questions about kafka

[Jun Rao <ju...@gmail.com>]

This seems to be caused by https://issues.apache.org/jira/browse/KAFKA-278.
For now, you can create the topic on the new broker by directly producing
at least 1 message for that topic to it.

Thanks,

Jun

On Mon, Oct 8, 2012 at 11:46 PM, 田晓亮 <xi...@gmail.com> wrote:

> Here is my environment:
> 1 kafka server with 5 partition,3 zookeeper server,1 producer,1 consumer
>
> There is kafka log.
> topic-0
> ....
> topic-4
>
>
> Producer:
> when I produce message I just let zookeeper to decide which partition
> should be chosen,Like that:
> *producer.send(new ProducerData<String, String>("topic", messageStr)); *
>
>
>
> Consumer:
> In consumer,I just use one partition to consume message from kafka
> ,eventually I get every message from kafka
>
> *Map<String, Integer> topicCountMap = new ConcurrentHashMap<String,
> Integer>();*
> * topicCountMap.put(topic, new Integer(1));*
> *
> *
> *Map<String, List<KafkaMessageStream<Message>>> consumerMap = consumer*
> * .createMessageStreams(topicCountMap);*
>
>
>
>
> Then I start another same consumer to consume message from kafka,eventually
> I get every message from kafka too.
> Here is the new environment:
> 1 kafka server with 5 partition in each of them,3 zookeeper server,1
> producer,2 consumer
>
>
> But after I start another kafka server.
>
> Here is the new environment:
> 2 kafka server with 5 partition in each of them,3 zookeeper server,1
> producer,2 consumer
>
> There is no kafka log in kafka server.
>
> And eventually I just get about half of  messages from kafka.
>
>
> I think maybe kafka didn't join to cluster? but actually the config is
> right:
> zk.connect=184.72.143.58:2181,23.20.230.102:2181,23.22.146.46:2181
>
> then why there is no kafka log in new kafka server
>
> and what is the relation ship between the stream number in code and
> partition number in kafka.
>
> even if  there is 5 partition in kafka,I can still just  use 1 stream to
> consume  5 partition's messages,why?
>
> if I use 5 stream to consume 5 partion,what will happend.
>
> I'm confused about those conceptions.
>
> Please help me ,thanks.
>