are topics and partitions dynamic?

[S Ahmed <sa...@gmail.com>]

I'm looking at the kafka server.properties that is in /config folder and
didn't find any reference to topics or partitions.

Are topics and optionally partitions dynamic in nature or do they have to
be defined before starting the broker? (and consequently producers can't
send to a topic and partition that isn't predefined).

Re: are topics and partitions dynamic?

[小宇 <mo...@gmail.com>]

 "auto.create.topics.enable" are default true, so you don't need care about
it. And you can set it false in the server.properties


2013/12/4 Magnus Edenhill <ma...@edenhill.se>

> See the configuration reference here:
>
> https://kafka.apache.org/documentation.html#configuration
>
>
> 2013/12/4 Arjun <ar...@socialtwist.com>
>
> > hi,
> >
> > I was just looking at kafka 0.8, I could not find any option in
> > config/server.properties with key "auto.create.topics.enable"  or
> > "default.replication.factor".
> >
> > Can some one help me out,where i can find these. I want my topics to be
> > created dynamically.
> >
> >
> > thanks
> > Arjun Narasimha Kota
> >
>

Re: are topics and partitions dynamic?

[Magnus Edenhill <ma...@edenhill.se>]

See the configuration reference here:

https://kafka.apache.org/documentation.html#configuration


2013/12/4 Arjun <ar...@socialtwist.com>

> hi,
>
> I was just looking at kafka 0.8, I could not find any option in
> config/server.properties with key "auto.create.topics.enable"  or
> "default.replication.factor".
>
> Can some one help me out,where i can find these. I want my topics to be
> created dynamically.
>
>
> thanks
> Arjun Narasimha Kota
>

Re: are topics and partitions dynamic?

[Arjun <ar...@socialtwist.com>]

hi,

I was just looking at kafka 0.8, I could not find any option in 
config/server.properties with key "auto.create.topics.enable"  or 
"default.replication.factor".

Can some one help me out,where i can find these. I want my topics to be 
created dynamically.


thanks
Arjun Narasimha Kota

Re: are topics and partitions dynamic?

[Jun Rao <ju...@gmail.com>]

In 0.8, when a topic is created, we decide how each replica of a partition
is mapped to a broker. In general, we try to spread the replicas across all
brokers.

Thanks,

Jun

On Sun, Jan 27, 2013 at 12:40 PM, S Ahmed <sa...@gmail.com> wrote:

> So from what I understand, a single broker will contain all the messages
> for a given topic (and its partitions).  Who decides which broker will own
> a particular topic?
>
>
> On Sun, Jan 27, 2013 at 3:36 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > In Kafka 0.7, topics and partitions are dynamic in the sense that a
> > partition is created when the broker receives the first message for that
> > partition.
> > In Kafka 0.8, there are 2 ways of creating a new topic -
> > 1. Turn on auto.create.topics.enable option on the broker. When the
> broker
> > receives the first message for a new topic, it creates that topic with
> > num.partitions and default.replication.factor.
> > 2. Use the admin command bin/kafka-create-topic.sh
> >
> > Thanks,
> > Neha
> >
> >
> > On Sun, Jan 27, 2013 at 12:23 PM, S Ahmed <sa...@gmail.com> wrote:
> >
> > > I'm looking at the kafka server.properties that is in /config folder
> and
> > > didn't find any reference to topics or partitions.
> > >
> > > Are topics and optionally partitions dynamic in nature or do they have
> to
> > > be defined before starting the broker? (and consequently producers
> can't
> > > send to a topic and partition that isn't predefined).
> > >
> >
>

Re: are topics and partitions dynamic?

[S Ahmed <sa...@gmail.com>]

So from what I understand, a single broker will contain all the messages
for a given topic (and its partitions).  Who decides which broker will own
a particular topic?


On Sun, Jan 27, 2013 at 3:36 PM, Neha Narkhede <ne...@gmail.com>wrote:

> In Kafka 0.7, topics and partitions are dynamic in the sense that a
> partition is created when the broker receives the first message for that
> partition.
> In Kafka 0.8, there are 2 ways of creating a new topic -
> 1. Turn on auto.create.topics.enable option on the broker. When the broker
> receives the first message for a new topic, it creates that topic with
> num.partitions and default.replication.factor.
> 2. Use the admin command bin/kafka-create-topic.sh
>
> Thanks,
> Neha
>
>
> On Sun, Jan 27, 2013 at 12:23 PM, S Ahmed <sa...@gmail.com> wrote:
>
> > I'm looking at the kafka server.properties that is in /config folder and
> > didn't find any reference to topics or partitions.
> >
> > Are topics and optionally partitions dynamic in nature or do they have to
> > be defined before starting the broker? (and consequently producers can't
> > send to a topic and partition that isn't predefined).
> >
>

Re: are topics and partitions dynamic?

[Neha Narkhede <ne...@gmail.com>]

In Kafka 0.7, topics and partitions are dynamic in the sense that a
partition is created when the broker receives the first message for that
partition.
In Kafka 0.8, there are 2 ways of creating a new topic -
1. Turn on auto.create.topics.enable option on the broker. When the broker
receives the first message for a new topic, it creates that topic with
num.partitions and default.replication.factor.
2. Use the admin command bin/kafka-create-topic.sh

Thanks,
Neha


On Sun, Jan 27, 2013 at 12:23 PM, S Ahmed <sa...@gmail.com> wrote:

> I'm looking at the kafka server.properties that is in /config folder and
> didn't find any reference to topics or partitions.
>
> Are topics and optionally partitions dynamic in nature or do they have to
> be defined before starting the broker? (and consequently producers can't
> send to a topic and partition that isn't predefined).
>