You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Рябков Алексей Николаевич <a....@ntc-vulkan.ru> on 2014/05/28 17:15:41 UTC
how to control assign policy for consumers
How can I tell consumer to connect to one broker ...For example:
- I have 3 topic and 3 broker...also each topic have 6 partitions...
- I than start 6 consumers to listen this 3 topics...And in such case we can found from 6 (each consumer connect only to one broker) to 18 (each consumer connect to every broker) TCP connection .... How I can minimize network connections?
With best regards, Aleksey Ryabkov
HA: how to control assign policy for consumers
Posted by Рябков Алексей Николаевич <a....@ntc-vulkan.ru>.
Well for my task you are right... I just need to make sure that a producer would connect to as few brokers as possible, and to achieve this you just needs to know the partition assignment map, which exist on any broker's metadata cache....
But if we try (for example) to rebalance the hole system I think we must use something like maximum bipartite matching (http://en.wikipedia.org/wiki/Matching_(graph_theory)) or even maximum flow (http://en.wikipedia.org/wiki/Maximum_flow_problem )
-----Исходное сообщение-----
От: Guozhang Wang [mailto:wangguoz@gmail.com]
Отправлено: Friday, May 30, 2014 10:02 AM
Кому: users@kafka.apache.org
Тема: Re: how to control assign policy for consumers
Does your case requires analyzing connection graphs? I think you just need to make sure that a producer would connect to as few brokers as possible, and to achieve this you just needs to know the partition assignment map, which exist on any broker's metadata cache.
Guozhang
On Thu, May 29, 2014 at 12:18 PM, Рябков Алексей Николаевич < a.ryabkov@ntc-vulkan.ru> wrote:
>
> Well may be yes and may no....
> I think that in design we must point out that co-ordinator also must
> analize connection graph and must map consumers to broker with minimal
> network overhead...
>
> -----Исходное сообщение-----
> От: Guozhang Wang [mailto:wangguoz@gmail.com]
> Отправлено: Thursday, May 29, 2014 9:02 PM
> Кому: users@kafka.apache.org
> Тема: Re: how to control assign policy for consumers
>
> In the new consumer design the partitioning algorithm can be
> customized by the user:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+R
> ewrite+Design
>
> Would that fit your use case?
>
> Guozhang
>
>
> On Thu, May 29, 2014 at 8:59 AM, Рябков Алексей Николаевич <
> a.ryabkov@ntc-vulkan.ru> wrote:
>
> > May be right place to add such optimization within
> > https://cwiki.apache.org/confluence/display/KAFKA/Hierarchical+Topics ?
> >
> > -----Исходное сообщение-----
> > От: Guozhang Wang [mailto:wangguoz@gmail.com]
> > Отправлено: Wednesday, May 28, 2014 8:57 PM
> > Кому: users@kafka.apache.org
> > Тема: Re: how to control assign policy for consumers
> >
> > With high-level consumers you cannot control the assignment of
> > partitions to consumers, and if partitions are located across all
> > brokers each consumer is likely to connect to each of the brokers.
> >
> > One (little hacky) thing you can do is to manually assign the
> > partitions to brokers such that each broker i of topic t is in
> > broker i % 3, the for consumer fetching partition i of topic t, it
> > will only connect to broker i % 3. By default the assignment is not
> > aligned hence you will likely to have TCP between each pair of consumer/broker.
> >
> > Guozhang
> >
> >
> > On Wed, May 28, 2014 at 8:15 AM, Рябков Алексей Николаевич <
> > a.ryabkov@ntc-vulkan.ru> wrote:
> >
> > > How can I tell consumer to connect to one broker ...For example:
> > > - I have 3 topic and 3 broker...also each topic have 6 partitions...
> > > - I than start 6 consumers to listen this 3 topics...And in such
> > > case we can found from 6 (each consumer connect only to one
> > > broker) to
> > > 18 (each consumer connect to every broker) TCP connection .... How
> > > I can minimize network connections?
> > >
> > > With best regards, Aleksey Ryabkov
> > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> -- Guozhang
>
--
-- Guozhang
Re: how to control assign policy for consumers
Posted by Guozhang Wang <wa...@gmail.com>.
Does your case requires analyzing connection graphs? I think you just need
to make sure that a producer would connect to as few brokers as possible,
and to achieve this you just needs to know the partition assignment map,
which exist on any broker's metadata cache.
Guozhang
On Thu, May 29, 2014 at 12:18 PM, Рябков Алексей Николаевич <
a.ryabkov@ntc-vulkan.ru> wrote:
>
> Well may be yes and may no....
> I think that in design we must point out that co-ordinator also must
> analize connection graph and must map consumers to broker with minimal
> network overhead...
>
> -----Исходное сообщение-----
> От: Guozhang Wang [mailto:wangguoz@gmail.com]
> Отправлено: Thursday, May 29, 2014 9:02 PM
> Кому: users@kafka.apache.org
> Тема: Re: how to control assign policy for consumers
>
> In the new consumer design the partitioning algorithm can be customized by
> the user:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
>
> Would that fit your use case?
>
> Guozhang
>
>
> On Thu, May 29, 2014 at 8:59 AM, Рябков Алексей Николаевич <
> a.ryabkov@ntc-vulkan.ru> wrote:
>
> > May be right place to add such optimization within
> > https://cwiki.apache.org/confluence/display/KAFKA/Hierarchical+Topics ?
> >
> > -----Исходное сообщение-----
> > От: Guozhang Wang [mailto:wangguoz@gmail.com]
> > Отправлено: Wednesday, May 28, 2014 8:57 PM
> > Кому: users@kafka.apache.org
> > Тема: Re: how to control assign policy for consumers
> >
> > With high-level consumers you cannot control the assignment of
> > partitions to consumers, and if partitions are located across all
> > brokers each consumer is likely to connect to each of the brokers.
> >
> > One (little hacky) thing you can do is to manually assign the
> > partitions to brokers such that each broker i of topic t is in broker
> > i % 3, the for consumer fetching partition i of topic t, it will only
> > connect to broker i % 3. By default the assignment is not aligned
> > hence you will likely to have TCP between each pair of consumer/broker.
> >
> > Guozhang
> >
> >
> > On Wed, May 28, 2014 at 8:15 AM, Рябков Алексей Николаевич <
> > a.ryabkov@ntc-vulkan.ru> wrote:
> >
> > > How can I tell consumer to connect to one broker ...For example:
> > > - I have 3 topic and 3 broker...also each topic have 6 partitions...
> > > - I than start 6 consumers to listen this 3 topics...And in such
> > > case we can found from 6 (each consumer connect only to one broker)
> > > to
> > > 18 (each consumer connect to every broker) TCP connection .... How I
> > > can minimize network connections?
> > >
> > > With best regards, Aleksey Ryabkov
> > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> -- Guozhang
>
--
-- Guozhang
HA: how to control assign policy for consumers
Posted by Рябков Алексей Николаевич <a....@ntc-vulkan.ru>.
Well may be yes and may no....
I think that in design we must point out that co-ordinator also must analize connection graph and must map consumers to broker with minimal network overhead...
-----Исходное сообщение-----
От: Guozhang Wang [mailto:wangguoz@gmail.com]
Отправлено: Thursday, May 29, 2014 9:02 PM
Кому: users@kafka.apache.org
Тема: Re: how to control assign policy for consumers
In the new consumer design the partitioning algorithm can be customized by the user:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
Would that fit your use case?
Guozhang
On Thu, May 29, 2014 at 8:59 AM, Рябков Алексей Николаевич < a.ryabkov@ntc-vulkan.ru> wrote:
> May be right place to add such optimization within
> https://cwiki.apache.org/confluence/display/KAFKA/Hierarchical+Topics ?
>
> -----Исходное сообщение-----
> От: Guozhang Wang [mailto:wangguoz@gmail.com]
> Отправлено: Wednesday, May 28, 2014 8:57 PM
> Кому: users@kafka.apache.org
> Тема: Re: how to control assign policy for consumers
>
> With high-level consumers you cannot control the assignment of
> partitions to consumers, and if partitions are located across all
> brokers each consumer is likely to connect to each of the brokers.
>
> One (little hacky) thing you can do is to manually assign the
> partitions to brokers such that each broker i of topic t is in broker
> i % 3, the for consumer fetching partition i of topic t, it will only
> connect to broker i % 3. By default the assignment is not aligned
> hence you will likely to have TCP between each pair of consumer/broker.
>
> Guozhang
>
>
> On Wed, May 28, 2014 at 8:15 AM, Рябков Алексей Николаевич <
> a.ryabkov@ntc-vulkan.ru> wrote:
>
> > How can I tell consumer to connect to one broker ...For example:
> > - I have 3 topic and 3 broker...also each topic have 6 partitions...
> > - I than start 6 consumers to listen this 3 topics...And in such
> > case we can found from 6 (each consumer connect only to one broker)
> > to
> > 18 (each consumer connect to every broker) TCP connection .... How I
> > can minimize network connections?
> >
> > With best regards, Aleksey Ryabkov
> >
> >
>
>
> --
> -- Guozhang
>
--
-- Guozhang
Re: how to control assign policy for consumers
Posted by Guozhang Wang <wa...@gmail.com>.
In the new consumer design the partitioning algorithm can be customized by
the user:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
Would that fit your use case?
Guozhang
On Thu, May 29, 2014 at 8:59 AM, Рябков Алексей Николаевич <
a.ryabkov@ntc-vulkan.ru> wrote:
> May be right place to add such optimization within
> https://cwiki.apache.org/confluence/display/KAFKA/Hierarchical+Topics ?
>
> -----Исходное сообщение-----
> От: Guozhang Wang [mailto:wangguoz@gmail.com]
> Отправлено: Wednesday, May 28, 2014 8:57 PM
> Кому: users@kafka.apache.org
> Тема: Re: how to control assign policy for consumers
>
> With high-level consumers you cannot control the assignment of partitions
> to consumers, and if partitions are located across all brokers each
> consumer is likely to connect to each of the brokers.
>
> One (little hacky) thing you can do is to manually assign the partitions
> to brokers such that each broker i of topic t is in broker i % 3, the for
> consumer fetching partition i of topic t, it will only connect to broker i
> % 3. By default the assignment is not aligned hence you will likely to have
> TCP between each pair of consumer/broker.
>
> Guozhang
>
>
> On Wed, May 28, 2014 at 8:15 AM, Рябков Алексей Николаевич <
> a.ryabkov@ntc-vulkan.ru> wrote:
>
> > How can I tell consumer to connect to one broker ...For example:
> > - I have 3 topic and 3 broker...also each topic have 6 partitions...
> > - I than start 6 consumers to listen this 3 topics...And in such
> > case we can found from 6 (each consumer connect only to one broker) to
> > 18 (each consumer connect to every broker) TCP connection .... How I
> > can minimize network connections?
> >
> > With best regards, Aleksey Ryabkov
> >
> >
>
>
> --
> -- Guozhang
>
--
-- Guozhang
HA: how to control assign policy for consumers
Posted by Рябков Алексей Николаевич <a....@ntc-vulkan.ru>.
May be right place to add such optimization within https://cwiki.apache.org/confluence/display/KAFKA/Hierarchical+Topics ?
-----Исходное сообщение-----
От: Guozhang Wang [mailto:wangguoz@gmail.com]
Отправлено: Wednesday, May 28, 2014 8:57 PM
Кому: users@kafka.apache.org
Тема: Re: how to control assign policy for consumers
With high-level consumers you cannot control the assignment of partitions to consumers, and if partitions are located across all brokers each consumer is likely to connect to each of the brokers.
One (little hacky) thing you can do is to manually assign the partitions to brokers such that each broker i of topic t is in broker i % 3, the for consumer fetching partition i of topic t, it will only connect to broker i % 3. By default the assignment is not aligned hence you will likely to have TCP between each pair of consumer/broker.
Guozhang
On Wed, May 28, 2014 at 8:15 AM, Рябков Алексей Николаевич < a.ryabkov@ntc-vulkan.ru> wrote:
> How can I tell consumer to connect to one broker ...For example:
> - I have 3 topic and 3 broker...also each topic have 6 partitions...
> - I than start 6 consumers to listen this 3 topics...And in such
> case we can found from 6 (each consumer connect only to one broker) to
> 18 (each consumer connect to every broker) TCP connection .... How I
> can minimize network connections?
>
> With best regards, Aleksey Ryabkov
>
>
--
-- Guozhang
Re: how to control assign policy for consumers
Posted by Guozhang Wang <wa...@gmail.com>.
With high-level consumers you cannot control the assignment of partitions
to consumers, and if partitions are located across all brokers each
consumer is likely to connect to each of the brokers.
One (little hacky) thing you can do is to manually assign the partitions to
brokers such that each broker i of topic t is in broker i % 3, the for
consumer fetching partition i of topic t, it will only connect to broker i
% 3. By default the assignment is not aligned hence you will likely to have
TCP between each pair of consumer/broker.
Guozhang
On Wed, May 28, 2014 at 8:15 AM, Рябков Алексей Николаевич <
a.ryabkov@ntc-vulkan.ru> wrote:
> How can I tell consumer to connect to one broker ...For example:
> - I have 3 topic and 3 broker...also each topic have 6 partitions...
> - I than start 6 consumers to listen this 3 topics...And in such case we
> can found from 6 (each consumer connect only to one broker) to 18 (each
> consumer connect to every broker) TCP connection .... How I can minimize
> network connections?
>
> With best regards, Aleksey Ryabkov
>
>
--
-- Guozhang