New consumer - consumer group init

[Stevo Slavić <ss...@gmail.com>]

Hello Apache Kafka community,

In new KafkaConsumer API on trunk, it seems it's only possible to define
consumer group id at construction time of KafkaConsumer, through property
with group.id key.

Would it make sense and be possible to support setting/changing consumer
group id after construction, but before it's actually used for the first
time, similar to how subscription is supported through "public void
subscribe(String... topics)"?

Maybe this can be done through additional method "public void
subscribe(String consumerGroupId, String... topics)" which would first set
provided consumer group id in coordinator and then call "public void
subscribe(String... topics)".

Kind regards,
Stevo Slavic.

Re: New consumer - consumer group init

[Stevo Slavić <ss...@gmail.com>]

Thanks all for fast feedback!

It's great news if that aspect is improved as well in new HLC. I will test
and get back with any related findings.

Kind regards,
Stevo Slavic.

On Mon, Jul 20, 2015 at 9:57 PM, Guozhang Wang <wa...@gmail.com> wrote:

> Hi Stevo,
>
> I am still not very clear on your point yet, I guess I was trying to figure
> out under which circumstances would users prefer to re-set the group id at
> an existing consumer rather than creating a new instance. As Jason
> mentioned, since the new consumer is single threaded it should usually be
> cheap to construct.
>
> Guozhang
>
> On Mon, Jul 20, 2015 at 11:06 AM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Hey Stevo,
> >
> > The new consumer doesn't have any threads of its own, so I think
> > construction should be fairly cheap.
> >
> > -Jason
> >
> > On Sun, Jul 19, 2015 at 2:13 PM, Stevo Slavić <ss...@gmail.com> wrote:
> >
> > > Hello Guozhang,
> > >
> > > It would be enough if consumer group could, besides at construction
> time,
> > > be set once only after construction. Have to retest, but high level
> > > consumer in 0.8.1 used to be very heavy weight object (lots of threads
> > > started, and it would block and take time to construct it). It's
> > > understandable, considering all of the high level features it has, and
> > > since it's supposed to be long living object. What would improve with
> > this
> > > change is that construction penalty could be paid upfront, while topic
> > > subscription and joining consumer group ensemble could be done on first
> > > use, so that first use does not have to suffer from both init and
> > > subscription penalties.
> > >
> > > It would be nice also if consumer group just as subscription could be
> > > changed later even, so multiple times throughout lifetime of high level
> > > consumer instance, to avoid constructing new consumer when instance
> > purpose
> > > changes.
> > >
> > > After looking more into the HLC API, thought maybe this is not needed,
> > > since there is "public void subscribe(TopicPartition... partitions)"
> > which
> > > does not use consumer group management, but problem is that there is no
> > > matching explicit commit where one could pass consumer group parameter
> as
> > > well, to label for which consumer group should offset(s) be committed.
> > >
> > > Seems like new HLC has split personality. Maybe (at least) two APIs
> could
> > > have been provided instead of one with such differing behaviors.
> > >
> > > Kind regards,
> > > Stevo Slavic.
> > >
> > > On Sun, Jul 19, 2015 at 12:01 AM, Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > >
> > > > Hi Stevo,
> > > >
> > > > Hmm this is interesting, do you have any use cases in mind that need
> > > > dynamic group changing?
> > > >
> > > > Guozhang
> > > >
> > > > On Fri, Jul 17, 2015 at 11:13 PM, Stevo Slavić <ss...@gmail.com>
> > > wrote:
> > > >
> > > > > Hello Apache Kafka community,
> > > > >
> > > > > In new KafkaConsumer API on trunk, it seems it's only possible to
> > > define
> > > > > consumer group id at construction time of KafkaConsumer, through
> > > property
> > > > > with group.id key.
> > > > >
> > > > > Would it make sense and be possible to support setting/changing
> > > consumer
> > > > > group id after construction, but before it's actually used for the
> > > first
> > > > > time, similar to how subscription is supported through "public void
> > > > > subscribe(String... topics)"?
> > > > >
> > > > > Maybe this can be done through additional method "public void
> > > > > subscribe(String consumerGroupId, String... topics)" which would
> > first
> > > > set
> > > > > provided consumer group id in coordinator and then call "public
> void
> > > > > subscribe(String... topics)".
> > > > >
> > > > > Kind regards,
> > > > > Stevo Slavic.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: New consumer - consumer group init

[Guozhang Wang <wa...@gmail.com>]

Hi Stevo,

I am still not very clear on your point yet, I guess I was trying to figure
out under which circumstances would users prefer to re-set the group id at
an existing consumer rather than creating a new instance. As Jason
mentioned, since the new consumer is single threaded it should usually be
cheap to construct.

Guozhang

On Mon, Jul 20, 2015 at 11:06 AM, Jason Gustafson <ja...@confluent.io>
wrote:

> Hey Stevo,
>
> The new consumer doesn't have any threads of its own, so I think
> construction should be fairly cheap.
>
> -Jason
>
> On Sun, Jul 19, 2015 at 2:13 PM, Stevo Slavić <ss...@gmail.com> wrote:
>
> > Hello Guozhang,
> >
> > It would be enough if consumer group could, besides at construction time,
> > be set once only after construction. Have to retest, but high level
> > consumer in 0.8.1 used to be very heavy weight object (lots of threads
> > started, and it would block and take time to construct it). It's
> > understandable, considering all of the high level features it has, and
> > since it's supposed to be long living object. What would improve with
> this
> > change is that construction penalty could be paid upfront, while topic
> > subscription and joining consumer group ensemble could be done on first
> > use, so that first use does not have to suffer from both init and
> > subscription penalties.
> >
> > It would be nice also if consumer group just as subscription could be
> > changed later even, so multiple times throughout lifetime of high level
> > consumer instance, to avoid constructing new consumer when instance
> purpose
> > changes.
> >
> > After looking more into the HLC API, thought maybe this is not needed,
> > since there is "public void subscribe(TopicPartition... partitions)"
> which
> > does not use consumer group management, but problem is that there is no
> > matching explicit commit where one could pass consumer group parameter as
> > well, to label for which consumer group should offset(s) be committed.
> >
> > Seems like new HLC has split personality. Maybe (at least) two APIs could
> > have been provided instead of one with such differing behaviors.
> >
> > Kind regards,
> > Stevo Slavic.
> >
> > On Sun, Jul 19, 2015 at 12:01 AM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> >
> > > Hi Stevo,
> > >
> > > Hmm this is interesting, do you have any use cases in mind that need
> > > dynamic group changing?
> > >
> > > Guozhang
> > >
> > > On Fri, Jul 17, 2015 at 11:13 PM, Stevo Slavić <ss...@gmail.com>
> > wrote:
> > >
> > > > Hello Apache Kafka community,
> > > >
> > > > In new KafkaConsumer API on trunk, it seems it's only possible to
> > define
> > > > consumer group id at construction time of KafkaConsumer, through
> > property
> > > > with group.id key.
> > > >
> > > > Would it make sense and be possible to support setting/changing
> > consumer
> > > > group id after construction, but before it's actually used for the
> > first
> > > > time, similar to how subscription is supported through "public void
> > > > subscribe(String... topics)"?
> > > >
> > > > Maybe this can be done through additional method "public void
> > > > subscribe(String consumerGroupId, String... topics)" which would
> first
> > > set
> > > > provided consumer group id in coordinator and then call "public void
> > > > subscribe(String... topics)".
> > > >
> > > > Kind regards,
> > > > Stevo Slavic.
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>



-- 
-- Guozhang

Re: New consumer - consumer group init

[Jason Gustafson <ja...@confluent.io>]

Hey Stevo,

The new consumer doesn't have any threads of its own, so I think
construction should be fairly cheap.

-Jason

On Sun, Jul 19, 2015 at 2:13 PM, Stevo Slavić <ss...@gmail.com> wrote:

> Hello Guozhang,
>
> It would be enough if consumer group could, besides at construction time,
> be set once only after construction. Have to retest, but high level
> consumer in 0.8.1 used to be very heavy weight object (lots of threads
> started, and it would block and take time to construct it). It's
> understandable, considering all of the high level features it has, and
> since it's supposed to be long living object. What would improve with this
> change is that construction penalty could be paid upfront, while topic
> subscription and joining consumer group ensemble could be done on first
> use, so that first use does not have to suffer from both init and
> subscription penalties.
>
> It would be nice also if consumer group just as subscription could be
> changed later even, so multiple times throughout lifetime of high level
> consumer instance, to avoid constructing new consumer when instance purpose
> changes.
>
> After looking more into the HLC API, thought maybe this is not needed,
> since there is "public void subscribe(TopicPartition... partitions)" which
> does not use consumer group management, but problem is that there is no
> matching explicit commit where one could pass consumer group parameter as
> well, to label for which consumer group should offset(s) be committed.
>
> Seems like new HLC has split personality. Maybe (at least) two APIs could
> have been provided instead of one with such differing behaviors.
>
> Kind regards,
> Stevo Slavic.
>
> On Sun, Jul 19, 2015 at 12:01 AM, Guozhang Wang <wa...@gmail.com>
> wrote:
>
> > Hi Stevo,
> >
> > Hmm this is interesting, do you have any use cases in mind that need
> > dynamic group changing?
> >
> > Guozhang
> >
> > On Fri, Jul 17, 2015 at 11:13 PM, Stevo Slavić <ss...@gmail.com>
> wrote:
> >
> > > Hello Apache Kafka community,
> > >
> > > In new KafkaConsumer API on trunk, it seems it's only possible to
> define
> > > consumer group id at construction time of KafkaConsumer, through
> property
> > > with group.id key.
> > >
> > > Would it make sense and be possible to support setting/changing
> consumer
> > > group id after construction, but before it's actually used for the
> first
> > > time, similar to how subscription is supported through "public void
> > > subscribe(String... topics)"?
> > >
> > > Maybe this can be done through additional method "public void
> > > subscribe(String consumerGroupId, String... topics)" which would first
> > set
> > > provided consumer group id in coordinator and then call "public void
> > > subscribe(String... topics)".
> > >
> > > Kind regards,
> > > Stevo Slavic.
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: New consumer - consumer group init

[Stevo Slavić <ss...@gmail.com>]

Hello Guozhang,

It would be enough if consumer group could, besides at construction time,
be set once only after construction. Have to retest, but high level
consumer in 0.8.1 used to be very heavy weight object (lots of threads
started, and it would block and take time to construct it). It's
understandable, considering all of the high level features it has, and
since it's supposed to be long living object. What would improve with this
change is that construction penalty could be paid upfront, while topic
subscription and joining consumer group ensemble could be done on first
use, so that first use does not have to suffer from both init and
subscription penalties.

It would be nice also if consumer group just as subscription could be
changed later even, so multiple times throughout lifetime of high level
consumer instance, to avoid constructing new consumer when instance purpose
changes.

After looking more into the HLC API, thought maybe this is not needed,
since there is "public void subscribe(TopicPartition... partitions)" which
does not use consumer group management, but problem is that there is no
matching explicit commit where one could pass consumer group parameter as
well, to label for which consumer group should offset(s) be committed.

Seems like new HLC has split personality. Maybe (at least) two APIs could
have been provided instead of one with such differing behaviors.

Kind regards,
Stevo Slavic.

On Sun, Jul 19, 2015 at 12:01 AM, Guozhang Wang <wa...@gmail.com> wrote:

> Hi Stevo,
>
> Hmm this is interesting, do you have any use cases in mind that need
> dynamic group changing?
>
> Guozhang
>
> On Fri, Jul 17, 2015 at 11:13 PM, Stevo Slavić <ss...@gmail.com> wrote:
>
> > Hello Apache Kafka community,
> >
> > In new KafkaConsumer API on trunk, it seems it's only possible to define
> > consumer group id at construction time of KafkaConsumer, through property
> > with group.id key.
> >
> > Would it make sense and be possible to support setting/changing consumer
> > group id after construction, but before it's actually used for the first
> > time, similar to how subscription is supported through "public void
> > subscribe(String... topics)"?
> >
> > Maybe this can be done through additional method "public void
> > subscribe(String consumerGroupId, String... topics)" which would first
> set
> > provided consumer group id in coordinator and then call "public void
> > subscribe(String... topics)".
> >
> > Kind regards,
> > Stevo Slavic.
> >
>
>
>
> --
> -- Guozhang
>

Re: New consumer - consumer group init

[Guozhang Wang <wa...@gmail.com>]

Hi Stevo,

Hmm this is interesting, do you have any use cases in mind that need
dynamic group changing?

Guozhang

On Fri, Jul 17, 2015 at 11:13 PM, Stevo Slavić <ss...@gmail.com> wrote:

> Hello Apache Kafka community,
>
> In new KafkaConsumer API on trunk, it seems it's only possible to define
> consumer group id at construction time of KafkaConsumer, through property
> with group.id key.
>
> Would it make sense and be possible to support setting/changing consumer
> group id after construction, but before it's actually used for the first
> time, similar to how subscription is supported through "public void
> subscribe(String... topics)"?
>
> Maybe this can be done through additional method "public void
> subscribe(String consumerGroupId, String... topics)" which would first set
> provided consumer group id in coordinator and then call "public void
> subscribe(String... topics)".
>
> Kind regards,
> Stevo Slavic.
>



-- 
-- Guozhang