Pooling Kafka Java Consumers

[Enrico Olivelli <eo...@gmail.com>]

Hi,
I'm using Java Consumer API (latest version, 0.10.1.0), I store
consumed offsets offline, that is that I do not use kafka builtin
offset storage nor consumer groups.

I need to read data from different topics, but from one topic at once,
I would like to pool my consumers (using something like Apache Commons
Pool).

My ideal flow is:
- borrow a consumer from a pool (maybe it gets created)
- assign partitions for topic A
- poll
- return the consumer to the pool

- borrow a consumer
- assign partitions for topic B
- poll
- return the consumer to the pool

- an so on....

I can guarantee that only one thread uses a consumer at once. Any kind
of error will lead to consumer disposition and removal from the pool.
Is such a way of pooling the consumer safe ? Is the a risk of getting
data from topic A while polling from topic B ?


Thank you

Enrico

Re: Pooling Kafka Java Consumers

[Guozhang Wang <wa...@gmail.com>]

This seems workable to me. But how about just using a single consumer and
re-assigning topic-partitions on-the-fly then?


Guozhang

On Mon, Nov 14, 2016 at 12:17 AM, Enrico Olivelli <eo...@gmail.com>
wrote:

> Hi,
> I'm using Java Consumer API (latest version, 0.10.1.0), I store
> consumed offsets offline, that is that I do not use kafka builtin
> offset storage nor consumer groups.
>
> I need to read data from different topics, but from one topic at once,
> I would like to pool my consumers (using something like Apache Commons
> Pool).
>
> My ideal flow is:
> - borrow a consumer from a pool (maybe it gets created)
> - assign partitions for topic A
> - poll
> - return the consumer to the pool
>
> - borrow a consumer
> - assign partitions for topic B
> - poll
> - return the consumer to the pool
>
> - an so on....
>
> I can guarantee that only one thread uses a consumer at once. Any kind
> of error will lead to consumer disposition and removal from the pool.
> Is such a way of pooling the consumer safe ? Is the a risk of getting
> data from topic A while polling from topic B ?
>
>
> Thank you
>
> Enrico
>



-- 
-- Guozhang