Question regarding dynamic subscriber environment

[Janagan Sivagnanasundaram <ja...@gmail.com>]

Kafka's current nature is does not support to dynamic subscriber
environment where the topic interest of the subscriber is keep change
overtime unless subscriber terminate the particular connection and
reconnect.

What I have planned to implement or design is to enable the dynamic
subscriber environment where the subscriber can change his topic interests
without terminating the connection with it's broker. Following are some
ideas that we can use to do such thing.

1) Modifying the current protocol by adding an extra attribute
2) Create a control connection channel from subscriber to broker

Is there easy/efficient way to do such thing?

Re: Question regarding dynamic subscriber environment

[Gerrit Jansen van Vuuren <ge...@gmail.com>]

yes. You open a connection, and the fetch threads will look at a shared
variable for the topics to fetch, its this shared variable that is updated
when you add and remove topics. The connection itself is not closed. There
is no relation between a connection and the topics being consumed, other
than the connection is used to fetch data for a particular topic, for each
broker used a pool of connections is kept, and when a topic is fetched at
any point a connection for the required broker is selected from the pool,
then returned to the pool after use.

See: https://github.com/gerritjvv/kafka-fast#java-1

Consumer consumer = Consumer.connect(new KafkaConf(), new
BrokerConf[]{new BrokerConf("192.168.4.40", 9092)}, new
RedisConf("192.168.4.10", 6379, "test-group"), "my-topic");Message msg
= consumer.readMsg();

//Add topics
consumer.addTopics("topic1", "topic2");
//Remove topics
consumer.removeTopics("topic1", "topic2");



On Wed, Nov 2, 2016 at 4:37 PM, Janagan Sivagnanasundaram <
janagan1993@gmail.com> wrote:

> Does this really address the respective problem? Ultimate task is that, the
> connection between broker and subscriber should not be terminated.
> Subscriber is free to change his topic interests without closing the
> connection.
>
> On Wed, Nov 2, 2016 at 12:43 PM, Gerrit Jansen van Vuuren <
> gerritjvv@gmail.com> wrote:
>
> > Hi,
> >
> > Have a look at the kafka client lib
> > https://github.com/gerritjvv/kafka-fast#java-1, it already provides this
> > functionality.
> >
> >
> > On Wed, Nov 2, 2016 at 2:34 AM, Janagan Sivagnanasundaram <
> > janagan1993@gmail.com> wrote:
> >
> > > Kafka's current nature is does not support to dynamic subscriber
> > > environment where the topic interest of the subscriber is keep change
> > > overtime unless subscriber terminate the particular connection and
> > > reconnect.
> > >
> > > What I have planned to implement or design is to enable the dynamic
> > > subscriber environment where the subscriber can change his topic
> > interests
> > > without terminating the connection with it's broker. Following are some
> > > ideas that we can use to do such thing.
> > >
> > > 1) Modifying the current protocol by adding an extra attribute
> > > 2) Create a control connection channel from subscriber to broker
> > >
> > > Is there easy/efficient way to do such thing?
> > >
> >
>

Re: Question regarding dynamic subscriber environment

[Janagan Sivagnanasundaram <ja...@gmail.com>]

Does this really address the respective problem? Ultimate task is that, the
connection between broker and subscriber should not be terminated.
Subscriber is free to change his topic interests without closing the
connection.

On Wed, Nov 2, 2016 at 12:43 PM, Gerrit Jansen van Vuuren <
gerritjvv@gmail.com> wrote:

> Hi,
>
> Have a look at the kafka client lib
> https://github.com/gerritjvv/kafka-fast#java-1, it already provides this
> functionality.
>
>
> On Wed, Nov 2, 2016 at 2:34 AM, Janagan Sivagnanasundaram <
> janagan1993@gmail.com> wrote:
>
> > Kafka's current nature is does not support to dynamic subscriber
> > environment where the topic interest of the subscriber is keep change
> > overtime unless subscriber terminate the particular connection and
> > reconnect.
> >
> > What I have planned to implement or design is to enable the dynamic
> > subscriber environment where the subscriber can change his topic
> interests
> > without terminating the connection with it's broker. Following are some
> > ideas that we can use to do such thing.
> >
> > 1) Modifying the current protocol by adding an extra attribute
> > 2) Create a control connection channel from subscriber to broker
> >
> > Is there easy/efficient way to do such thing?
> >
>

Re: Question regarding dynamic subscriber environment

[Gerrit Jansen van Vuuren <ge...@gmail.com>]

Hi,

Have a look at the kafka client lib
https://github.com/gerritjvv/kafka-fast#java-1, it already provides this
functionality.


On Wed, Nov 2, 2016 at 2:34 AM, Janagan Sivagnanasundaram <
janagan1993@gmail.com> wrote:

> Kafka's current nature is does not support to dynamic subscriber
> environment where the topic interest of the subscriber is keep change
> overtime unless subscriber terminate the particular connection and
> reconnect.
>
> What I have planned to implement or design is to enable the dynamic
> subscriber environment where the subscriber can change his topic interests
> without terminating the connection with it's broker. Following are some
> ideas that we can use to do such thing.
>
> 1) Modifying the current protocol by adding an extra attribute
> 2) Create a control connection channel from subscriber to broker
>
> Is there easy/efficient way to do such thing?
>