You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Janagan Sivagnanasundaram <ja...@gmail.com> on 2016/05/18 18:05:46 UTC
Question regarding enhancement for Apache Kafka
Hello,
I Janagan Sivagnanasundaram from Sri Lanka, and pursuing final year
undergraduate studies under software engineering at University of Colombo
School of Computing, Sri Lanka. As a part of the fulfillment of the degree
we have to do a final year software engineering project with 3 members. So
our project we thought of doing an enhancement for current apache kafka
framework, and the proposed enhancement is changing apache kafka from topic
based to content based, which will provide many features.
Our idea for the above proposed method is to leave the kafka as it is
without modifying and build a middleware on top of the kafka framework,
which gets the content based subscription as the input and passes to the
kafka module as set of topics, and finally kafka will process as it is
(i.e: content is divided into set of topics)
Is the above proposed way is possible to implement? Or is there any
alternative best way to solve the problem?
More importantly, our whole project should not exceed 6-8 months of period.
We have to finish the project before Dec, 2016
It will be great if you all can propose a suitable way of implement such
thing, I hope that implementing such thing would be a great contribution to
kafka community.
Thank you
Regards,
S.Janagan
RE: Question regarding enhancement for Apache Kafka
Posted by "Tauzell, Dave" <Da...@surescripts.com>.
Can you give some examples of what "content based" means?
Thanks,
Dave
-----Original Message-----
From: Janagan Sivagnanasundaram [mailto:janagan1993@gmail.com]
Sent: Wednesday, May 18, 2016 1:06 PM
To: dev@kafka.apache.org; users@kafka.apache.org
Subject: Question regarding enhancement for Apache Kafka
Hello,
I Janagan Sivagnanasundaram from Sri Lanka, and pursuing final year undergraduate studies under software engineering at University of Colombo School of Computing, Sri Lanka. As a part of the fulfillment of the degree we have to do a final year software engineering project with 3 members. So our project we thought of doing an enhancement for current apache kafka framework, and the proposed enhancement is changing apache kafka from topic based to content based, which will provide many features.
Our idea for the above proposed method is to leave the kafka as it is without modifying and build a middleware on top of the kafka framework, which gets the content based subscription as the input and passes to the kafka module as set of topics, and finally kafka will process as it is
(i.e: content is divided into set of topics)
Is the above proposed way is possible to implement? Or is there any alternative best way to solve the problem?
More importantly, our whole project should not exceed 6-8 months of period.
We have to finish the project before Dec, 2016
It will be great if you all can propose a suitable way of implement such thing, I hope that implementing such thing would be a great contribution to kafka community.
Thank you
Regards,
S.Janagan
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
Re: Question regarding enhancement for Apache Kafka
Posted by Marut Singh <ma...@delhivery.com>.
Content-filtering may mean different to different to people depending on
use case. So it's better to have examples.
On Thu, May 19, 2016 at 3:03 PM, Radoslaw Gruchalski <ra...@gruchalski.com>
wrote:
> Hey, you should have a look at Apache Samza. You put Samza on top of Kafka
> and you can inject content filtering rules into a Samza system. This will
> give you a "content subscription" system you intend to build.
>
> Get Outlook for iOS
>
>
>
>
> On Thu, May 19, 2016 at 1:56 AM -0700, "Janagan Sivagnanasundaram" <
> janagan1993@gmail.com> wrote:
>
>
>
>
>
>
>
>
>
>
> There are two type of publisher-subscriber system, topic based
> publisher-subscriber system and content based publisher-subscriber system.
> Kafka is a topic based publisher-subscriber system. We wanted to enhance
> the kafka to support content based subscriptions.
>
>
>
>
>
>
Re: Question regarding enhancement for Apache Kafka
Posted by Janagan Sivagnanasundaram <ja...@gmail.com>.
Suggested way is to use Samza on top of Kafka and then inject
content filtering rules into a Samza system. This will give a "content
subscription" system
you intend to build.
Or either to use KafkaStreams
Can anyone explain a bit regarding this? :)
On Thu, May 19, 2016 at 9:32 PM, Gwen Shapira <gw...@confluent.io> wrote:
> Or you can use KafkaStreams, which is already available in Kafka :)
>
> On Thu, May 19, 2016 at 2:33 AM, Radoslaw Gruchalski <radek@gruchalski.com
> >
> wrote:
>
> > Hey, you should have a look at Apache Samza. You put Samza on top of
> Kafka
> > and you can inject content filtering rules into a Samza system. This will
> > give you a "content subscription" system you intend to build.
> >
> > Get Outlook for iOS
> >
> >
> >
> >
> > On Thu, May 19, 2016 at 1:56 AM -0700, "Janagan Sivagnanasundaram" <
> > janagan1993@gmail.com> wrote:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > There are two type of publisher-subscriber system, topic based
> > publisher-subscriber system and content based publisher-subscriber
> system.
> > Kafka is a topic based publisher-subscriber system. We wanted to enhance
> > the kafka to support content based subscriptions.
> >
> >
> >
> >
> >
> >
>
Re: Question regarding enhancement for Apache Kafka
Posted by Gwen Shapira <gw...@confluent.io>.
Or you can use KafkaStreams, which is already available in Kafka :)
On Thu, May 19, 2016 at 2:33 AM, Radoslaw Gruchalski <ra...@gruchalski.com>
wrote:
> Hey, you should have a look at Apache Samza. You put Samza on top of Kafka
> and you can inject content filtering rules into a Samza system. This will
> give you a "content subscription" system you intend to build.
>
> Get Outlook for iOS
>
>
>
>
> On Thu, May 19, 2016 at 1:56 AM -0700, "Janagan Sivagnanasundaram" <
> janagan1993@gmail.com> wrote:
>
>
>
>
>
>
>
>
>
>
> There are two type of publisher-subscriber system, topic based
> publisher-subscriber system and content based publisher-subscriber system.
> Kafka is a topic based publisher-subscriber system. We wanted to enhance
> the kafka to support content based subscriptions.
>
>
>
>
>
>
Re: Question regarding enhancement for Apache Kafka
Posted by Radoslaw Gruchalski <ra...@gruchalski.com>.
Hey, you should have a look at Apache Samza. You put Samza on top of Kafka and you can inject content filtering rules into a Samza system. This will give you a "content subscription" system you intend to build.
Get Outlook for iOS
On Thu, May 19, 2016 at 1:56 AM -0700, "Janagan Sivagnanasundaram" <ja...@gmail.com> wrote:
There are two type of publisher-subscriber system, topic based
publisher-subscriber system and content based publisher-subscriber system.
Kafka is a topic based publisher-subscriber system. We wanted to enhance
the kafka to support content based subscriptions.
Re: Question regarding enhancement for Apache Kafka
Posted by Janagan Sivagnanasundaram <ja...@gmail.com>.
There are two type of publisher-subscriber system, topic based
publisher-subscriber system and content based publisher-subscriber system.
Kafka is a topic based publisher-subscriber system. We wanted to enhance
the kafka to support content based subscriptions.