You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Li Tao <ah...@gmail.com> on 2015/09/13 07:34:50 UTC
open source projects based on kafka
Hi Hackers,
This is Lee, a learner of kafka, i have read the original paper on kafka,
and walked through the document.
I think the best way to learn sth is to write and read code about it. I am
wondering is there any open source code / system which is based on kafka so
that i can read or contribute to? Not too complex, not too simple.
Thanks a lot!
Re: open source projects based on kafka
Posted by Jeff Holoman <jh...@cloudera.com>.
You could try implementing your own client.
https://cwiki.apache.org/confluence/display/KAFKA/Clients
On Sun, Sep 13, 2015 at 7:46 PM, Stevo Slavić <ss...@gmail.com> wrote:
> Have a look at https://github.com/allegro/hermes
>
> On Mon, Sep 14, 2015, 01:28 David Luu <ma...@gmail.com> wrote:
>
> > The toy project idea is good. Another option I think could be to look at
> > the various Kafka client langague bindings and/or utilities (like
> > kafkacat). And from there, another option is to build a client language
> > binding for a language that's kind of lacking Kafka support, some have
> > better support than others.
> >
> > On Sun, Sep 13, 2015 at 11:46 AM, Hisham Mardam-Bey <hisham@mate1inc.com
> >
> > wrote:
> >
> > > Hi Li,
> > >
> > > You can take a look at mypipe.
> > >
> > > "MySQL binary log consumer with the ability to act on changed rows and
> > > publish changes to different systems with emphasis on Apache Kafka."
> > >
> > > https://github.com/mardambey/mypipe
> > >
> > > It uses Kafka as well as some related concepts and technologies like
> > Avro,
> > > a schema repository for data, and Akka.
> > >
> > > Hope this helps (= Happy hacking!
> > >
> > > hmb.
> > >
> > >
> > > On Sun, Sep 13, 2015 at 4:45 AM, Manasvi Gupta <fa...@gmail.com>
> > > wrote:
> > >
> > > > Here's a toy project - analyzing twitter stream.
> > > >
> > > > 1) Create dev. account on twitter
> > > > 2) Using your dev credentials, connect to twitter stream api to
> > retrieve
> > > > stream of tweets
> > > > 3) Store tweets in Kafka (using Kafka producer)
> > > > 4) Retrieve tweets (using Kafka consumer)
> > > > 5) For each tweet (or group of tweets), compute some analysis either
> > > using
> > > > custom java OR use storm/samza/spark. e.g. country of origin of
> tweet,
> > > > sentiment analysis etc.
> > > >
> > > > Its very simple to do this and should not take you more than 1-2 days
> > to
> > > > implement.
> > > >
> > > > Thanks
> > > > Manasvi
> > > >
> > > >
> > > > On Sun, Sep 13, 2015 at 1:11 PM, Li Tao <ah...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Roger,
> > > > >
> > > > > Thanks for your recommendation. I just got to know Samza. and
> checked
> > > its
> > > > > code base. It is a little too huge for me.
> > > > >
> > > > > Maybe for now, I need to start a small project/application which
> > > utilize
> > > > > kafka as its infrastructure, so that I can use Kafka's API a lot
> and
> > > know
> > > > > Kafka better.
> > > > >
> > > > > It's hard for me to initiate such project(small, useful/meaningful,
> > > kafka
> > > > > based). Anyone has better idea?
> > > > >
> > > > > On Sun, Sep 13, 2015 at 2:21 PM, Roger Hoover <
> > roger.hoover@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Li,
> > > > > >
> > > > > > You might take a look at Apache Samza. It's conceptually simple
> > but
> > > > > > powerful and makes heavy use of Kafka.
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Roger
> > > > > >
> > > > > > Sent from my iPhone
> > > > > >
> > > > > > > On Sep 12, 2015, at 10:34 PM, Li Tao <ah...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > Hi Hackers,
> > > > > > >
> > > > > > > This is Lee, a learner of kafka, i have read the original paper
> > on
> > > > > kafka,
> > > > > > > and walked through the document.
> > > > > > >
> > > > > > > I think the best way to learn sth is to write and read code
> about
> > > > it. I
> > > > > > am
> > > > > > > wondering is there any open source code / system which is based
> > on
> > > > > kafka
> > > > > > so
> > > > > > > that i can read or contribute to? Not too complex, not too
> > simple.
> > > > > > >
> > > > > > > Thanks a lot!
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Hisham Mardam-Bey
> > > -=[ CTO ]-=-[ Mate1 Inc. ]=-
> > >
> > > A: Because it messes up the order in which people normally read text.
> > > Q: Why is top-posting such a bad thing?
> > > A: Top-posting.
> > > Q: What is the most annoying thing in e-mail?
> > >
> > > -=[ Codito Ergo Sum ]=-
> > >
> >
>
--
Jeff Holoman
Systems Engineer
Re: open source projects based on kafka
Posted by Stevo Slavić <ss...@gmail.com>.
Have a look at https://github.com/allegro/hermes
On Mon, Sep 14, 2015, 01:28 David Luu <ma...@gmail.com> wrote:
> The toy project idea is good. Another option I think could be to look at
> the various Kafka client langague bindings and/or utilities (like
> kafkacat). And from there, another option is to build a client language
> binding for a language that's kind of lacking Kafka support, some have
> better support than others.
>
> On Sun, Sep 13, 2015 at 11:46 AM, Hisham Mardam-Bey <hi...@mate1inc.com>
> wrote:
>
> > Hi Li,
> >
> > You can take a look at mypipe.
> >
> > "MySQL binary log consumer with the ability to act on changed rows and
> > publish changes to different systems with emphasis on Apache Kafka."
> >
> > https://github.com/mardambey/mypipe
> >
> > It uses Kafka as well as some related concepts and technologies like
> Avro,
> > a schema repository for data, and Akka.
> >
> > Hope this helps (= Happy hacking!
> >
> > hmb.
> >
> >
> > On Sun, Sep 13, 2015 at 4:45 AM, Manasvi Gupta <fa...@gmail.com>
> > wrote:
> >
> > > Here's a toy project - analyzing twitter stream.
> > >
> > > 1) Create dev. account on twitter
> > > 2) Using your dev credentials, connect to twitter stream api to
> retrieve
> > > stream of tweets
> > > 3) Store tweets in Kafka (using Kafka producer)
> > > 4) Retrieve tweets (using Kafka consumer)
> > > 5) For each tweet (or group of tweets), compute some analysis either
> > using
> > > custom java OR use storm/samza/spark. e.g. country of origin of tweet,
> > > sentiment analysis etc.
> > >
> > > Its very simple to do this and should not take you more than 1-2 days
> to
> > > implement.
> > >
> > > Thanks
> > > Manasvi
> > >
> > >
> > > On Sun, Sep 13, 2015 at 1:11 PM, Li Tao <ah...@gmail.com>
> wrote:
> > >
> > > > Hi Roger,
> > > >
> > > > Thanks for your recommendation. I just got to know Samza. and checked
> > its
> > > > code base. It is a little too huge for me.
> > > >
> > > > Maybe for now, I need to start a small project/application which
> > utilize
> > > > kafka as its infrastructure, so that I can use Kafka's API a lot and
> > know
> > > > Kafka better.
> > > >
> > > > It's hard for me to initiate such project(small, useful/meaningful,
> > kafka
> > > > based). Anyone has better idea?
> > > >
> > > > On Sun, Sep 13, 2015 at 2:21 PM, Roger Hoover <
> roger.hoover@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Li,
> > > > >
> > > > > You might take a look at Apache Samza. It's conceptually simple
> but
> > > > > powerful and makes heavy use of Kafka.
> > > > >
> > > > > Best,
> > > > >
> > > > > Roger
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Sep 12, 2015, at 10:34 PM, Li Tao <ah...@gmail.com>
> > wrote:
> > > > > >
> > > > > > Hi Hackers,
> > > > > >
> > > > > > This is Lee, a learner of kafka, i have read the original paper
> on
> > > > kafka,
> > > > > > and walked through the document.
> > > > > >
> > > > > > I think the best way to learn sth is to write and read code about
> > > it. I
> > > > > am
> > > > > > wondering is there any open source code / system which is based
> on
> > > > kafka
> > > > > so
> > > > > > that i can read or contribute to? Not too complex, not too
> simple.
> > > > > >
> > > > > > Thanks a lot!
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Hisham Mardam-Bey
> > -=[ CTO ]-=-[ Mate1 Inc. ]=-
> >
> > A: Because it messes up the order in which people normally read text.
> > Q: Why is top-posting such a bad thing?
> > A: Top-posting.
> > Q: What is the most annoying thing in e-mail?
> >
> > -=[ Codito Ergo Sum ]=-
> >
>
Re: open source projects based on kafka
Posted by David Luu <ma...@gmail.com>.
The toy project idea is good. Another option I think could be to look at
the various Kafka client langague bindings and/or utilities (like
kafkacat). And from there, another option is to build a client language
binding for a language that's kind of lacking Kafka support, some have
better support than others.
On Sun, Sep 13, 2015 at 11:46 AM, Hisham Mardam-Bey <hi...@mate1inc.com>
wrote:
> Hi Li,
>
> You can take a look at mypipe.
>
> "MySQL binary log consumer with the ability to act on changed rows and
> publish changes to different systems with emphasis on Apache Kafka."
>
> https://github.com/mardambey/mypipe
>
> It uses Kafka as well as some related concepts and technologies like Avro,
> a schema repository for data, and Akka.
>
> Hope this helps (= Happy hacking!
>
> hmb.
>
>
> On Sun, Sep 13, 2015 at 4:45 AM, Manasvi Gupta <fa...@gmail.com>
> wrote:
>
> > Here's a toy project - analyzing twitter stream.
> >
> > 1) Create dev. account on twitter
> > 2) Using your dev credentials, connect to twitter stream api to retrieve
> > stream of tweets
> > 3) Store tweets in Kafka (using Kafka producer)
> > 4) Retrieve tweets (using Kafka consumer)
> > 5) For each tweet (or group of tweets), compute some analysis either
> using
> > custom java OR use storm/samza/spark. e.g. country of origin of tweet,
> > sentiment analysis etc.
> >
> > Its very simple to do this and should not take you more than 1-2 days to
> > implement.
> >
> > Thanks
> > Manasvi
> >
> >
> > On Sun, Sep 13, 2015 at 1:11 PM, Li Tao <ah...@gmail.com> wrote:
> >
> > > Hi Roger,
> > >
> > > Thanks for your recommendation. I just got to know Samza. and checked
> its
> > > code base. It is a little too huge for me.
> > >
> > > Maybe for now, I need to start a small project/application which
> utilize
> > > kafka as its infrastructure, so that I can use Kafka's API a lot and
> know
> > > Kafka better.
> > >
> > > It's hard for me to initiate such project(small, useful/meaningful,
> kafka
> > > based). Anyone has better idea?
> > >
> > > On Sun, Sep 13, 2015 at 2:21 PM, Roger Hoover <ro...@gmail.com>
> > > wrote:
> > >
> > > > Hi Li,
> > > >
> > > > You might take a look at Apache Samza. It's conceptually simple but
> > > > powerful and makes heavy use of Kafka.
> > > >
> > > > Best,
> > > >
> > > > Roger
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On Sep 12, 2015, at 10:34 PM, Li Tao <ah...@gmail.com>
> wrote:
> > > > >
> > > > > Hi Hackers,
> > > > >
> > > > > This is Lee, a learner of kafka, i have read the original paper on
> > > kafka,
> > > > > and walked through the document.
> > > > >
> > > > > I think the best way to learn sth is to write and read code about
> > it. I
> > > > am
> > > > > wondering is there any open source code / system which is based on
> > > kafka
> > > > so
> > > > > that i can read or contribute to? Not too complex, not too simple.
> > > > >
> > > > > Thanks a lot!
> > > >
> > >
> >
>
>
>
> --
> Hisham Mardam-Bey
> -=[ CTO ]-=-[ Mate1 Inc. ]=-
>
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing in e-mail?
>
> -=[ Codito Ergo Sum ]=-
>
Re: open source projects based on kafka
Posted by Hisham Mardam-Bey <hi...@mate1inc.com>.
Hi Li,
You can take a look at mypipe.
"MySQL binary log consumer with the ability to act on changed rows and
publish changes to different systems with emphasis on Apache Kafka."
https://github.com/mardambey/mypipe
It uses Kafka as well as some related concepts and technologies like Avro,
a schema repository for data, and Akka.
Hope this helps (= Happy hacking!
hmb.
On Sun, Sep 13, 2015 at 4:45 AM, Manasvi Gupta <fa...@gmail.com> wrote:
> Here's a toy project - analyzing twitter stream.
>
> 1) Create dev. account on twitter
> 2) Using your dev credentials, connect to twitter stream api to retrieve
> stream of tweets
> 3) Store tweets in Kafka (using Kafka producer)
> 4) Retrieve tweets (using Kafka consumer)
> 5) For each tweet (or group of tweets), compute some analysis either using
> custom java OR use storm/samza/spark. e.g. country of origin of tweet,
> sentiment analysis etc.
>
> Its very simple to do this and should not take you more than 1-2 days to
> implement.
>
> Thanks
> Manasvi
>
>
> On Sun, Sep 13, 2015 at 1:11 PM, Li Tao <ah...@gmail.com> wrote:
>
> > Hi Roger,
> >
> > Thanks for your recommendation. I just got to know Samza. and checked its
> > code base. It is a little too huge for me.
> >
> > Maybe for now, I need to start a small project/application which utilize
> > kafka as its infrastructure, so that I can use Kafka's API a lot and know
> > Kafka better.
> >
> > It's hard for me to initiate such project(small, useful/meaningful, kafka
> > based). Anyone has better idea?
> >
> > On Sun, Sep 13, 2015 at 2:21 PM, Roger Hoover <ro...@gmail.com>
> > wrote:
> >
> > > Hi Li,
> > >
> > > You might take a look at Apache Samza. It's conceptually simple but
> > > powerful and makes heavy use of Kafka.
> > >
> > > Best,
> > >
> > > Roger
> > >
> > > Sent from my iPhone
> > >
> > > > On Sep 12, 2015, at 10:34 PM, Li Tao <ah...@gmail.com> wrote:
> > > >
> > > > Hi Hackers,
> > > >
> > > > This is Lee, a learner of kafka, i have read the original paper on
> > kafka,
> > > > and walked through the document.
> > > >
> > > > I think the best way to learn sth is to write and read code about
> it. I
> > > am
> > > > wondering is there any open source code / system which is based on
> > kafka
> > > so
> > > > that i can read or contribute to? Not too complex, not too simple.
> > > >
> > > > Thanks a lot!
> > >
> >
>
--
Hisham Mardam-Bey
-=[ CTO ]-=-[ Mate1 Inc. ]=-
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?
-=[ Codito Ergo Sum ]=-
Re: open source projects based on kafka
Posted by Manasvi Gupta <fa...@gmail.com>.
Here's a toy project - analyzing twitter stream.
1) Create dev. account on twitter
2) Using your dev credentials, connect to twitter stream api to retrieve
stream of tweets
3) Store tweets in Kafka (using Kafka producer)
4) Retrieve tweets (using Kafka consumer)
5) For each tweet (or group of tweets), compute some analysis either using
custom java OR use storm/samza/spark. e.g. country of origin of tweet,
sentiment analysis etc.
Its very simple to do this and should not take you more than 1-2 days to
implement.
Thanks
Manasvi
On Sun, Sep 13, 2015 at 1:11 PM, Li Tao <ah...@gmail.com> wrote:
> Hi Roger,
>
> Thanks for your recommendation. I just got to know Samza. and checked its
> code base. It is a little too huge for me.
>
> Maybe for now, I need to start a small project/application which utilize
> kafka as its infrastructure, so that I can use Kafka's API a lot and know
> Kafka better.
>
> It's hard for me to initiate such project(small, useful/meaningful, kafka
> based). Anyone has better idea?
>
> On Sun, Sep 13, 2015 at 2:21 PM, Roger Hoover <ro...@gmail.com>
> wrote:
>
> > Hi Li,
> >
> > You might take a look at Apache Samza. It's conceptually simple but
> > powerful and makes heavy use of Kafka.
> >
> > Best,
> >
> > Roger
> >
> > Sent from my iPhone
> >
> > > On Sep 12, 2015, at 10:34 PM, Li Tao <ah...@gmail.com> wrote:
> > >
> > > Hi Hackers,
> > >
> > > This is Lee, a learner of kafka, i have read the original paper on
> kafka,
> > > and walked through the document.
> > >
> > > I think the best way to learn sth is to write and read code about it. I
> > am
> > > wondering is there any open source code / system which is based on
> kafka
> > so
> > > that i can read or contribute to? Not too complex, not too simple.
> > >
> > > Thanks a lot!
> >
>
Re: open source projects based on kafka
Posted by Li Tao <ah...@gmail.com>.
Hi Roger,
Thanks for your recommendation. I just got to know Samza. and checked its
code base. It is a little too huge for me.
Maybe for now, I need to start a small project/application which utilize
kafka as its infrastructure, so that I can use Kafka's API a lot and know
Kafka better.
It's hard for me to initiate such project(small, useful/meaningful, kafka
based). Anyone has better idea?
On Sun, Sep 13, 2015 at 2:21 PM, Roger Hoover <ro...@gmail.com>
wrote:
> Hi Li,
>
> You might take a look at Apache Samza. It's conceptually simple but
> powerful and makes heavy use of Kafka.
>
> Best,
>
> Roger
>
> Sent from my iPhone
>
> > On Sep 12, 2015, at 10:34 PM, Li Tao <ah...@gmail.com> wrote:
> >
> > Hi Hackers,
> >
> > This is Lee, a learner of kafka, i have read the original paper on kafka,
> > and walked through the document.
> >
> > I think the best way to learn sth is to write and read code about it. I
> am
> > wondering is there any open source code / system which is based on kafka
> so
> > that i can read or contribute to? Not too complex, not too simple.
> >
> > Thanks a lot!
>
Re: open source projects based on kafka
Posted by Roger Hoover <ro...@gmail.com>.
Hi Li,
You might take a look at Apache Samza. It's conceptually simple but powerful and makes heavy use of Kafka.
Best,
Roger
Sent from my iPhone
> On Sep 12, 2015, at 10:34 PM, Li Tao <ah...@gmail.com> wrote:
>
> Hi Hackers,
>
> This is Lee, a learner of kafka, i have read the original paper on kafka,
> and walked through the document.
>
> I think the best way to learn sth is to write and read code about it. I am
> wondering is there any open source code / system which is based on kafka so
> that i can read or contribute to? Not too complex, not too simple.
>
> Thanks a lot!