You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mark <st...@gmail.com> on 2013/08/29 22:39:36 UTC
What are my options? (Ruby/Rails environment)
We are thinking about using Kafka to collect events from our Rails application and I was hoping to get some input from the Kafka community.
Currently the only gems available are:
https://github.com/acrosa/kafka-rb
https://github.com/bpot/poseidon (Can't use since we are only running 1.8.7)
Now neither of these integrate with Zookeeper so we are missing quite a few features:
- Auto discovery of brokers
- Auto discover of partitions for consumers
- … fill in the rest here, new to Kafka so don't know everything that is missing
I was wondering what my best options are going forward with Kakfa? I think we have the following choices
A) Instead of writing directly to Kafka from our application we can write our events/messages to some other source (Syslog, File, ?) and then have a separate Java process that reads these sources and writes to Kafka. This is a little annoying since we now have to worry about every machine also running the above separate process to write to Kafka.
B) Work around the above limitations. Auto-discover of brokers is terrible since we don't foresee us adding/removing brokers that frequently. The lack of auto-discover of partitions is definitely a loss since we now have to know which broker/partition to read from at all times. Of course we can just write to Kafka using the above Gems and have our consumers written in another language.
C)?
Any thoughts/opinions?
Thanks
- M
Re: What are my options? (Ruby/Rails environment)
Posted by Jun Rao <ju...@gmail.com>.
In 0.8, the producer no longer depends on ZK. It only takes a list of
brokers. At LinkedIn, we have a 0.8 C producer implementation and plan to
open source it soon.
Thanks,
Jun
On Fri, Aug 30, 2013 at 9:00 AM, Travis Brady <tr...@gmail.com>wrote:
> I think this points out the need for a single canonical cross-platform C
> client lib with support for Zookeeper that could easily wrapped for use in
> other languages.
>
> It would make it much easier for people using Python, Ruby, Node, Lua,
> Haskell, Go, OCaml, etc to have such a library that matches the features of
> the Scala/Java client.
> Currently writing a client amounts to re-implementing the entire Scala
> client from scratch, which is why so many clients settle for simple
> producer/consumer support and skip Zk altogether.
>
>
>
>
> On Thu, Aug 29, 2013 at 10:44 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > I assume this for Kafka 0.7. One option is to use a VIP in front of the
> > brokers for load balancing.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Aug 29, 2013 at 1:39 PM, Mark <st...@gmail.com> wrote:
> >
> > > We are thinking about using Kafka to collect events from our Rails
> > > application and I was hoping to get some input from the Kafka
> community.
> > >
> > > Currently the only gems available are:
> > >
> > > https://github.com/acrosa/kafka-rb
> > > https://github.com/bpot/poseidon (Can't use since we are only running
> > > 1.8.7)
> > >
> > > Now neither of these integrate with Zookeeper so we are missing quite a
> > > few features:
> > >
> > > - Auto discovery of brokers
> > > - Auto discover of partitions for consumers
> > > - … fill in the rest here, new to Kafka so don't know everything that
> is
> > > missing
> > >
> > > I was wondering what my best options are going forward with Kakfa? I
> > think
> > > we have the following choices
> > >
> > > A) Instead of writing directly to Kafka from our application we can
> write
> > > our events/messages to some other source (Syslog, File, ?) and then
> have
> > a
> > > separate Java process that reads these sources and writes to Kafka.
> This
> > is
> > > a little annoying since we now have to worry about every machine also
> > > running the above separate process to write to Kafka.
> > >
> > > B) Work around the above limitations. Auto-discover of brokers is
> > terrible
> > > since we don't foresee us adding/removing brokers that frequently. The
> > lack
> > > of auto-discover of partitions is definitely a loss since we now have
> to
> > > know which broker/partition to read from at all times. Of course we can
> > > just write to Kafka using the above Gems and have our consumers written
> > in
> > > another language.
> > >
> > > C)?
> > >
> > > Any thoughts/opinions?
> > >
> > > Thanks
> > >
> > > - M
> > >
> > >
> >
>
Re: What are my options? (Ruby/Rails environment)
Posted by Travis Brady <tr...@gmail.com>.
I think this points out the need for a single canonical cross-platform C
client lib with support for Zookeeper that could easily wrapped for use in
other languages.
It would make it much easier for people using Python, Ruby, Node, Lua,
Haskell, Go, OCaml, etc to have such a library that matches the features of
the Scala/Java client.
Currently writing a client amounts to re-implementing the entire Scala
client from scratch, which is why so many clients settle for simple
producer/consumer support and skip Zk altogether.
On Thu, Aug 29, 2013 at 10:44 PM, Jun Rao <ju...@gmail.com> wrote:
> I assume this for Kafka 0.7. One option is to use a VIP in front of the
> brokers for load balancing.
>
> Thanks,
>
> Jun
>
>
> On Thu, Aug 29, 2013 at 1:39 PM, Mark <st...@gmail.com> wrote:
>
> > We are thinking about using Kafka to collect events from our Rails
> > application and I was hoping to get some input from the Kafka community.
> >
> > Currently the only gems available are:
> >
> > https://github.com/acrosa/kafka-rb
> > https://github.com/bpot/poseidon (Can't use since we are only running
> > 1.8.7)
> >
> > Now neither of these integrate with Zookeeper so we are missing quite a
> > few features:
> >
> > - Auto discovery of brokers
> > - Auto discover of partitions for consumers
> > - … fill in the rest here, new to Kafka so don't know everything that is
> > missing
> >
> > I was wondering what my best options are going forward with Kakfa? I
> think
> > we have the following choices
> >
> > A) Instead of writing directly to Kafka from our application we can write
> > our events/messages to some other source (Syslog, File, ?) and then have
> a
> > separate Java process that reads these sources and writes to Kafka. This
> is
> > a little annoying since we now have to worry about every machine also
> > running the above separate process to write to Kafka.
> >
> > B) Work around the above limitations. Auto-discover of brokers is
> terrible
> > since we don't foresee us adding/removing brokers that frequently. The
> lack
> > of auto-discover of partitions is definitely a loss since we now have to
> > know which broker/partition to read from at all times. Of course we can
> > just write to Kafka using the above Gems and have our consumers written
> in
> > another language.
> >
> > C)?
> >
> > Any thoughts/opinions?
> >
> > Thanks
> >
> > - M
> >
> >
>
Re: What are my options? (Ruby/Rails environment)
Posted by Jun Rao <ju...@gmail.com>.
I assume this for Kafka 0.7. One option is to use a VIP in front of the
brokers for load balancing.
Thanks,
Jun
On Thu, Aug 29, 2013 at 1:39 PM, Mark <st...@gmail.com> wrote:
> We are thinking about using Kafka to collect events from our Rails
> application and I was hoping to get some input from the Kafka community.
>
> Currently the only gems available are:
>
> https://github.com/acrosa/kafka-rb
> https://github.com/bpot/poseidon (Can't use since we are only running
> 1.8.7)
>
> Now neither of these integrate with Zookeeper so we are missing quite a
> few features:
>
> - Auto discovery of brokers
> - Auto discover of partitions for consumers
> - … fill in the rest here, new to Kafka so don't know everything that is
> missing
>
> I was wondering what my best options are going forward with Kakfa? I think
> we have the following choices
>
> A) Instead of writing directly to Kafka from our application we can write
> our events/messages to some other source (Syslog, File, ?) and then have a
> separate Java process that reads these sources and writes to Kafka. This is
> a little annoying since we now have to worry about every machine also
> running the above separate process to write to Kafka.
>
> B) Work around the above limitations. Auto-discover of brokers is terrible
> since we don't foresee us adding/removing brokers that frequently. The lack
> of auto-discover of partitions is definitely a loss since we now have to
> know which broker/partition to read from at all times. Of course we can
> just write to Kafka using the above Gems and have our consumers written in
> another language.
>
> C)?
>
> Any thoughts/opinions?
>
> Thanks
>
> - M
>
>