You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Otis Gospodnetic <ot...@gmail.com> on 2015/01/20 20:52:16 UTC
Poll: Producer/Consumer impl/language you use?
Hi,
I was wondering which implementations/languages people use for their Kafka
Producer/Consumers.... not everyone is using the Java APIs. So here's a
1-question poll:
http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/
Will share the results in about a week when we have enough votes.
Thanks!
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
Re: Poll: Producer/Consumer impl/language you use?
Posted by Jay Kreps <ja...@gmail.com>.
Yeah Joe is exactly right.
Let's not confuse scala apis with the existing Scala clients There are a
ton of downsides to those clients. They aren't going away any time in the
forceable future, so don't stress, but I think we can kind of "deprecate"
them and try to shame people into upgrading.
For the Java producer I actually think that API is no worse in Scala than
the existing Scala api, so I'm not sure if there is much we can do to
improve it for scala, but if there is there would be no harm in adding a
scala wrapper.
For the new Java consumer there are some Java-isms in the client (e.g. a
couple methods take java maps as arguments). I actually think the existing
scala implicit conversions for java collections might be totally sufficient
but someone would have to try. If not we can add a scala wrapper.
I actually think there is room for lots of wrappers that experiment with
different styles of data access, especially for the consumer. The reactive
people have a bunch of stream related things, Joe mentioned scalaz streams,
etc. Actually the iterator api in the existing consumer was an attempt to
make stream processing easy (since you can apply some of the scala
collections things to the resulting iterator) but I think it wasn't thought
all the way through. I think having this simpler, more flexible base api
will let people experiment with this stuff in any way they want.
-Jay
On Wed, Jan 28, 2015 at 10:45 AM, Joe Stein <jo...@stealth.ly> wrote:
> I kind of look at the Storm, Spark, Samza, etc integrations as
> producers/consumers too.
>
> Not sure if that maybe was getting lumped also into other.
>
> I think Jason's 90/10 80/20 70/30 would be found to be typical.
>
> As far as the Scala API goes, I think we should have a wrapper around the
> shiny new Java Consumer. Folks I know use things like scalaz streams
> https://github.com/scalaz/scalaz-stream which the new consumer can work
> nicely with I think. It would be great if we could come up with a new Scala
> layer on top of the Java consumer that we release in the project. One of my
> engineers is taking a look at that now unless someone is already working on
> that? We are using the partition static assignment in the new consumer and
> just using Mesos for handling re-balance for us. When he gets further along
> and if it makes sense we will shoot a KIP around people can chat about it
> on dev.
>
> - Joe Stein
>
> On Wed, Jan 28, 2015 at 10:33 AM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> > Good point, Jason. Not sure how we could account for that easily. But
> > maybe that is at least a partial explanation of the Java % being under
> 50%
> > when Java in general is more popular than that...
> >
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Wed, Jan 28, 2015 at 1:22 PM, Jason Rosenberg <jb...@squareup.com>
> wrote:
> >
> > > I think the results could be a bit skewed, in cases where an
> organization
> > > uses multiple languages, but not equally. In our case, we
> overwhelmingly
> > > use java clients (>90%). But we also have ruby and Go clients too.
> But
> > in
> > > the poll, these come out as equally used client languages.
> > >
> > > Jason
> > >
> > > On Wed, Jan 28, 2015 at 12:05 PM, David McNelis <
> > > dmcnelis@emergingthreats.net> wrote:
> > >
> > > > I agree with Stephen, it would be really unfortunate to see the Scala
> > api
> > > > go away.
> > > >
> > > > On Wed, Jan 28, 2015 at 11:57 AM, Stephen Boesch <ja...@gmail.com>
> > > > wrote:
> > > >
> > > > > The scala API going away would be a minus. As Koert mentioned we
> > could
> > > > use
> > > > > the java api but it is less .. well .. functional.
> > > > >
> > > > > Kafka is included in the Spark examples and external modules and is
> > > > popular
> > > > > as a component of ecosystems on Spark (for which scala is the
> primary
> > > > > language).
> > > > >
> > > > > 2015-01-28 8:51 GMT-08:00 Otis Gospodnetic <
> > otis.gospodnetic@gmail.com
> > > >:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I don't have a good excuse here. :(
> > > > > > I thought about including Scala, but for some reason didn't do
> > it. I
> > > > see
> > > > > > 12-13% of people chose "Other". Do you think that is because I
> > > didn't
> > > > > > include Scala?
> > > > > >
> > > > > > Also, is the Scala API reeeeeeally going away?
> > > > > >
> > > > > > Otis
> > > > > > --
> > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > > Management
> > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 20, 2015 at 4:43 PM, Koert Kuipers <
> koert@tresata.com>
> > > > > wrote:
> > > > > >
> > > > > > > no scala? although scala can indeed use the java api, its
> > ugly....
> > > we
> > > > > > > prefer to use the scala api (which i believe will go away
> > > > > unfortunately)
> > > > > > >
> > > > > > > On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic <
> > > > > > > otis.gospodnetic@gmail.com> wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I was wondering which implementations/languages people use
> for
> > > > their
> > > > > > > Kafka
> > > > > > > > Producer/Consumers.... not everyone is using the Java APIs.
> So
> > > > > here's
> > > > > > a
> > > > > > > > 1-question poll:
> > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/
> > > > > > > >
> > > > > > > > Will share the results in about a week when we have enough
> > votes.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > Otis
> > > > > > > > --
> > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > > > > Management
> > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Poll: Producer/Consumer impl/language you use?
Posted by Joe Stein <jo...@stealth.ly>.
I kind of look at the Storm, Spark, Samza, etc integrations as
producers/consumers too.
Not sure if that maybe was getting lumped also into other.
I think Jason's 90/10 80/20 70/30 would be found to be typical.
As far as the Scala API goes, I think we should have a wrapper around the
shiny new Java Consumer. Folks I know use things like scalaz streams
https://github.com/scalaz/scalaz-stream which the new consumer can work
nicely with I think. It would be great if we could come up with a new Scala
layer on top of the Java consumer that we release in the project. One of my
engineers is taking a look at that now unless someone is already working on
that? We are using the partition static assignment in the new consumer and
just using Mesos for handling re-balance for us. When he gets further along
and if it makes sense we will shoot a KIP around people can chat about it
on dev.
- Joe Stein
On Wed, Jan 28, 2015 at 10:33 AM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Good point, Jason. Not sure how we could account for that easily. But
> maybe that is at least a partial explanation of the Java % being under 50%
> when Java in general is more popular than that...
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Wed, Jan 28, 2015 at 1:22 PM, Jason Rosenberg <jb...@squareup.com> wrote:
>
> > I think the results could be a bit skewed, in cases where an organization
> > uses multiple languages, but not equally. In our case, we overwhelmingly
> > use java clients (>90%). But we also have ruby and Go clients too. But
> in
> > the poll, these come out as equally used client languages.
> >
> > Jason
> >
> > On Wed, Jan 28, 2015 at 12:05 PM, David McNelis <
> > dmcnelis@emergingthreats.net> wrote:
> >
> > > I agree with Stephen, it would be really unfortunate to see the Scala
> api
> > > go away.
> > >
> > > On Wed, Jan 28, 2015 at 11:57 AM, Stephen Boesch <ja...@gmail.com>
> > > wrote:
> > >
> > > > The scala API going away would be a minus. As Koert mentioned we
> could
> > > use
> > > > the java api but it is less .. well .. functional.
> > > >
> > > > Kafka is included in the Spark examples and external modules and is
> > > popular
> > > > as a component of ecosystems on Spark (for which scala is the primary
> > > > language).
> > > >
> > > > 2015-01-28 8:51 GMT-08:00 Otis Gospodnetic <
> otis.gospodnetic@gmail.com
> > >:
> > > >
> > > > > Hi,
> > > > >
> > > > > I don't have a good excuse here. :(
> > > > > I thought about including Scala, but for some reason didn't do
> it. I
> > > see
> > > > > 12-13% of people chose "Other". Do you think that is because I
> > didn't
> > > > > include Scala?
> > > > >
> > > > > Also, is the Scala API reeeeeeally going away?
> > > > >
> > > > > Otis
> > > > > --
> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > Management
> > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > >
> > > > >
> > > > > On Tue, Jan 20, 2015 at 4:43 PM, Koert Kuipers <ko...@tresata.com>
> > > > wrote:
> > > > >
> > > > > > no scala? although scala can indeed use the java api, its
> ugly....
> > we
> > > > > > prefer to use the scala api (which i believe will go away
> > > > unfortunately)
> > > > > >
> > > > > > On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic <
> > > > > > otis.gospodnetic@gmail.com> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I was wondering which implementations/languages people use for
> > > their
> > > > > > Kafka
> > > > > > > Producer/Consumers.... not everyone is using the Java APIs. So
> > > > here's
> > > > > a
> > > > > > > 1-question poll:
> > > > > > >
> > > > > > >
> > > > >
> > >
> http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/
> > > > > > >
> > > > > > > Will share the results in about a week when we have enough
> votes.
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Otis
> > > > > > > --
> > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > > > Management
> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Poll: Producer/Consumer impl/language you use?
Posted by Otis Gospodnetic <ot...@gmail.com>.
Good point, Jason. Not sure how we could account for that easily. But
maybe that is at least a partial explanation of the Java % being under 50%
when Java in general is more popular than that...
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Wed, Jan 28, 2015 at 1:22 PM, Jason Rosenberg <jb...@squareup.com> wrote:
> I think the results could be a bit skewed, in cases where an organization
> uses multiple languages, but not equally. In our case, we overwhelmingly
> use java clients (>90%). But we also have ruby and Go clients too. But in
> the poll, these come out as equally used client languages.
>
> Jason
>
> On Wed, Jan 28, 2015 at 12:05 PM, David McNelis <
> dmcnelis@emergingthreats.net> wrote:
>
> > I agree with Stephen, it would be really unfortunate to see the Scala api
> > go away.
> >
> > On Wed, Jan 28, 2015 at 11:57 AM, Stephen Boesch <ja...@gmail.com>
> > wrote:
> >
> > > The scala API going away would be a minus. As Koert mentioned we could
> > use
> > > the java api but it is less .. well .. functional.
> > >
> > > Kafka is included in the Spark examples and external modules and is
> > popular
> > > as a component of ecosystems on Spark (for which scala is the primary
> > > language).
> > >
> > > 2015-01-28 8:51 GMT-08:00 Otis Gospodnetic <otis.gospodnetic@gmail.com
> >:
> > >
> > > > Hi,
> > > >
> > > > I don't have a good excuse here. :(
> > > > I thought about including Scala, but for some reason didn't do it. I
> > see
> > > > 12-13% of people chose "Other". Do you think that is because I
> didn't
> > > > include Scala?
> > > >
> > > > Also, is the Scala API reeeeeeally going away?
> > > >
> > > > Otis
> > > > --
> > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> > > > Solr & Elasticsearch Support * http://sematext.com/
> > > >
> > > >
> > > > On Tue, Jan 20, 2015 at 4:43 PM, Koert Kuipers <ko...@tresata.com>
> > > wrote:
> > > >
> > > > > no scala? although scala can indeed use the java api, its ugly....
> we
> > > > > prefer to use the scala api (which i believe will go away
> > > unfortunately)
> > > > >
> > > > > On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic <
> > > > > otis.gospodnetic@gmail.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I was wondering which implementations/languages people use for
> > their
> > > > > Kafka
> > > > > > Producer/Consumers.... not everyone is using the Java APIs. So
> > > here's
> > > > a
> > > > > > 1-question poll:
> > > > > >
> > > > > >
> > > >
> > http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/
> > > > > >
> > > > > > Will share the results in about a week when we have enough votes.
> > > > > >
> > > > > > Thanks!
> > > > > > Otis
> > > > > > --
> > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > > Management
> > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Poll: Producer/Consumer impl/language you use?
Posted by Jason Rosenberg <jb...@squareup.com>.
I think the results could be a bit skewed, in cases where an organization
uses multiple languages, but not equally. In our case, we overwhelmingly
use java clients (>90%). But we also have ruby and Go clients too. But in
the poll, these come out as equally used client languages.
Jason
On Wed, Jan 28, 2015 at 12:05 PM, David McNelis <
dmcnelis@emergingthreats.net> wrote:
> I agree with Stephen, it would be really unfortunate to see the Scala api
> go away.
>
> On Wed, Jan 28, 2015 at 11:57 AM, Stephen Boesch <ja...@gmail.com>
> wrote:
>
> > The scala API going away would be a minus. As Koert mentioned we could
> use
> > the java api but it is less .. well .. functional.
> >
> > Kafka is included in the Spark examples and external modules and is
> popular
> > as a component of ecosystems on Spark (for which scala is the primary
> > language).
> >
> > 2015-01-28 8:51 GMT-08:00 Otis Gospodnetic <ot...@gmail.com>:
> >
> > > Hi,
> > >
> > > I don't have a good excuse here. :(
> > > I thought about including Scala, but for some reason didn't do it. I
> see
> > > 12-13% of people chose "Other". Do you think that is because I didn't
> > > include Scala?
> > >
> > > Also, is the Scala API reeeeeeally going away?
> > >
> > > Otis
> > > --
> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> > > On Tue, Jan 20, 2015 at 4:43 PM, Koert Kuipers <ko...@tresata.com>
> > wrote:
> > >
> > > > no scala? although scala can indeed use the java api, its ugly.... we
> > > > prefer to use the scala api (which i believe will go away
> > unfortunately)
> > > >
> > > > On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic <
> > > > otis.gospodnetic@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I was wondering which implementations/languages people use for
> their
> > > > Kafka
> > > > > Producer/Consumers.... not everyone is using the Java APIs. So
> > here's
> > > a
> > > > > 1-question poll:
> > > > >
> > > > >
> > >
> http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/
> > > > >
> > > > > Will share the results in about a week when we have enough votes.
> > > > >
> > > > > Thanks!
> > > > > Otis
> > > > > --
> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > Management
> > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > >
> > > >
> > >
> >
>
Re: Poll: Producer/Consumer impl/language you use?
Posted by David McNelis <dm...@emergingthreats.net>.
I agree with Stephen, it would be really unfortunate to see the Scala api
go away.
On Wed, Jan 28, 2015 at 11:57 AM, Stephen Boesch <ja...@gmail.com> wrote:
> The scala API going away would be a minus. As Koert mentioned we could use
> the java api but it is less .. well .. functional.
>
> Kafka is included in the Spark examples and external modules and is popular
> as a component of ecosystems on Spark (for which scala is the primary
> language).
>
> 2015-01-28 8:51 GMT-08:00 Otis Gospodnetic <ot...@gmail.com>:
>
> > Hi,
> >
> > I don't have a good excuse here. :(
> > I thought about including Scala, but for some reason didn't do it. I see
> > 12-13% of people chose "Other". Do you think that is because I didn't
> > include Scala?
> >
> > Also, is the Scala API reeeeeeally going away?
> >
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Tue, Jan 20, 2015 at 4:43 PM, Koert Kuipers <ko...@tresata.com>
> wrote:
> >
> > > no scala? although scala can indeed use the java api, its ugly.... we
> > > prefer to use the scala api (which i believe will go away
> unfortunately)
> > >
> > > On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic <
> > > otis.gospodnetic@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I was wondering which implementations/languages people use for their
> > > Kafka
> > > > Producer/Consumers.... not everyone is using the Java APIs. So
> here's
> > a
> > > > 1-question poll:
> > > >
> > > >
> > http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/
> > > >
> > > > Will share the results in about a week when we have enough votes.
> > > >
> > > > Thanks!
> > > > Otis
> > > > --
> > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> > > > Solr & Elasticsearch Support * http://sematext.com/
> > > >
> > >
> >
>
Re: Poll: Producer/Consumer impl/language you use?
Posted by Stephen Boesch <ja...@gmail.com>.
The scala API going away would be a minus. As Koert mentioned we could use
the java api but it is less .. well .. functional.
Kafka is included in the Spark examples and external modules and is popular
as a component of ecosystems on Spark (for which scala is the primary
language).
2015-01-28 8:51 GMT-08:00 Otis Gospodnetic <ot...@gmail.com>:
> Hi,
>
> I don't have a good excuse here. :(
> I thought about including Scala, but for some reason didn't do it. I see
> 12-13% of people chose "Other". Do you think that is because I didn't
> include Scala?
>
> Also, is the Scala API reeeeeeally going away?
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Tue, Jan 20, 2015 at 4:43 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
> > no scala? although scala can indeed use the java api, its ugly.... we
> > prefer to use the scala api (which i believe will go away unfortunately)
> >
> > On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I was wondering which implementations/languages people use for their
> > Kafka
> > > Producer/Consumers.... not everyone is using the Java APIs. So here's
> a
> > > 1-question poll:
> > >
> > >
> http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/
> > >
> > > Will share the results in about a week when we have enough votes.
> > >
> > > Thanks!
> > > Otis
> > > --
> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> >
>
Re: Poll: Producer/Consumer impl/language you use?
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
I don't have a good excuse here. :(
I thought about including Scala, but for some reason didn't do it. I see
12-13% of people chose "Other". Do you think that is because I didn't
include Scala?
Also, is the Scala API reeeeeeally going away?
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Tue, Jan 20, 2015 at 4:43 PM, Koert Kuipers <ko...@tresata.com> wrote:
> no scala? although scala can indeed use the java api, its ugly.... we
> prefer to use the scala api (which i believe will go away unfortunately)
>
> On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> > Hi,
> >
> > I was wondering which implementations/languages people use for their
> Kafka
> > Producer/Consumers.... not everyone is using the Java APIs. So here's a
> > 1-question poll:
> >
> > http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/
> >
> > Will share the results in about a week when we have enough votes.
> >
> > Thanks!
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
>
Re: Poll: Producer/Consumer impl/language you use?
Posted by Koert Kuipers <ko...@tresata.com>.
no scala? although scala can indeed use the java api, its ugly.... we
prefer to use the scala api (which i believe will go away unfortunately)
On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Hi,
>
> I was wondering which implementations/languages people use for their Kafka
> Producer/Consumers.... not everyone is using the Java APIs. So here's a
> 1-question poll:
>
> http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/
>
> Will share the results in about a week when we have enough votes.
>
> Thanks!
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>