Re: C/C++ Client

[anand nalya <a....@computer.org>]

Hi Joel,

Any updates on the c++ producer?

Thanks,
Anand

On 3 April 2013 05:59, Joel Koshy <jj...@gmail.com> wrote:

> Yes - we would be interested in doing that. I have been spending most of my
> time over the past couple weeks on the C++ library (currently, only for the
> producer). It is reasonably stable, although it has not been tried and
> tested in production.
>
> I can start with publishing a wiki describing the design/implementation and
> list out future work that others can contribute to if there is interest,
> but I'll quickly summarize some of its goals here:
> - Use non-blocking I/O for sending requests/receiving responses. This
> mitigates the request-response RTT latency that the 0.8 protocol introduces
> (if acks != 0).
> - C-compliant API, so that other languages e.g., Ruby/Python can wrap the
> library. I understand there are people working on 0.8 clients in those
> languages so this is just an alternative approach.
> - Non-blocking variants for all operations in the API: there are some
> use-cases where blocking in the user-thread is unacceptable.
> - Keep third-party libraries to a minimum to make porting to other
> platforms easier.
>
> There are two options in pursuing this:
> 1 - Contribute it as part of the Kafka project. i.e., re-introduce the
> clients directory. We removed it because none of the committers were
> original authors of any of the non-Java clients and did not have the
> bandwidth/background to review patches for those clients.
> 2 - Maintain it externally.
> I was going to also suggest "sub-project" as a third option, but I heard
> from Jakob that it is no longer encouraged by the board and a client by
> itself would not be a substantial sub-project.
>
> If we want to do (1), we would obviously go through the standard
> contribution process: i.e., create a jira for it and provide a patch that
> people can review; and other implementations will also be considered if
> contributors provide patches.
>
> (2) is also fine, although one benefit with (1) is that it would be
> reasonable to use the existing infrastructure (mailing lists, issue
> tracking) for the Kafka project.
>
> Any preferences/thoughts?
>
> Thanks,
>
> Joel
>
> On Mon, Apr 1, 2013 at 8:22 AM, mrevilgnome <mr...@gmail.com> wrote:
>
> > Any interest in open sourcing it now and picking up contributors?
> >
> >
> > On Mon, Apr 1, 2013 at 7:54 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > At LinkedIn, we are also building a native C producer client for 0.8.
> It
> > > uses non-blocking socket I/O to improve the producer throughput. We
> plan
> > to
> > > open source it when it's fully tested, hopefully in a couple of months.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Mar 27, 2013 at 8:48 PM, Matthew Stump <
> mstump@matthewstump.com
> > > >wrote:
> > >
> > > > Howdy,
> > > >
> > > > I'm considering the use of Kafka in the rewrite of a big legacy
> > product.
> > > A
> > > > good chunk of the back end code is going to be written in C++ (large
> in
> > > > memory data-structures). The two possible options available to me for
> > > > clients appear to be:
> > > >
> > > > https://github.com/edenhill/librdkafka
> > > >
> > > > and
> > > >
> > > > https://github.com/quipo/kafka-cpp
> > > >
> > > > The problem is that librdkafka currently only works on Linux due to
> the
> > > use
> > > > of the AF_NETLINK API, and thread local storage. There may be other
> > > issues,
> > > > but I just started playing with it today and that's what I've
> > discovered
> > > > thus far.
> > > >
> > > > kafka-cpp is incomplete (no consumer) and it looks unused.
> > > >
> > > > For either I would need to hop in and do some significant work. Is
> > there
> > > > any client I'm missing that can shorten my path?
> > > >
> > > > If I adopt one of these projects (lets say kafka-cpp) am I better off
> > > > implementing the 0.8 protocol? I'de like to have something running in
> > > > staging a couple months from now. How far out is 0.8?
> > > >
> > > > Thanks,
> > > > Matt Stump
> > > >
> > >
> >
>