You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jun Rao <ju...@gmail.com> on 2013/04/01 16:54:02 UTC
Re: C/C++ Client
At LinkedIn, we are also building a native C producer client for 0.8. It
uses non-blocking socket I/O to improve the producer throughput. We plan to
open source it when it's fully tested, hopefully in a couple of months.
Thanks,
Jun
On Wed, Mar 27, 2013 at 8:48 PM, Matthew Stump <ms...@matthewstump.com>wrote:
> Howdy,
>
> I'm considering the use of Kafka in the rewrite of a big legacy product. A
> good chunk of the back end code is going to be written in C++ (large in
> memory data-structures). The two possible options available to me for
> clients appear to be:
>
> https://github.com/edenhill/librdkafka
>
> and
>
> https://github.com/quipo/kafka-cpp
>
> The problem is that librdkafka currently only works on Linux due to the use
> of the AF_NETLINK API, and thread local storage. There may be other issues,
> but I just started playing with it today and that's what I've discovered
> thus far.
>
> kafka-cpp is incomplete (no consumer) and it looks unused.
>
> For either I would need to hop in and do some significant work. Is there
> any client I'm missing that can shorten my path?
>
> If I adopt one of these projects (lets say kafka-cpp) am I better off
> implementing the 0.8 protocol? I'de like to have something running in
> staging a couple months from now. How far out is 0.8?
>
> Thanks,
> Matt Stump
>
Re: C/C++ Client
Posted by anand nalya <a....@computer.org>.
Hi Joel,
Any updates on the c++ producer?
Thanks,
Anand
On 3 April 2013 05:59, Joel Koshy <jj...@gmail.com> wrote:
> Yes - we would be interested in doing that. I have been spending most of my
> time over the past couple weeks on the C++ library (currently, only for the
> producer). It is reasonably stable, although it has not been tried and
> tested in production.
>
> I can start with publishing a wiki describing the design/implementation and
> list out future work that others can contribute to if there is interest,
> but I'll quickly summarize some of its goals here:
> - Use non-blocking I/O for sending requests/receiving responses. This
> mitigates the request-response RTT latency that the 0.8 protocol introduces
> (if acks != 0).
> - C-compliant API, so that other languages e.g., Ruby/Python can wrap the
> library. I understand there are people working on 0.8 clients in those
> languages so this is just an alternative approach.
> - Non-blocking variants for all operations in the API: there are some
> use-cases where blocking in the user-thread is unacceptable.
> - Keep third-party libraries to a minimum to make porting to other
> platforms easier.
>
> There are two options in pursuing this:
> 1 - Contribute it as part of the Kafka project. i.e., re-introduce the
> clients directory. We removed it because none of the committers were
> original authors of any of the non-Java clients and did not have the
> bandwidth/background to review patches for those clients.
> 2 - Maintain it externally.
> I was going to also suggest "sub-project" as a third option, but I heard
> from Jakob that it is no longer encouraged by the board and a client by
> itself would not be a substantial sub-project.
>
> If we want to do (1), we would obviously go through the standard
> contribution process: i.e., create a jira for it and provide a patch that
> people can review; and other implementations will also be considered if
> contributors provide patches.
>
> (2) is also fine, although one benefit with (1) is that it would be
> reasonable to use the existing infrastructure (mailing lists, issue
> tracking) for the Kafka project.
>
> Any preferences/thoughts?
>
> Thanks,
>
> Joel
>
> On Mon, Apr 1, 2013 at 8:22 AM, mrevilgnome <mr...@gmail.com> wrote:
>
> > Any interest in open sourcing it now and picking up contributors?
> >
> >
> > On Mon, Apr 1, 2013 at 7:54 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > At LinkedIn, we are also building a native C producer client for 0.8.
> It
> > > uses non-blocking socket I/O to improve the producer throughput. We
> plan
> > to
> > > open source it when it's fully tested, hopefully in a couple of months.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Mar 27, 2013 at 8:48 PM, Matthew Stump <
> mstump@matthewstump.com
> > > >wrote:
> > >
> > > > Howdy,
> > > >
> > > > I'm considering the use of Kafka in the rewrite of a big legacy
> > product.
> > > A
> > > > good chunk of the back end code is going to be written in C++ (large
> in
> > > > memory data-structures). The two possible options available to me for
> > > > clients appear to be:
> > > >
> > > > https://github.com/edenhill/librdkafka
> > > >
> > > > and
> > > >
> > > > https://github.com/quipo/kafka-cpp
> > > >
> > > > The problem is that librdkafka currently only works on Linux due to
> the
> > > use
> > > > of the AF_NETLINK API, and thread local storage. There may be other
> > > issues,
> > > > but I just started playing with it today and that's what I've
> > discovered
> > > > thus far.
> > > >
> > > > kafka-cpp is incomplete (no consumer) and it looks unused.
> > > >
> > > > For either I would need to hop in and do some significant work. Is
> > there
> > > > any client I'm missing that can shorten my path?
> > > >
> > > > If I adopt one of these projects (lets say kafka-cpp) am I better off
> > > > implementing the 0.8 protocol? I'de like to have something running in
> > > > staging a couple months from now. How far out is 0.8?
> > > >
> > > > Thanks,
> > > > Matt Stump
> > > >
> > >
> >
>
Re: C/C++ Client
Posted by anand nalya <a....@computer.org>.
Hi Joel,
Any updates on the c++ producer?
Thanks,
Anand
On 3 April 2013 05:59, Joel Koshy <jj...@gmail.com> wrote:
> Yes - we would be interested in doing that. I have been spending most of my
> time over the past couple weeks on the C++ library (currently, only for the
> producer). It is reasonably stable, although it has not been tried and
> tested in production.
>
> I can start with publishing a wiki describing the design/implementation and
> list out future work that others can contribute to if there is interest,
> but I'll quickly summarize some of its goals here:
> - Use non-blocking I/O for sending requests/receiving responses. This
> mitigates the request-response RTT latency that the 0.8 protocol introduces
> (if acks != 0).
> - C-compliant API, so that other languages e.g., Ruby/Python can wrap the
> library. I understand there are people working on 0.8 clients in those
> languages so this is just an alternative approach.
> - Non-blocking variants for all operations in the API: there are some
> use-cases where blocking in the user-thread is unacceptable.
> - Keep third-party libraries to a minimum to make porting to other
> platforms easier.
>
> There are two options in pursuing this:
> 1 - Contribute it as part of the Kafka project. i.e., re-introduce the
> clients directory. We removed it because none of the committers were
> original authors of any of the non-Java clients and did not have the
> bandwidth/background to review patches for those clients.
> 2 - Maintain it externally.
> I was going to also suggest "sub-project" as a third option, but I heard
> from Jakob that it is no longer encouraged by the board and a client by
> itself would not be a substantial sub-project.
>
> If we want to do (1), we would obviously go through the standard
> contribution process: i.e., create a jira for it and provide a patch that
> people can review; and other implementations will also be considered if
> contributors provide patches.
>
> (2) is also fine, although one benefit with (1) is that it would be
> reasonable to use the existing infrastructure (mailing lists, issue
> tracking) for the Kafka project.
>
> Any preferences/thoughts?
>
> Thanks,
>
> Joel
>
> On Mon, Apr 1, 2013 at 8:22 AM, mrevilgnome <mr...@gmail.com> wrote:
>
> > Any interest in open sourcing it now and picking up contributors?
> >
> >
> > On Mon, Apr 1, 2013 at 7:54 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > At LinkedIn, we are also building a native C producer client for 0.8.
> It
> > > uses non-blocking socket I/O to improve the producer throughput. We
> plan
> > to
> > > open source it when it's fully tested, hopefully in a couple of months.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Mar 27, 2013 at 8:48 PM, Matthew Stump <
> mstump@matthewstump.com
> > > >wrote:
> > >
> > > > Howdy,
> > > >
> > > > I'm considering the use of Kafka in the rewrite of a big legacy
> > product.
> > > A
> > > > good chunk of the back end code is going to be written in C++ (large
> in
> > > > memory data-structures). The two possible options available to me for
> > > > clients appear to be:
> > > >
> > > > https://github.com/edenhill/librdkafka
> > > >
> > > > and
> > > >
> > > > https://github.com/quipo/kafka-cpp
> > > >
> > > > The problem is that librdkafka currently only works on Linux due to
> the
> > > use
> > > > of the AF_NETLINK API, and thread local storage. There may be other
> > > issues,
> > > > but I just started playing with it today and that's what I've
> > discovered
> > > > thus far.
> > > >
> > > > kafka-cpp is incomplete (no consumer) and it looks unused.
> > > >
> > > > For either I would need to hop in and do some significant work. Is
> > there
> > > > any client I'm missing that can shorten my path?
> > > >
> > > > If I adopt one of these projects (lets say kafka-cpp) am I better off
> > > > implementing the 0.8 protocol? I'de like to have something running in
> > > > staging a couple months from now. How far out is 0.8?
> > > >
> > > > Thanks,
> > > > Matt Stump
> > > >
> > >
> >
>
Re: C/C++ Client
Posted by Joel Koshy <jj...@gmail.com>.
Yes - we would be interested in doing that. I have been spending most of my
time over the past couple weeks on the C++ library (currently, only for the
producer). It is reasonably stable, although it has not been tried and
tested in production.
I can start with publishing a wiki describing the design/implementation and
list out future work that others can contribute to if there is interest,
but I'll quickly summarize some of its goals here:
- Use non-blocking I/O for sending requests/receiving responses. This
mitigates the request-response RTT latency that the 0.8 protocol introduces
(if acks != 0).
- C-compliant API, so that other languages e.g., Ruby/Python can wrap the
library. I understand there are people working on 0.8 clients in those
languages so this is just an alternative approach.
- Non-blocking variants for all operations in the API: there are some
use-cases where blocking in the user-thread is unacceptable.
- Keep third-party libraries to a minimum to make porting to other
platforms easier.
There are two options in pursuing this:
1 - Contribute it as part of the Kafka project. i.e., re-introduce the
clients directory. We removed it because none of the committers were
original authors of any of the non-Java clients and did not have the
bandwidth/background to review patches for those clients.
2 - Maintain it externally.
I was going to also suggest "sub-project" as a third option, but I heard
from Jakob that it is no longer encouraged by the board and a client by
itself would not be a substantial sub-project.
If we want to do (1), we would obviously go through the standard
contribution process: i.e., create a jira for it and provide a patch that
people can review; and other implementations will also be considered if
contributors provide patches.
(2) is also fine, although one benefit with (1) is that it would be
reasonable to use the existing infrastructure (mailing lists, issue
tracking) for the Kafka project.
Any preferences/thoughts?
Thanks,
Joel
On Mon, Apr 1, 2013 at 8:22 AM, mrevilgnome <mr...@gmail.com> wrote:
> Any interest in open sourcing it now and picking up contributors?
>
>
> On Mon, Apr 1, 2013 at 7:54 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > At LinkedIn, we are also building a native C producer client for 0.8. It
> > uses non-blocking socket I/O to improve the producer throughput. We plan
> to
> > open source it when it's fully tested, hopefully in a couple of months.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Mar 27, 2013 at 8:48 PM, Matthew Stump <mstump@matthewstump.com
> > >wrote:
> >
> > > Howdy,
> > >
> > > I'm considering the use of Kafka in the rewrite of a big legacy
> product.
> > A
> > > good chunk of the back end code is going to be written in C++ (large in
> > > memory data-structures). The two possible options available to me for
> > > clients appear to be:
> > >
> > > https://github.com/edenhill/librdkafka
> > >
> > > and
> > >
> > > https://github.com/quipo/kafka-cpp
> > >
> > > The problem is that librdkafka currently only works on Linux due to the
> > use
> > > of the AF_NETLINK API, and thread local storage. There may be other
> > issues,
> > > but I just started playing with it today and that's what I've
> discovered
> > > thus far.
> > >
> > > kafka-cpp is incomplete (no consumer) and it looks unused.
> > >
> > > For either I would need to hop in and do some significant work. Is
> there
> > > any client I'm missing that can shorten my path?
> > >
> > > If I adopt one of these projects (lets say kafka-cpp) am I better off
> > > implementing the 0.8 protocol? I'de like to have something running in
> > > staging a couple months from now. How far out is 0.8?
> > >
> > > Thanks,
> > > Matt Stump
> > >
> >
>
Re: C/C++ Client
Posted by Joel Koshy <jj...@gmail.com>.
Yes - we would be interested in doing that. I have been spending most of my
time over the past couple weeks on the C++ library (currently, only for the
producer). It is reasonably stable, although it has not been tried and
tested in production.
I can start with publishing a wiki describing the design/implementation and
list out future work that others can contribute to if there is interest,
but I'll quickly summarize some of its goals here:
- Use non-blocking I/O for sending requests/receiving responses. This
mitigates the request-response RTT latency that the 0.8 protocol introduces
(if acks != 0).
- C-compliant API, so that other languages e.g., Ruby/Python can wrap the
library. I understand there are people working on 0.8 clients in those
languages so this is just an alternative approach.
- Non-blocking variants for all operations in the API: there are some
use-cases where blocking in the user-thread is unacceptable.
- Keep third-party libraries to a minimum to make porting to other
platforms easier.
There are two options in pursuing this:
1 - Contribute it as part of the Kafka project. i.e., re-introduce the
clients directory. We removed it because none of the committers were
original authors of any of the non-Java clients and did not have the
bandwidth/background to review patches for those clients.
2 - Maintain it externally.
I was going to also suggest "sub-project" as a third option, but I heard
from Jakob that it is no longer encouraged by the board and a client by
itself would not be a substantial sub-project.
If we want to do (1), we would obviously go through the standard
contribution process: i.e., create a jira for it and provide a patch that
people can review; and other implementations will also be considered if
contributors provide patches.
(2) is also fine, although one benefit with (1) is that it would be
reasonable to use the existing infrastructure (mailing lists, issue
tracking) for the Kafka project.
Any preferences/thoughts?
Thanks,
Joel
On Mon, Apr 1, 2013 at 8:22 AM, mrevilgnome <mr...@gmail.com> wrote:
> Any interest in open sourcing it now and picking up contributors?
>
>
> On Mon, Apr 1, 2013 at 7:54 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > At LinkedIn, we are also building a native C producer client for 0.8. It
> > uses non-blocking socket I/O to improve the producer throughput. We plan
> to
> > open source it when it's fully tested, hopefully in a couple of months.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Mar 27, 2013 at 8:48 PM, Matthew Stump <mstump@matthewstump.com
> > >wrote:
> >
> > > Howdy,
> > >
> > > I'm considering the use of Kafka in the rewrite of a big legacy
> product.
> > A
> > > good chunk of the back end code is going to be written in C++ (large in
> > > memory data-structures). The two possible options available to me for
> > > clients appear to be:
> > >
> > > https://github.com/edenhill/librdkafka
> > >
> > > and
> > >
> > > https://github.com/quipo/kafka-cpp
> > >
> > > The problem is that librdkafka currently only works on Linux due to the
> > use
> > > of the AF_NETLINK API, and thread local storage. There may be other
> > issues,
> > > but I just started playing with it today and that's what I've
> discovered
> > > thus far.
> > >
> > > kafka-cpp is incomplete (no consumer) and it looks unused.
> > >
> > > For either I would need to hop in and do some significant work. Is
> there
> > > any client I'm missing that can shorten my path?
> > >
> > > If I adopt one of these projects (lets say kafka-cpp) am I better off
> > > implementing the 0.8 protocol? I'de like to have something running in
> > > staging a couple months from now. How far out is 0.8?
> > >
> > > Thanks,
> > > Matt Stump
> > >
> >
>
Re: C/C++ Client
Posted by mrevilgnome <mr...@gmail.com>.
Any interest in open sourcing it now and picking up contributors?
On Mon, Apr 1, 2013 at 7:54 AM, Jun Rao <ju...@gmail.com> wrote:
> At LinkedIn, we are also building a native C producer client for 0.8. It
> uses non-blocking socket I/O to improve the producer throughput. We plan to
> open source it when it's fully tested, hopefully in a couple of months.
>
> Thanks,
>
> Jun
>
> On Wed, Mar 27, 2013 at 8:48 PM, Matthew Stump <mstump@matthewstump.com
> >wrote:
>
> > Howdy,
> >
> > I'm considering the use of Kafka in the rewrite of a big legacy product.
> A
> > good chunk of the back end code is going to be written in C++ (large in
> > memory data-structures). The two possible options available to me for
> > clients appear to be:
> >
> > https://github.com/edenhill/librdkafka
> >
> > and
> >
> > https://github.com/quipo/kafka-cpp
> >
> > The problem is that librdkafka currently only works on Linux due to the
> use
> > of the AF_NETLINK API, and thread local storage. There may be other
> issues,
> > but I just started playing with it today and that's what I've discovered
> > thus far.
> >
> > kafka-cpp is incomplete (no consumer) and it looks unused.
> >
> > For either I would need to hop in and do some significant work. Is there
> > any client I'm missing that can shorten my path?
> >
> > If I adopt one of these projects (lets say kafka-cpp) am I better off
> > implementing the 0.8 protocol? I'de like to have something running in
> > staging a couple months from now. How far out is 0.8?
> >
> > Thanks,
> > Matt Stump
> >
>