You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Christopher Alexander <ca...@gravycard.com> on 2013/04/24 19:19:16 UTC
Node-Kafka Client Review and Question
Hi Everyone,
I just wanted to follow-up on a previous thread concerning our investigation of identifying a stable Node-Kafka client. To date we have tested the following:
1. Franz-Kafka (https://github.com/dannycoates/franz-kafka)
2. Node-Kafka (v2.1, https://github.com/radekg/node-kafka)
3. Node-Kafka (v2.3, https://github.com/marcuswestin/node-kafka)
4. Prozess (v0.3.5, https://github.com/cainus/Prozess)
Results:
1. Could not get Franz-Kafka and Prozess to work. Requires funky dependencies.
2. Node-Kafka, v2.1 was successfully setup but performed less stable than #3.
3. Node-Kafka, v2.3 was successfully setup, exhibited the best performance profile but the consumer is highly inconsistent - specifically, consumer object remained in-memory regardless what we did (i.e. var consumer = undefined after receiving message). Nothing appears to mitigate this and ALL consumed messaged get replayed on reception of a new message.
With this said, is there a Node-Kafka client people are actually using in production that doesn't exhibit the profiles we have seen? We have back-tracked using Node-Kafka (v2.3) to only produce messages and rely on Redis PubSub channels for asynchronous acking of these messages. We would be willing to roll-up our sleeves with the community to develop a much more stable Node-Kafka client.
Kind regards,
Chris Alexander
Chief Technical Architect and Engineer
Gravy, Inc.
Re: Node-Kafka Client Review and Question
Posted by Christopher Alexander <ca...@gravycard.com>.
We did an examination of the tagged branch but the version was 0.1.7 and its been static for over a year now. I will say that the Node-Kafka (v2.3) producer has been stable however. A previous thread concerning Node-Kafka client development revealed that a C library will be out for 0.8, supporting more stable library client development (hint hint too ;-)
Unfortunately, we are moving rapidly with what works now to get our platform to an MVP state. Hopefully, there won't be too much refactoring when the next gen Node-Kafka client is complete. :(
Chris
----- Original Message -----
From: "Taylor Gautier" <tg...@gmail.com>
To: users@kafka.apache.org
Sent: Wednesday, April 24, 2013 4:39:24 PM
Subject: Re: Node-Kafka Client Review and Question
This implementation is what I worked on while at Tagged, which was forked
from Marcus' version, but I don't think it ever merged back to Marcus':
https://github.com/tagged/node-kafka
It was in production for about a year when I left Tagged about 6 months
ago. I know that there were some internal fixes that never made it back
out. I think the version that is public is pretty stable, but it's hard
for me to know since I am no longer at Tagged and don't have access to the
internal repos to know for sure what fixes are still private and which have
been published back.
That said, I think you should definitely give this version a shot first
before moving on.
Finally, if I were to do it all over again, with about a solid additional
year of node programming experience under my belt, I'd probably rewrite
everything from the interfaces to the implementation.
In my current job there's a strong possibility of us adopting a Node.js +
Kafka implementation, but not for at least a few months, so I wouldn't
expect to be back in the community to work on this for a little while.
Also, I'm kind of waiting for 0.8 (hint hint ;-)
On Wed, Apr 24, 2013 at 10:45 AM, Christian Carollo <cc...@gmail.com>wrote:
> Hi Everyone,
>
> I have been experimenting with the libraries listed below and experienced
> the same problems.
> I have not found any another other node clients. I am interested in
> finding a node solution as well.
> Happy to contribute on a common solution.
>
> Christian Carollo
>
> On Apr 24, 2013, at 10:19 AM, Christopher Alexander <
> calexander@gravycard.com> wrote:
>
> > Hi Everyone,
> >
> > I just wanted to follow-up on a previous thread concerning our
> investigation of identifying a stable Node-Kafka client. To date we have
> tested the following:
> >
> > 1. Franz-Kafka (https://github.com/dannycoates/franz-kafka)
> > 2. Node-Kafka (v2.1, https://github.com/radekg/node-kafka)
> > 3. Node-Kafka (v2.3, https://github.com/marcuswestin/node-kafka)
> > 4. Prozess (v0.3.5, https://github.com/cainus/Prozess)
> >
> > Results:
> >
> > 1. Could not get Franz-Kafka and Prozess to work. Requires funky
> dependencies.
> > 2. Node-Kafka, v2.1 was successfully setup but performed less stable
> than #3.
> > 3. Node-Kafka, v2.3 was successfully setup, exhibited the best
> performance profile but the consumer is highly inconsistent - specifically,
> consumer object remained in-memory regardless what we did (i.e. var
> consumer = undefined after receiving message). Nothing appears to mitigate
> this and ALL consumed messaged get replayed on reception of a new message.
> >
> > With this said, is there a Node-Kafka client people are actually using
> in production that doesn't exhibit the profiles we have seen? We have
> back-tracked using Node-Kafka (v2.3) to only produce messages and rely on
> Redis PubSub channels for asynchronous acking of these messages. We would
> be willing to roll-up our sleeves with the community to develop a much more
> stable Node-Kafka client.
> >
> > Kind regards,
> >
> > Chris Alexander
> > Chief Technical Architect and Engineer
> > Gravy, Inc.
>
>
Re: Node-Kafka Client Review and Question
Posted by Taylor Gautier <tg...@gmail.com>.
This implementation is what I worked on while at Tagged, which was forked
from Marcus' version, but I don't think it ever merged back to Marcus':
https://github.com/tagged/node-kafka
It was in production for about a year when I left Tagged about 6 months
ago. I know that there were some internal fixes that never made it back
out. I think the version that is public is pretty stable, but it's hard
for me to know since I am no longer at Tagged and don't have access to the
internal repos to know for sure what fixes are still private and which have
been published back.
That said, I think you should definitely give this version a shot first
before moving on.
Finally, if I were to do it all over again, with about a solid additional
year of node programming experience under my belt, I'd probably rewrite
everything from the interfaces to the implementation.
In my current job there's a strong possibility of us adopting a Node.js +
Kafka implementation, but not for at least a few months, so I wouldn't
expect to be back in the community to work on this for a little while.
Also, I'm kind of waiting for 0.8 (hint hint ;-)
On Wed, Apr 24, 2013 at 10:45 AM, Christian Carollo <cc...@gmail.com>wrote:
> Hi Everyone,
>
> I have been experimenting with the libraries listed below and experienced
> the same problems.
> I have not found any another other node clients. I am interested in
> finding a node solution as well.
> Happy to contribute on a common solution.
>
> Christian Carollo
>
> On Apr 24, 2013, at 10:19 AM, Christopher Alexander <
> calexander@gravycard.com> wrote:
>
> > Hi Everyone,
> >
> > I just wanted to follow-up on a previous thread concerning our
> investigation of identifying a stable Node-Kafka client. To date we have
> tested the following:
> >
> > 1. Franz-Kafka (https://github.com/dannycoates/franz-kafka)
> > 2. Node-Kafka (v2.1, https://github.com/radekg/node-kafka)
> > 3. Node-Kafka (v2.3, https://github.com/marcuswestin/node-kafka)
> > 4. Prozess (v0.3.5, https://github.com/cainus/Prozess)
> >
> > Results:
> >
> > 1. Could not get Franz-Kafka and Prozess to work. Requires funky
> dependencies.
> > 2. Node-Kafka, v2.1 was successfully setup but performed less stable
> than #3.
> > 3. Node-Kafka, v2.3 was successfully setup, exhibited the best
> performance profile but the consumer is highly inconsistent - specifically,
> consumer object remained in-memory regardless what we did (i.e. var
> consumer = undefined after receiving message). Nothing appears to mitigate
> this and ALL consumed messaged get replayed on reception of a new message.
> >
> > With this said, is there a Node-Kafka client people are actually using
> in production that doesn't exhibit the profiles we have seen? We have
> back-tracked using Node-Kafka (v2.3) to only produce messages and rely on
> Redis PubSub channels for asynchronous acking of these messages. We would
> be willing to roll-up our sleeves with the community to develop a much more
> stable Node-Kafka client.
> >
> > Kind regards,
> >
> > Chris Alexander
> > Chief Technical Architect and Engineer
> > Gravy, Inc.
>
>
Re: Node-Kafka Client Review and Question
Posted by Christian Carollo <cc...@gmail.com>.
Hi Everyone,
I have been experimenting with the libraries listed below and experienced the same problems.
I have not found any another other node clients. I am interested in finding a node solution as well.
Happy to contribute on a common solution.
Christian Carollo
On Apr 24, 2013, at 10:19 AM, Christopher Alexander <ca...@gravycard.com> wrote:
> Hi Everyone,
>
> I just wanted to follow-up on a previous thread concerning our investigation of identifying a stable Node-Kafka client. To date we have tested the following:
>
> 1. Franz-Kafka (https://github.com/dannycoates/franz-kafka)
> 2. Node-Kafka (v2.1, https://github.com/radekg/node-kafka)
> 3. Node-Kafka (v2.3, https://github.com/marcuswestin/node-kafka)
> 4. Prozess (v0.3.5, https://github.com/cainus/Prozess)
>
> Results:
>
> 1. Could not get Franz-Kafka and Prozess to work. Requires funky dependencies.
> 2. Node-Kafka, v2.1 was successfully setup but performed less stable than #3.
> 3. Node-Kafka, v2.3 was successfully setup, exhibited the best performance profile but the consumer is highly inconsistent - specifically, consumer object remained in-memory regardless what we did (i.e. var consumer = undefined after receiving message). Nothing appears to mitigate this and ALL consumed messaged get replayed on reception of a new message.
>
> With this said, is there a Node-Kafka client people are actually using in production that doesn't exhibit the profiles we have seen? We have back-tracked using Node-Kafka (v2.3) to only produce messages and rely on Redis PubSub channels for asynchronous acking of these messages. We would be willing to roll-up our sleeves with the community to develop a much more stable Node-Kafka client.
>
> Kind regards,
>
> Chris Alexander
> Chief Technical Architect and Engineer
> Gravy, Inc.