You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Matti Waarna <mw...@sapient.com> on 2014/12/22 16:55:18 UTC
Kafka in C#
We are using kafka version 0.8.1.1 and trying to produce from a C# app.
I realize that there is no official C# library release and want to get your experience with the existing solutions that are currently available.
I am looking for a solution that is a) stable enough for production environment and b) performs well.
1) A couple of active github projects are available along with a few forks each.
Has Anybody worked on either of the following two options to contribute their findings?
https://github.com/Jroland/kafka-net
https://github.com/miknil/Kafka4n
2) Also there is the option of IKVM to import kafka JARS into a .net DLL.
Maybe even another solution?
Thanks
MATTI
RE: Kafka in C#
Posted by Thunder Stumpges <ts...@ntent.com>.
Hi Matti,
We are actively testing in staging, under load and with extended reliability tests. It has been running well from a producer side for many days. We are pushing hard to get to production in the next couple of weeks however much of the testing is not directly related to the kafka client, but other changes in our core product.
As such, we have not had time to update documentation. Still, we would be pleased if you had a look and evaluated it; I think it is more fully featured than any of the other .net clients. If you do, please let me know how you get on. We'd be glad to help you out however we can.
Cheers,
Thunder
-----Original Message-----
From: Matti Waarna [mailto:mwaarna@sapient.com]
Sent: Friday, January 09, 2015 9:43 AM
To: users@kafka.apache.org; Thunder Stumpges
Subject: RE: Kafka in C#
Thunder,
Have you had a chance to go to production with the C# producer and any updates on documentation?
I will be checking out the C# library sometime next week.
Thanks
-Matti
-----Original Message-----
From: Thunder Stumpges [mailto:tstumpges@ntent.com]
Sent: Monday, December 22, 2014 1:10 PM
To: users@kafka.apache.org
Subject: RE: Kafka in C#
We definitely will. Wanted to wait until we got more confident the API was solid, ran in production for a little, and added a bit more documentation.
Cheers,
Thunder
-----Original Message-----
From: Joe Stein [joe.stein@stealth.ly]
Received: Monday, 22 Dec 2014, 10:05AM
To: users@kafka.apache.org [users@kafka.apache.org]
Subject: Re: Kafka in C#
Thunder, can you add that to
https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-.net
didn't know it existed but cool that it uses Rx
On Mon, Dec 22, 2014 at 12:52 PM, Thunder Stumpges <ts...@ntent.com>
wrote:
> Hi there,
>
> We looked at both of these a while back and ended up writing our own (
> https://github.com/ntent-ad/kafka4net).
>
> The first one in the list was completely synchronous, and had no
> concept of batching. We initially attempted to use the second one
> (kafka-net) but had some issues with detecting leader changes,
> temporary errors, and general topic metadata changes. Also had some
> problems with the use of async and the tracking of correlation IDs on tcp messages.
>
> I know our client does not have a lot of documentation (yet) and says
> "Work in progress, not ready yet!" but we have been testing this a
> lot, and have been running in staging under load with good results. We
> will be entering production in the next few weeks (waiting until after
> the holidays). I wouldn't be comfortable with it going straight to
> production without some testing in your environment, but especially
> the Producer is very robust, and is resilient to all sorts of changes
> in the environment (the integration tests use Vagrant and a set of VMs
> and test Producing to non-existent partition and waiting for
> auto-creation of topic, partition rebalancing, broker down or other
> partition re-assignment, etc.) The client is fully Async and leverages
> Rx (https://rx.codeplex.com/) and an event-loop-scheduler to do all
> processing. Changes in partition state are broadcast Rx style to
> listening components (Producer, Consumer, PartitionRecoveryMonitor).
>
> One of the reasons we have not finalized the documentation and notice
> on the github page is we weren't sure if the API might change based on usage.
> To this end, we'd like to know what you think and if you have any
> use-cases not handled by the API.
>
> I understand if you're not comfortable with the beta state of the
> client, but we'd love to have you check it out. We are actively
> developing on this and can help with any issues.
>
> Thanks,
> Thunder
>
>
> -----Original Message-----
> From: Matti Waarna [mailto:mwaarna@sapient.com]
> Sent: Monday, December 22, 2014 7:55 AM
> To: users@kafka.apache.org
> Subject: Kafka in C#
>
> We are using kafka version 0.8.1.1 and trying to produce from a C# app.
>
> I realize that there is no official C# library release and want to get
> your experience with the existing solutions that are currently available.
>
> I am looking for a solution that is a) stable enough for production
> environment and b) performs well.
>
> 1) A couple of active github projects are available along with a few
> forks each.
> Has Anybody worked on either of the following two options to
> contribute their findings?
> https://github.com/Jroland/kafka-net
>
> https://github.com/miknil/Kafka4n
>
> 2) Also there is the option of IKVM to import kafka JARS into a .net DLL.
>
> Maybe even another solution?
>
> Thanks
>
> MATTI
>
>
RE: Kafka in C#
Posted by Matti Waarna <mw...@sapient.com>.
Thunder,
Have you had a chance to go to production with the C# producer and any updates on documentation?
I will be checking out the C# library sometime next week.
Thanks
-Matti
-----Original Message-----
From: Thunder Stumpges [mailto:tstumpges@ntent.com]
Sent: Monday, December 22, 2014 1:10 PM
To: users@kafka.apache.org
Subject: RE: Kafka in C#
We definitely will. Wanted to wait until we got more confident the API was solid, ran in production for a little, and added a bit more documentation.
Cheers,
Thunder
-----Original Message-----
From: Joe Stein [joe.stein@stealth.ly]
Received: Monday, 22 Dec 2014, 10:05AM
To: users@kafka.apache.org [users@kafka.apache.org]
Subject: Re: Kafka in C#
Thunder, can you add that to
https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-.net
didn't know it existed but cool that it uses Rx
On Mon, Dec 22, 2014 at 12:52 PM, Thunder Stumpges <ts...@ntent.com>
wrote:
> Hi there,
>
> We looked at both of these a while back and ended up writing our own (
> https://github.com/ntent-ad/kafka4net).
>
> The first one in the list was completely synchronous, and had no concept
> of batching. We initially attempted to use the second one (kafka-net) but
> had some issues with detecting leader changes, temporary errors, and
> general topic metadata changes. Also had some problems with the use of
> async and the tracking of correlation IDs on tcp messages.
>
> I know our client does not have a lot of documentation (yet) and says
> "Work in progress, not ready yet!" but we have been testing this a lot, and
> have been running in staging under load with good results. We will be
> entering production in the next few weeks (waiting until after the
> holidays). I wouldn't be comfortable with it going straight to production
> without some testing in your environment, but especially the Producer is
> very robust, and is resilient to all sorts of changes in the environment
> (the integration tests use Vagrant and a set of VMs and test Producing to
> non-existent partition and waiting for auto-creation of topic, partition
> rebalancing, broker down or other partition re-assignment, etc.) The client
> is fully Async and leverages Rx (https://rx.codeplex.com/) and an
> event-loop-scheduler to do all processing. Changes in partition state are
> broadcast Rx style to listening components (Producer, Consumer,
> PartitionRecoveryMonitor).
>
> One of the reasons we have not finalized the documentation and notice on
> the github page is we weren't sure if the API might change based on usage.
> To this end, we'd like to know what you think and if you have any use-cases
> not handled by the API.
>
> I understand if you're not comfortable with the beta state of the client,
> but we'd love to have you check it out. We are actively developing on this
> and can help with any issues.
>
> Thanks,
> Thunder
>
>
> -----Original Message-----
> From: Matti Waarna [mailto:mwaarna@sapient.com]
> Sent: Monday, December 22, 2014 7:55 AM
> To: users@kafka.apache.org
> Subject: Kafka in C#
>
> We are using kafka version 0.8.1.1 and trying to produce from a C# app.
>
> I realize that there is no official C# library release and want to get
> your experience with the existing solutions that are currently available.
>
> I am looking for a solution that is a) stable enough for production
> environment and b) performs well.
>
> 1) A couple of active github projects are available along with a few forks
> each.
> Has Anybody worked on either of the following two options to contribute
> their findings?
> https://github.com/Jroland/kafka-net
>
> https://github.com/miknil/Kafka4n
>
> 2) Also there is the option of IKVM to import kafka JARS into a .net DLL.
>
> Maybe even another solution?
>
> Thanks
>
> MATTI
>
>
RE: Kafka in C#
Posted by Thunder Stumpges <ts...@ntent.com>.
We definitely will. Wanted to wait until we got more confident the API was solid, ran in production for a little, and added a bit more documentation.
Cheers,
Thunder
-----Original Message-----
From: Joe Stein [joe.stein@stealth.ly]
Received: Monday, 22 Dec 2014, 10:05AM
To: users@kafka.apache.org [users@kafka.apache.org]
Subject: Re: Kafka in C#
Thunder, can you add that to
https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-.net
didn't know it existed but cool that it uses Rx
On Mon, Dec 22, 2014 at 12:52 PM, Thunder Stumpges <ts...@ntent.com>
wrote:
> Hi there,
>
> We looked at both of these a while back and ended up writing our own (
> https://github.com/ntent-ad/kafka4net).
>
> The first one in the list was completely synchronous, and had no concept
> of batching. We initially attempted to use the second one (kafka-net) but
> had some issues with detecting leader changes, temporary errors, and
> general topic metadata changes. Also had some problems with the use of
> async and the tracking of correlation IDs on tcp messages.
>
> I know our client does not have a lot of documentation (yet) and says
> "Work in progress, not ready yet!" but we have been testing this a lot, and
> have been running in staging under load with good results. We will be
> entering production in the next few weeks (waiting until after the
> holidays). I wouldn't be comfortable with it going straight to production
> without some testing in your environment, but especially the Producer is
> very robust, and is resilient to all sorts of changes in the environment
> (the integration tests use Vagrant and a set of VMs and test Producing to
> non-existent partition and waiting for auto-creation of topic, partition
> rebalancing, broker down or other partition re-assignment, etc.) The client
> is fully Async and leverages Rx (https://rx.codeplex.com/) and an
> event-loop-scheduler to do all processing. Changes in partition state are
> broadcast Rx style to listening components (Producer, Consumer,
> PartitionRecoveryMonitor).
>
> One of the reasons we have not finalized the documentation and notice on
> the github page is we weren't sure if the API might change based on usage.
> To this end, we'd like to know what you think and if you have any use-cases
> not handled by the API.
>
> I understand if you're not comfortable with the beta state of the client,
> but we'd love to have you check it out. We are actively developing on this
> and can help with any issues.
>
> Thanks,
> Thunder
>
>
> -----Original Message-----
> From: Matti Waarna [mailto:mwaarna@sapient.com]
> Sent: Monday, December 22, 2014 7:55 AM
> To: users@kafka.apache.org
> Subject: Kafka in C#
>
> We are using kafka version 0.8.1.1 and trying to produce from a C# app.
>
> I realize that there is no official C# library release and want to get
> your experience with the existing solutions that are currently available.
>
> I am looking for a solution that is a) stable enough for production
> environment and b) performs well.
>
> 1) A couple of active github projects are available along with a few forks
> each.
> Has Anybody worked on either of the following two options to contribute
> their findings?
> https://github.com/Jroland/kafka-net
>
> https://github.com/miknil/Kafka4n
>
> 2) Also there is the option of IKVM to import kafka JARS into a .net DLL.
>
> Maybe even another solution?
>
> Thanks
>
> MATTI
>
>
Re: Kafka in C#
Posted by Joe Stein <jo...@stealth.ly>.
Thunder, can you add that to
https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-.net
didn't know it existed but cool that it uses Rx
On Mon, Dec 22, 2014 at 12:52 PM, Thunder Stumpges <ts...@ntent.com>
wrote:
> Hi there,
>
> We looked at both of these a while back and ended up writing our own (
> https://github.com/ntent-ad/kafka4net).
>
> The first one in the list was completely synchronous, and had no concept
> of batching. We initially attempted to use the second one (kafka-net) but
> had some issues with detecting leader changes, temporary errors, and
> general topic metadata changes. Also had some problems with the use of
> async and the tracking of correlation IDs on tcp messages.
>
> I know our client does not have a lot of documentation (yet) and says
> "Work in progress, not ready yet!" but we have been testing this a lot, and
> have been running in staging under load with good results. We will be
> entering production in the next few weeks (waiting until after the
> holidays). I wouldn't be comfortable with it going straight to production
> without some testing in your environment, but especially the Producer is
> very robust, and is resilient to all sorts of changes in the environment
> (the integration tests use Vagrant and a set of VMs and test Producing to
> non-existent partition and waiting for auto-creation of topic, partition
> rebalancing, broker down or other partition re-assignment, etc.) The client
> is fully Async and leverages Rx (https://rx.codeplex.com/) and an
> event-loop-scheduler to do all processing. Changes in partition state are
> broadcast Rx style to listening components (Producer, Consumer,
> PartitionRecoveryMonitor).
>
> One of the reasons we have not finalized the documentation and notice on
> the github page is we weren't sure if the API might change based on usage.
> To this end, we'd like to know what you think and if you have any use-cases
> not handled by the API.
>
> I understand if you're not comfortable with the beta state of the client,
> but we'd love to have you check it out. We are actively developing on this
> and can help with any issues.
>
> Thanks,
> Thunder
>
>
> -----Original Message-----
> From: Matti Waarna [mailto:mwaarna@sapient.com]
> Sent: Monday, December 22, 2014 7:55 AM
> To: users@kafka.apache.org
> Subject: Kafka in C#
>
> We are using kafka version 0.8.1.1 and trying to produce from a C# app.
>
> I realize that there is no official C# library release and want to get
> your experience with the existing solutions that are currently available.
>
> I am looking for a solution that is a) stable enough for production
> environment and b) performs well.
>
> 1) A couple of active github projects are available along with a few forks
> each.
> Has Anybody worked on either of the following two options to contribute
> their findings?
> https://github.com/Jroland/kafka-net
>
> https://github.com/miknil/Kafka4n
>
> 2) Also there is the option of IKVM to import kafka JARS into a .net DLL.
>
> Maybe even another solution?
>
> Thanks
>
> MATTI
>
>
RE: Kafka in C#
Posted by Thunder Stumpges <ts...@ntent.com>.
Hi there,
We looked at both of these a while back and ended up writing our own (https://github.com/ntent-ad/kafka4net).
The first one in the list was completely synchronous, and had no concept of batching. We initially attempted to use the second one (kafka-net) but had some issues with detecting leader changes, temporary errors, and general topic metadata changes. Also had some problems with the use of async and the tracking of correlation IDs on tcp messages.
I know our client does not have a lot of documentation (yet) and says "Work in progress, not ready yet!" but we have been testing this a lot, and have been running in staging under load with good results. We will be entering production in the next few weeks (waiting until after the holidays). I wouldn't be comfortable with it going straight to production without some testing in your environment, but especially the Producer is very robust, and is resilient to all sorts of changes in the environment (the integration tests use Vagrant and a set of VMs and test Producing to non-existent partition and waiting for auto-creation of topic, partition rebalancing, broker down or other partition re-assignment, etc.) The client is fully Async and leverages Rx (https://rx.codeplex.com/) and an event-loop-scheduler to do all processing. Changes in partition state are broadcast Rx style to listening components (Producer, Consumer, PartitionRecoveryMonitor).
One of the reasons we have not finalized the documentation and notice on the github page is we weren't sure if the API might change based on usage. To this end, we'd like to know what you think and if you have any use-cases not handled by the API.
I understand if you're not comfortable with the beta state of the client, but we'd love to have you check it out. We are actively developing on this and can help with any issues.
Thanks,
Thunder
-----Original Message-----
From: Matti Waarna [mailto:mwaarna@sapient.com]
Sent: Monday, December 22, 2014 7:55 AM
To: users@kafka.apache.org
Subject: Kafka in C#
We are using kafka version 0.8.1.1 and trying to produce from a C# app.
I realize that there is no official C# library release and want to get your experience with the existing solutions that are currently available.
I am looking for a solution that is a) stable enough for production environment and b) performs well.
1) A couple of active github projects are available along with a few forks each.
Has Anybody worked on either of the following two options to contribute their findings?
https://github.com/Jroland/kafka-net
https://github.com/miknil/Kafka4n
2) Also there is the option of IKVM to import kafka JARS into a .net DLL.
Maybe even another solution?
Thanks
MATTI
Re: Kafka in C#
Posted by Joe Stein <jo...@stealth.ly>.
Another option is a HTTP wrapper around the actual producer and doing a
HTTP POST from C# to a REST server e.g.
https://github.com/stealthly/dropwizard-kafka-http which I know folks have
done successfully.
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Mon, Dec 22, 2014 at 10:55 AM, Matti Waarna <mw...@sapient.com> wrote:
> We are using kafka version 0.8.1.1 and trying to produce from a C# app.
>
> I realize that there is no official C# library release and want to get
> your experience with the existing solutions that are currently available.
>
> I am looking for a solution that is a) stable enough for production
> environment and b) performs well.
>
> 1) A couple of active github projects are available along with a few forks
> each.
> Has Anybody worked on either of the following two options to contribute
> their findings?
> https://github.com/Jroland/kafka-net
>
> https://github.com/miknil/Kafka4n
>
> 2) Also there is the option of IKVM to import kafka JARS into a .net DLL.
>
> Maybe even another solution?
>
> Thanks
>
> MATTI
>
>