You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by siddharth ubale <si...@gmail.com> on 2014/08/20 14:04:47 UTC
Data inputs for Kafka.
Hi guys,
We are implementing a messaging system in order to perform Real time
analytics. However, we are confused on using ActiveMQ or Apache Kafka. The
point of worry for us are :
1. What kind of data can be ingested by Apache kafka? If we see , Active MQ
uses the JMS which has an API to Java as well as other open source
technologies. Therefore, in future we need not worry about interface to or
from ActiveMQ while performing analytics. Does kafka have any feature like
this where it can use a messaging service which can handle wide range of
input types and also have connectors to various other systems?
2. As compared to network of brokers concept in ActiveMQ do we have
something like that in kafka also??
Thanks,
Siddharth Ubale
Re: Data inputs for Kafka.
Posted by siddharth ubale <si...@gmail.com>.
Thnks sharninder!!!
You kinda nailed it for me ;-)
On Thu, Aug 21, 2014 at 6:15 PM, Sharninder <sh...@gmail.com> wrote:
> You can write your producer in c# (
> https://github.com/kafka-dev/kafka/blob/master/clients/csharp/README.md),
> java, python or a number of other languages.
>
> This of Kafka as a separate system that is waiting to receive data in a
> particular format. The language used to send that data doesn't matter.
>
> You can even abstract the producer with a, for eg, REST service. That way
> your applications only need to send a REST request to the producer and
> it'll parse it and do the needful.
>
>
>
> On Thu, Aug 21, 2014 at 3:59 PM, siddharth ubale <
> siddharth.ubale@gmail.com>
> wrote:
>
> > Hi ,
> >
> > In continuity to this thread, can you let me know if i will have to
> write
> > a kafka producer in C# if i have to get data from c# application onto the
> > kafka bus or it will be handled by the native java producer API?? and
> like
> > wise for any system generating data which is using any other technology
> > other than java??
> >
> > Thanks,
> > Siddharth Ubale
> >
> >
> > On Thu, Aug 21, 2014 at 11:48 AM, Sharninder <sh...@gmail.com>
> wrote:
> >
> > > there are kafka producers/consumers in a lot of languages. There is an
> > > Ecosystem page somewhere in the wiki. Take a look at that.
> > >
> > > Finally, I think you should experiment with both activemq and kafka and
> > > then take a decision. Don't try to find a solution and fit your problem
> > to
> > > it, should be the other way round.
> > >
> > > --
> > > Sharninder
> > >
> > >
> > >
> > > On Thu, Aug 21, 2014 at 11:40 AM, siddharth ubale <
> > > siddharth.ubale@gmail.com
> > > > wrote:
> > >
> > > > Hi ,
> > > > Thanks for the quick follow up Philip.
> > > > Also , can you let me know whether the kafka implementation can be as
> > > > versatile as ActiveMQ with regards to connectivity?
> > > > I mean to say a web-app generating transactional data in a data base
> > from
> > > > java webapp, ruby web app or .net web app .... can kafka talk to
> these
> > > with
> > > > ease as ActiveMQ does or any AMQP protocol messaging system does?
> > > > finally, i would also like to know whether 0.9.2 incubating can be
> used
> > > in
> > > > production
> > > >
> > > > Thanks,
> > > > Siddharth
> > > >
> > > >
> > > > On Wed, Aug 20, 2014 at 9:50 PM, Philip O'Toole <
> > > > philip.otoole@yahoo.com.invalid> wrote:
> > > >
> > > > > Kafka can ingest any kind of data, and connect to many types of
> > > systems.
> > > > > Much work exists in this area already, for hooking a wide variety
> of
> > > > > systems to Kafka. If your system isn't supported, then you write a
> > > Kafka
> > > > > Producer to pull (or receive) messages from your system, and write
> > them
> > > > to
> > > > > Kafka. Take a look at the Ecosystem page:
> > > > >
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
> > > > >
> > > > >
> > > > > Philip
> > > > >
> > > > >
> > > > > -----------------------------------------
> > > > > http://www.philipotoole.com
> > > > >
> > > > >
> > > > > On Wednesday, August 20, 2014 5:05 AM, siddharth ubale <
> > > > > siddharth.ubale@gmail.com> wrote:
> > > > >
> > > > >
> > > > >
> > > > > Hi guys,
> > > > >
> > > > > We are implementing a messaging system in order to perform Real
> time
> > > > > analytics. However, we are confused on using ActiveMQ or Apache
> > Kafka.
> > > > The
> > > > > point of worry for us are :
> > > > >
> > > > > 1. What kind of data can be ingested by Apache kafka? If we see ,
> > > Active
> > > > MQ
> > > > > uses the JMS which has an API to Java as well as other open source
> > > > > technologies. Therefore, in future we need not worry about
> interface
> > to
> > > > or
> > > > > from ActiveMQ while performing analytics. Does kafka have any
> feature
> > > > like
> > > > > this where it can use a messaging service which can handle wide
> range
> > > of
> > > > > input types and also have connectors to various other systems?
> > > > >
> > > > > 2. As compared to network of brokers concept in ActiveMQ do we have
> > > > > something like that in kafka also??
> > > > >
> > > > > Thanks,
> > > > > Siddharth Ubale
> > > > >
> > > >
> > >
> >
>
Re: Data inputs for Kafka.
Posted by Sharninder <sh...@gmail.com>.
You can write your producer in c# (
https://github.com/kafka-dev/kafka/blob/master/clients/csharp/README.md),
java, python or a number of other languages.
This of Kafka as a separate system that is waiting to receive data in a
particular format. The language used to send that data doesn't matter.
You can even abstract the producer with a, for eg, REST service. That way
your applications only need to send a REST request to the producer and
it'll parse it and do the needful.
On Thu, Aug 21, 2014 at 3:59 PM, siddharth ubale <si...@gmail.com>
wrote:
> Hi ,
>
> In continuity to this thread, can you let me know if i will have to write
> a kafka producer in C# if i have to get data from c# application onto the
> kafka bus or it will be handled by the native java producer API?? and like
> wise for any system generating data which is using any other technology
> other than java??
>
> Thanks,
> Siddharth Ubale
>
>
> On Thu, Aug 21, 2014 at 11:48 AM, Sharninder <sh...@gmail.com> wrote:
>
> > there are kafka producers/consumers in a lot of languages. There is an
> > Ecosystem page somewhere in the wiki. Take a look at that.
> >
> > Finally, I think you should experiment with both activemq and kafka and
> > then take a decision. Don't try to find a solution and fit your problem
> to
> > it, should be the other way round.
> >
> > --
> > Sharninder
> >
> >
> >
> > On Thu, Aug 21, 2014 at 11:40 AM, siddharth ubale <
> > siddharth.ubale@gmail.com
> > > wrote:
> >
> > > Hi ,
> > > Thanks for the quick follow up Philip.
> > > Also , can you let me know whether the kafka implementation can be as
> > > versatile as ActiveMQ with regards to connectivity?
> > > I mean to say a web-app generating transactional data in a data base
> from
> > > java webapp, ruby web app or .net web app .... can kafka talk to these
> > with
> > > ease as ActiveMQ does or any AMQP protocol messaging system does?
> > > finally, i would also like to know whether 0.9.2 incubating can be used
> > in
> > > production
> > >
> > > Thanks,
> > > Siddharth
> > >
> > >
> > > On Wed, Aug 20, 2014 at 9:50 PM, Philip O'Toole <
> > > philip.otoole@yahoo.com.invalid> wrote:
> > >
> > > > Kafka can ingest any kind of data, and connect to many types of
> > systems.
> > > > Much work exists in this area already, for hooking a wide variety of
> > > > systems to Kafka. If your system isn't supported, then you write a
> > Kafka
> > > > Producer to pull (or receive) messages from your system, and write
> them
> > > to
> > > > Kafka. Take a look at the Ecosystem page:
> > > >
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
> > > >
> > > >
> > > > Philip
> > > >
> > > >
> > > > -----------------------------------------
> > > > http://www.philipotoole.com
> > > >
> > > >
> > > > On Wednesday, August 20, 2014 5:05 AM, siddharth ubale <
> > > > siddharth.ubale@gmail.com> wrote:
> > > >
> > > >
> > > >
> > > > Hi guys,
> > > >
> > > > We are implementing a messaging system in order to perform Real time
> > > > analytics. However, we are confused on using ActiveMQ or Apache
> Kafka.
> > > The
> > > > point of worry for us are :
> > > >
> > > > 1. What kind of data can be ingested by Apache kafka? If we see ,
> > Active
> > > MQ
> > > > uses the JMS which has an API to Java as well as other open source
> > > > technologies. Therefore, in future we need not worry about interface
> to
> > > or
> > > > from ActiveMQ while performing analytics. Does kafka have any feature
> > > like
> > > > this where it can use a messaging service which can handle wide range
> > of
> > > > input types and also have connectors to various other systems?
> > > >
> > > > 2. As compared to network of brokers concept in ActiveMQ do we have
> > > > something like that in kafka also??
> > > >
> > > > Thanks,
> > > > Siddharth Ubale
> > > >
> > >
> >
>
Re: Data inputs for Kafka.
Posted by siddharth ubale <si...@gmail.com>.
Hi ,
In continuity to this thread, can you let me know if i will have to write
a kafka producer in C# if i have to get data from c# application onto the
kafka bus or it will be handled by the native java producer API?? and like
wise for any system generating data which is using any other technology
other than java??
Thanks,
Siddharth Ubale
On Thu, Aug 21, 2014 at 11:48 AM, Sharninder <sh...@gmail.com> wrote:
> there are kafka producers/consumers in a lot of languages. There is an
> Ecosystem page somewhere in the wiki. Take a look at that.
>
> Finally, I think you should experiment with both activemq and kafka and
> then take a decision. Don't try to find a solution and fit your problem to
> it, should be the other way round.
>
> --
> Sharninder
>
>
>
> On Thu, Aug 21, 2014 at 11:40 AM, siddharth ubale <
> siddharth.ubale@gmail.com
> > wrote:
>
> > Hi ,
> > Thanks for the quick follow up Philip.
> > Also , can you let me know whether the kafka implementation can be as
> > versatile as ActiveMQ with regards to connectivity?
> > I mean to say a web-app generating transactional data in a data base from
> > java webapp, ruby web app or .net web app .... can kafka talk to these
> with
> > ease as ActiveMQ does or any AMQP protocol messaging system does?
> > finally, i would also like to know whether 0.9.2 incubating can be used
> in
> > production
> >
> > Thanks,
> > Siddharth
> >
> >
> > On Wed, Aug 20, 2014 at 9:50 PM, Philip O'Toole <
> > philip.otoole@yahoo.com.invalid> wrote:
> >
> > > Kafka can ingest any kind of data, and connect to many types of
> systems.
> > > Much work exists in this area already, for hooking a wide variety of
> > > systems to Kafka. If your system isn't supported, then you write a
> Kafka
> > > Producer to pull (or receive) messages from your system, and write them
> > to
> > > Kafka. Take a look at the Ecosystem page:
> > >
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
> > >
> > >
> > > Philip
> > >
> > >
> > > -----------------------------------------
> > > http://www.philipotoole.com
> > >
> > >
> > > On Wednesday, August 20, 2014 5:05 AM, siddharth ubale <
> > > siddharth.ubale@gmail.com> wrote:
> > >
> > >
> > >
> > > Hi guys,
> > >
> > > We are implementing a messaging system in order to perform Real time
> > > analytics. However, we are confused on using ActiveMQ or Apache Kafka.
> > The
> > > point of worry for us are :
> > >
> > > 1. What kind of data can be ingested by Apache kafka? If we see ,
> Active
> > MQ
> > > uses the JMS which has an API to Java as well as other open source
> > > technologies. Therefore, in future we need not worry about interface to
> > or
> > > from ActiveMQ while performing analytics. Does kafka have any feature
> > like
> > > this where it can use a messaging service which can handle wide range
> of
> > > input types and also have connectors to various other systems?
> > >
> > > 2. As compared to network of brokers concept in ActiveMQ do we have
> > > something like that in kafka also??
> > >
> > > Thanks,
> > > Siddharth Ubale
> > >
> >
>
Re: Data inputs for Kafka.
Posted by Sharninder <sh...@gmail.com>.
there are kafka producers/consumers in a lot of languages. There is an
Ecosystem page somewhere in the wiki. Take a look at that.
Finally, I think you should experiment with both activemq and kafka and
then take a decision. Don't try to find a solution and fit your problem to
it, should be the other way round.
--
Sharninder
On Thu, Aug 21, 2014 at 11:40 AM, siddharth ubale <siddharth.ubale@gmail.com
> wrote:
> Hi ,
> Thanks for the quick follow up Philip.
> Also , can you let me know whether the kafka implementation can be as
> versatile as ActiveMQ with regards to connectivity?
> I mean to say a web-app generating transactional data in a data base from
> java webapp, ruby web app or .net web app .... can kafka talk to these with
> ease as ActiveMQ does or any AMQP protocol messaging system does?
> finally, i would also like to know whether 0.9.2 incubating can be used in
> production
>
> Thanks,
> Siddharth
>
>
> On Wed, Aug 20, 2014 at 9:50 PM, Philip O'Toole <
> philip.otoole@yahoo.com.invalid> wrote:
>
> > Kafka can ingest any kind of data, and connect to many types of systems.
> > Much work exists in this area already, for hooking a wide variety of
> > systems to Kafka. If your system isn't supported, then you write a Kafka
> > Producer to pull (or receive) messages from your system, and write them
> to
> > Kafka. Take a look at the Ecosystem page:
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
> >
> >
> > Philip
> >
> >
> > -----------------------------------------
> > http://www.philipotoole.com
> >
> >
> > On Wednesday, August 20, 2014 5:05 AM, siddharth ubale <
> > siddharth.ubale@gmail.com> wrote:
> >
> >
> >
> > Hi guys,
> >
> > We are implementing a messaging system in order to perform Real time
> > analytics. However, we are confused on using ActiveMQ or Apache Kafka.
> The
> > point of worry for us are :
> >
> > 1. What kind of data can be ingested by Apache kafka? If we see , Active
> MQ
> > uses the JMS which has an API to Java as well as other open source
> > technologies. Therefore, in future we need not worry about interface to
> or
> > from ActiveMQ while performing analytics. Does kafka have any feature
> like
> > this where it can use a messaging service which can handle wide range of
> > input types and also have connectors to various other systems?
> >
> > 2. As compared to network of brokers concept in ActiveMQ do we have
> > something like that in kafka also??
> >
> > Thanks,
> > Siddharth Ubale
> >
>
Re: Data inputs for Kafka.
Posted by siddharth ubale <si...@gmail.com>.
Hi ,
Thanks for the quick follow up Philip.
Also , can you let me know whether the kafka implementation can be as
versatile as ActiveMQ with regards to connectivity?
I mean to say a web-app generating transactional data in a data base from
java webapp, ruby web app or .net web app .... can kafka talk to these with
ease as ActiveMQ does or any AMQP protocol messaging system does?
finally, i would also like to know whether 0.9.2 incubating can be used in
production
Thanks,
Siddharth
On Wed, Aug 20, 2014 at 9:50 PM, Philip O'Toole <
philip.otoole@yahoo.com.invalid> wrote:
> Kafka can ingest any kind of data, and connect to many types of systems.
> Much work exists in this area already, for hooking a wide variety of
> systems to Kafka. If your system isn't supported, then you write a Kafka
> Producer to pull (or receive) messages from your system, and write them to
> Kafka. Take a look at the Ecosystem page:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
>
>
> Philip
>
>
> -----------------------------------------
> http://www.philipotoole.com
>
>
> On Wednesday, August 20, 2014 5:05 AM, siddharth ubale <
> siddharth.ubale@gmail.com> wrote:
>
>
>
> Hi guys,
>
> We are implementing a messaging system in order to perform Real time
> analytics. However, we are confused on using ActiveMQ or Apache Kafka. The
> point of worry for us are :
>
> 1. What kind of data can be ingested by Apache kafka? If we see , Active MQ
> uses the JMS which has an API to Java as well as other open source
> technologies. Therefore, in future we need not worry about interface to or
> from ActiveMQ while performing analytics. Does kafka have any feature like
> this where it can use a messaging service which can handle wide range of
> input types and also have connectors to various other systems?
>
> 2. As compared to network of brokers concept in ActiveMQ do we have
> something like that in kafka also??
>
> Thanks,
> Siddharth Ubale
>
Re: Data inputs for Kafka.
Posted by Philip O'Toole <ph...@yahoo.com.INVALID>.
Kafka can ingest any kind of data, and connect to many types of systems. Much work exists in this area already, for hooking a wide variety of systems to Kafka. If your system isn't supported, then you write a Kafka Producer to pull (or receive) messages from your system, and write them to Kafka. Take a look at the Ecosystem page:
https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
Philip
-----------------------------------------
http://www.philipotoole.com
On Wednesday, August 20, 2014 5:05 AM, siddharth ubale <si...@gmail.com> wrote:
Hi guys,
We are implementing a messaging system in order to perform Real time
analytics. However, we are confused on using ActiveMQ or Apache Kafka. The
point of worry for us are :
1. What kind of data can be ingested by Apache kafka? If we see , Active MQ
uses the JMS which has an API to Java as well as other open source
technologies. Therefore, in future we need not worry about interface to or
from ActiveMQ while performing analytics. Does kafka have any feature like
this where it can use a messaging service which can handle wide range of
input types and also have connectors to various other systems?
2. As compared to network of brokers concept in ActiveMQ do we have
something like that in kafka also??
Thanks,
Siddharth Ubale