Re: Role of Producer

[Michael Freeman <mi...@gmail.com>]

Could you write them a client that uses the Kafka producer?
You could also write some restful services that send the data to kafka.
If they use MQ you could listen to MQ and send to Kafka.



On Wed, Jul 13, 2016 at 9:31 PM, Luo, Chao <ch...@here.com> wrote:

> Dear Kafka guys,
>
> I just started to build up a Kafka system two weeks ago. Here I have a
> question about how to design/implement the producer.
>
> In my system, there are many data providers. I need to collect real-time
> data from them and store it in a NoSQL database. The problem is that
> different data providers have their own Java APIs, and they will not use
> Kafka-client to send data directly to my Kafka servers. So I need to first
> collect data from them and feed it to the Kafka servers. I guess I need to
> finish data acquisition in the Producers. My question is that there are a
> great number of data providers so I also need a lot of producers??? Or is
> there any more efficient ways to deal with it?
>
> Best,
> Chao
>

Re: Role of Producer

[Michael Freeman <mi...@gmail.com>]

MQ was just short hand for IBM MQ or Active MQ etc etc

On Wed, Jul 13, 2016 at 9:42 PM, Luo, Chao <ch...@here.com> wrote:

> Hi thanks!
>
> Yes, I agree it is the best if they can use a kafka producer client. But I
> need to discuss with them if they will accept that.
>
> Btw, what is MQ?
>
>
>
> -----Original Message-----
> From: Michael Freeman [mailto:mikfreeman@gmail.com]
> Sent: Wednesday, July 13, 2016 3:36 PM
> To: users@kafka.apache.org
> Subject: Re: Role of Producer
>
> Could you write them a client that uses the Kafka producer?
> You could also write some restful services that send the data to kafka.
> If they use MQ you could listen to MQ and send to Kafka.
>
>
>
> On Wed, Jul 13, 2016 at 9:31 PM, Luo, Chao <ch...@here.com> wrote:
>
> > Dear Kafka guys,
> >
> > I just started to build up a Kafka system two weeks ago. Here I have a
> > question about how to design/implement the producer.
> >
> > In my system, there are many data providers. I need to collect
> > real-time data from them and store it in a NoSQL database. The problem
> > is that different data providers have their own Java APIs, and they
> > will not use Kafka-client to send data directly to my Kafka servers.
> > So I need to first collect data from them and feed it to the Kafka
> > servers. I guess I need to finish data acquisition in the Producers.
> > My question is that there are a great number of data providers so I
> > also need a lot of producers??? Or is there any more efficient ways to
> deal with it?
> >
> > Best,
> > Chao
> >
>

RE: Role of Producer

["Luo, Chao" <ch...@here.com>]

Hi thanks!

Yes, I agree it is the best if they can use a kafka producer client. But I need to discuss with them if they will accept that. 

Btw, what is MQ?



-----Original Message-----
From: Michael Freeman [mailto:mikfreeman@gmail.com] 
Sent: Wednesday, July 13, 2016 3:36 PM
To: users@kafka.apache.org
Subject: Re: Role of Producer

Could you write them a client that uses the Kafka producer?
You could also write some restful services that send the data to kafka.
If they use MQ you could listen to MQ and send to Kafka.



On Wed, Jul 13, 2016 at 9:31 PM, Luo, Chao <ch...@here.com> wrote:

> Dear Kafka guys,
>
> I just started to build up a Kafka system two weeks ago. Here I have a 
> question about how to design/implement the producer.
>
> In my system, there are many data providers. I need to collect 
> real-time data from them and store it in a NoSQL database. The problem 
> is that different data providers have their own Java APIs, and they 
> will not use Kafka-client to send data directly to my Kafka servers. 
> So I need to first collect data from them and feed it to the Kafka 
> servers. I guess I need to finish data acquisition in the Producers. 
> My question is that there are a great number of data providers so I 
> also need a lot of producers??? Or is there any more efficient ways to deal with it?
>
> Best,
> Chao
>