You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Hussein Baghdadi <hu...@hotmail.com> on 2012/06/27 14:42:40 UTC
Sending web traffic data to Kafka
Hi,I want to send web traffic data to Kafka as events, something like:<page, timestamp, user-agent ....>And some other usual suspects.1. Which languages can send events to Kafka (in addition to Java)?2. Any guide how to write an encoder for my previous event structure?Thanks for help and time.
Re: Sending web traffic data to Kafka
Posted by Neha Narkhede <ne...@gmail.com>.
Hussein,
You can use MessagePack to send data to Kafka. As long as you can
convert an object to a byte array, you can send data to Kafka. For an
example, please see step 8 of the producer code section of the
quickstart -
http://incubator.apache.org/kafka/quickstart.html
HTH,
Neha
On Wed, Jun 27, 2012 at 8:01 AM, Jun Rao <ju...@gmail.com> wrote:
> https://github.com/apache/kafka/tree/trunk/clients
>
> Jun
>
> On Wed, Jun 27, 2012 at 7:54 AM, Hussein Baghdadi <hu...@hotmail.com>wrote:
>
>>
>> Which contrib? this one https://github.com/apache/kafka/tree/trunk/contrib?Nothing but Hadoop producers and consumers.
>> Can we send MessagePack http://msgpack.org/ to Kafka?
>> Thank you.
>>
>> > Date: Wed, 27 Jun 2012 07:24:00 -0700
>> > Subject: Re: Sending web traffic data to Kafka
>> > From: junrao@gmail.com
>> > To: kafka-users@incubator.apache.org
>> >
>> > There are a few non-java clients (mostly producers) in contrib that you
>> can
>> > try (not exactly sure how up to date each of them is). As for encoding,
>> > anything like JSON or avro should be fine.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Wed, Jun 27, 2012 at 5:42 AM, Hussein Baghdadi <
>> hubaghdadi@hotmail.com>wrote:
>> >
>> > >
>> > > Hi,I want to send web traffic data to Kafka as events, something
>> > > like:<page, timestamp, user-agent ....>And some other usual suspects.1.
>> > > Which languages can send events to Kafka (in addition to Java)?2. Any
>> guide
>> > > how to write an encoder for my previous event structure?Thanks for
>> help and
>> > > time.
>>
>>
Re: Sending web traffic data to Kafka
Posted by Jun Rao <ju...@gmail.com>.
https://github.com/apache/kafka/tree/trunk/clients
Jun
On Wed, Jun 27, 2012 at 7:54 AM, Hussein Baghdadi <hu...@hotmail.com>wrote:
>
> Which contrib? this one https://github.com/apache/kafka/tree/trunk/contrib?Nothing but Hadoop producers and consumers.
> Can we send MessagePack http://msgpack.org/ to Kafka?
> Thank you.
>
> > Date: Wed, 27 Jun 2012 07:24:00 -0700
> > Subject: Re: Sending web traffic data to Kafka
> > From: junrao@gmail.com
> > To: kafka-users@incubator.apache.org
> >
> > There are a few non-java clients (mostly producers) in contrib that you
> can
> > try (not exactly sure how up to date each of them is). As for encoding,
> > anything like JSON or avro should be fine.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Jun 27, 2012 at 5:42 AM, Hussein Baghdadi <
> hubaghdadi@hotmail.com>wrote:
> >
> > >
> > > Hi,I want to send web traffic data to Kafka as events, something
> > > like:<page, timestamp, user-agent ....>And some other usual suspects.1.
> > > Which languages can send events to Kafka (in addition to Java)?2. Any
> guide
> > > how to write an encoder for my previous event structure?Thanks for
> help and
> > > time.
>
>
RE: Sending web traffic data to Kafka
Posted by Hussein Baghdadi <hu...@hotmail.com>.
Which contrib? this one https://github.com/apache/kafka/tree/trunk/contrib ?Nothing but Hadoop producers and consumers.
Can we send MessagePack http://msgpack.org/ to Kafka?
Thank you.
> Date: Wed, 27 Jun 2012 07:24:00 -0700
> Subject: Re: Sending web traffic data to Kafka
> From: junrao@gmail.com
> To: kafka-users@incubator.apache.org
>
> There are a few non-java clients (mostly producers) in contrib that you can
> try (not exactly sure how up to date each of them is). As for encoding,
> anything like JSON or avro should be fine.
>
> Thanks,
>
> Jun
>
> On Wed, Jun 27, 2012 at 5:42 AM, Hussein Baghdadi <hu...@hotmail.com>wrote:
>
> >
> > Hi,I want to send web traffic data to Kafka as events, something
> > like:<page, timestamp, user-agent ....>And some other usual suspects.1.
> > Which languages can send events to Kafka (in addition to Java)?2. Any guide
> > how to write an encoder for my previous event structure?Thanks for help and
> > time.
Re: Sending web traffic data to Kafka
Posted by Tim Lossen <ti...@lossen.de>.
hi hussein,
we use an nginx "after request hook" written in perl that uses the kafka perl client to push events into kafka. only a few lines of code, and works much better than it sounds ;)
cheers
tim
On 2012-06-27, at 07:24 , Jun Rao wrote:
> There are a few non-java clients (mostly producers) in contrib that you can
> try (not exactly sure how up to date each of them is). As for encoding,
> anything like JSON or avro should be fine.
>
> Thanks,
>
> Jun
>
> On Wed, Jun 27, 2012 at 5:42 AM, Hussein Baghdadi <hu...@hotmail.com>wrote:
>
>>
>> Hi,I want to send web traffic data to Kafka as events, something
>> like:<page, timestamp, user-agent ....>And some other usual suspects.1.
>> Which languages can send events to Kafka (in addition to Java)?2. Any guide
>> how to write an encoder for my previous event structure?Thanks for help and
>> time.
--
http://tim.lossen.de
Re: Sending web traffic data to Kafka
Posted by Jun Rao <ju...@gmail.com>.
There are a few non-java clients (mostly producers) in contrib that you can
try (not exactly sure how up to date each of them is). As for encoding,
anything like JSON or avro should be fine.
Thanks,
Jun
On Wed, Jun 27, 2012 at 5:42 AM, Hussein Baghdadi <hu...@hotmail.com>wrote:
>
> Hi,I want to send web traffic data to Kafka as events, something
> like:<page, timestamp, user-agent ....>And some other usual suspects.1.
> Which languages can send events to Kafka (in addition to Java)?2. Any guide
> how to write an encoder for my previous event structure?Thanks for help and
> time.