Kafka PHP Client (+RESTful proxy)

[Michal Hariš <mi...@gmail.com>]

Hi all,

I have a long interest in pub-sub and have written several variants of
proxies/bridges to JMS ( ActiveMQ, HornetQ and some others) in Java and I
share the motivations behind Kafka project (esp. the idea of keeping the
consumption state at the consumer side really speaks to my heart and also
the focus on persistence).

So I started writing some proofs of concept for the 50GB of daily event
logs I work with and  which I would like to turn into a real time streams
rather than offline hive processes. I often use php for proof-of-concepts
because it's easy to write and throw away but after a few hours of using
and looking at the Kafka PHP Client which is in the incubator trunk I have
ended up writing and alternative because the existing client does few
things which don't work for me ( mainly the fact that it loads the entire
fetch-response into memory and then processes internal buffer plus few
other things.. ) and doesn't do few things I need ( mainly doesn't try to
deal with the long offsets and  decompression )

Please have a look if you're interested:
http://svn.xp-dev.com/svn/kafkaphp/trunk/

(Btw, having a robust and simple php api would be an easy step towards a
restful proxy which could be later turned into scala/java underneath once
the restful api would get stable)

Michal

Re: Kafka PHP Client (+RESTful proxy)

[Michal Hariš <mi...@gmail.com>]

Great, the produce and fetch requests are now working with the exception of
snappy compression.

I have used hexadecimal offsets as a way around the php long problem
bacause they are seamless to map from and to 64-bit longs. It works form me
but maybe if you'll find a problem with it, let me know, the pear Bigint
class or GMP extension can be used inside the Kafka_Offset class instead
but i prefer no extra dependencies if possible.

I'll finish the OffsetsRequest unix timestamps and add
MultiFetch/MultiProduce sometimes this week.
On Jul 16, 2012 7:11 PM, "Jonathan Creasy" <jc...@box.com> wrote:

> Awesome, I built a RESTful Proxy last week so I will integrate your library
> and check that out. I'll publish my Proxy code soon, it is getting thrashed
> internally right now.
> On Jul 15, 2012 4:21 PM, "Michal Hariš" <mi...@gmail.com> wrote:
>
> > Hi all,
> >
> > I have a long interest in pub-sub and have written several variants of
> > proxies/bridges to JMS ( ActiveMQ, HornetQ and some others) in Java and I
> > share the motivations behind Kafka project (esp. the idea of keeping the
> > consumption state at the consumer side really speaks to my heart and also
> > the focus on persistence).
> >
> > So I started writing some proofs of concept for the 50GB of daily event
> > logs I work with and  which I would like to turn into a real time streams
> > rather than offline hive processes. I often use php for proof-of-concepts
> > because it's easy to write and throw away but after a few hours of using
> > and looking at the Kafka PHP Client which is in the incubator trunk I
> have
> > ended up writing and alternative because the existing client does few
> > things which don't work for me ( mainly the fact that it loads the entire
> > fetch-response into memory and then processes internal buffer plus few
> > other things.. ) and doesn't do few things I need ( mainly doesn't try to
> > deal with the long offsets and  decompression )
> >
> > Please have a look if you're interested:
> > http://svn.xp-dev.com/svn/kafkaphp/trunk/
> >
> > (Btw, having a robust and simple php api would be an easy step towards a
> > restful proxy which could be later turned into scala/java underneath once
> > the restful api would get stable)
> >
> > Michal
> >
>

Re: Kafka PHP Client (+RESTful proxy)

[Jonathan Creasy <jc...@box.com>]

Awesome, I built a RESTful Proxy last week so I will integrate your library
and check that out. I'll publish my Proxy code soon, it is getting thrashed
internally right now.
On Jul 15, 2012 4:21 PM, "Michal Hariš" <mi...@gmail.com> wrote:

> Hi all,
>
> I have a long interest in pub-sub and have written several variants of
> proxies/bridges to JMS ( ActiveMQ, HornetQ and some others) in Java and I
> share the motivations behind Kafka project (esp. the idea of keeping the
> consumption state at the consumer side really speaks to my heart and also
> the focus on persistence).
>
> So I started writing some proofs of concept for the 50GB of daily event
> logs I work with and  which I would like to turn into a real time streams
> rather than offline hive processes. I often use php for proof-of-concepts
> because it's easy to write and throw away but after a few hours of using
> and looking at the Kafka PHP Client which is in the incubator trunk I have
> ended up writing and alternative because the existing client does few
> things which don't work for me ( mainly the fact that it loads the entire
> fetch-response into memory and then processes internal buffer plus few
> other things.. ) and doesn't do few things I need ( mainly doesn't try to
> deal with the long offsets and  decompression )
>
> Please have a look if you're interested:
> http://svn.xp-dev.com/svn/kafkaphp/trunk/
>
> (Btw, having a robust and simple php api would be an easy step towards a
> restful proxy which could be later turned into scala/java underneath once
> the restful api would get stable)
>
> Michal
>