notes from parsing the protocol in erlang + 1 wiki feedback

[Bosky <bo...@helpshift.com>]

Hello all,

I've enjoyed reading the docs, and have feedback that might save other
client contributors from hours of staring at binary data and tcpdump.

1) Parsing kafka responses with erlang
If you're interested in erlang and the kafka wire protocol - here are my
notes from going from never working with kafka, to patching an erlang kafka
client to read metadata from a 0.8 broker

https://coderwall.com/p/1lyfxg

2) Wiki suggestion
After metadata, i moved to generating produce requests, but spend 2 days to
find a subtle distinction that IMHO needs a clear note in the wiki.

"Arrays are encoded {length(A),A} MessageSets are spec'd as Array in BNF
but encoded {byte_size(M),M}"

There are references that suggest message sets don't have the length, but I
think it's not highlighted enough. I would even go far as to say that
MessageSets shouldnt be considered an array as well.


hat-tip to Snaps on #apache-kafka for lending a ear, and I look forward to
continue contributing to the ekafka erlang kafka client.

~Bosky | @bhaskerkode | Helpshift Inc.

Re: notes from parsing the protocol in erlang + 1 wiki feedback

[Jay Kreps <ja...@gmail.com>]

Hey Bosky,

This is super helpful, I'll make those changes on the wiki.

-Jay


On Fri, Mar 21, 2014 at 12:01 PM, Bosky <bo...@helpshift.com> wrote:

> Hello all,
>
> I've enjoyed reading the docs, and have feedback that might save other
> client contributors from hours of staring at binary data and tcpdump.
>
> 1) Parsing kafka responses with erlang
> If you're interested in erlang and the kafka wire protocol - here are my
> notes from going from never working with kafka, to patching an erlang kafka
> client to read metadata from a 0.8 broker
>
> https://coderwall.com/p/1lyfxg
>
> 2) Wiki suggestion
> After metadata, i moved to generating produce requests, but spend 2 days to
> find a subtle distinction that IMHO needs a clear note in the wiki.
>
> "Arrays are encoded {length(A),A} MessageSets are spec'd as Array in BNF
> but encoded {byte_size(M),M}"
>
> There are references that suggest message sets don't have the length, but I
> think it's not highlighted enough. I would even go far as to say that
> MessageSets shouldnt be considered an array as well.
>
>
> hat-tip to Snaps on #apache-kafka for lending a ear, and I look forward to
> continue contributing to the ekafka erlang kafka client.
>
> ~Bosky | @bhaskerkode | Helpshift Inc.
>