Comparison of kafka and Hedwig?

[Thomas Koch <th...@koch.ro>]

Flavio Junqueira:
> Thomas, Did you mean to say Hedwig instead of BookKeeper?
Oh sh..ugar yeah. Thanks. Start over again:

I've just had a look at the kafka slides[1] from January HUG. It seems to me, 
that Hedwig[2] and kafka are quite similar in there problem space. Is that 
so? What are notable differences?
(Kafka is written in scala and therefor must be a lot cooler :-)

[1] <http://developer.yahoo.com/blogs/hadoop/posts/2011/02/hadoop-user-group-
january-2011-recap/>
[2] http://cwiki.apache.org/confluence/display/ZOOKEEPER/HedWig

Thomas Koch, http://www.koch.ro

Re: Comparison of kafka and Hedwig?

[Benjamin Reed <br...@yahoo-inc.com>]

i'm not sure cross posting is such a good idea, it is probably better to 
discuss these comparisons relative to a project's viewpoint.

the problem space is similar, and i imagine as kafka develops the 
implementations may become even closer.

there are a couple of obvious things:

1) hedwig has strong durability guarantees. kafka can lose data due to 
failures.
2) hedwig was designed for lots of topics (100,000s) with low fan out 
(few subscribers/publishers). i think kafka is designed for a few topics 
with lots of subscribers and publishers.
3) hedwig tracks subscribers progress for gc of publishes. kafka uses 
time based gc.
4) hedwig will replay messages to subscribers starting from the last 
message they explicitly consumed. kafka allows subscribers to replay 
messages that they have already consumed.

there are probably others.

i really like the kafka design choices made for 3 and 4. hedwig will 
work on scaling to more subscribers/publishers per topic. i imagine, if 
needed, kafka will work on their durability guarantees and support for 
large number of topics.

ben

On 02/10/2011 01:27 AM, Thomas Koch wrote:
> Flavio Junqueira:
>> Thomas, Did you mean to say Hedwig instead of BookKeeper?
> Oh sh..ugar yeah. Thanks. Start over again:
>
> I've just had a look at the kafka slides[1] from January HUG. It seems to me,
> that Hedwig[2] and kafka are quite similar in there problem space. Is that
> so? What are notable differences?
> (Kafka is written in scala and therefor must be a lot cooler :-)
>
> [1]<http://developer.yahoo.com/blogs/hadoop/posts/2011/02/hadoop-user-group-
> january-2011-recap/>
> [2] http://cwiki.apache.org/confluence/display/ZOOKEEPER/HedWig
>
> Thomas Koch, http://www.koch.ro