You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by Yidong Cui <mo...@gmail.com> on 2013/05/31 14:27:44 UTC
Design new futures for Hedwig system——add "message queue model"
As we know, Hedwig system is a topic-based pub/sub system. One
subscriber in Hedwig system is simply one client, however, under the
environment of cloud-computing and distributed system, one subscriber may
not be just one client but a cluster with multiple clients in it. Besides,
original Hedwig system just has pub/sub model but actually many enterprises
ask a demand for message queue model.
To satisfy this, we want to modify original Hedwig system by
adding *message
queue model*. In the modified Hedwig system, not only pub/sub model but
also message queue model is available. The attachment of this email is
our design document. We want to discuss with you all, and waiting for
your valuable advice,thank you!
Re: Design new futures for Hedwig system——add "message queue model"
Posted by Flavio Junqueira <fp...@yahoo.com>.
There is an e-mail thread though:
http://mail-archives.apache.org/mod_mbox/zookeeper-bookkeeper-user/201302.mbox/%3CCD5117B7.AA88%25jiannan@yahoo-inc.com%3E
-Flavio
On May 31, 2013, at 3:19 PM, Ivan Kelly <iv...@apache.org> wrote:
> The mailing list seems to have blocked the attachment. Could you put
> the document somewhere we could download it? Perhaps you could create
> a JIRA for it on https://issues.apache.org/jira/browse/BOOKKEEPER. I
> thought we did have a jira for queue semantic in hedwig, but a quick
> search revealed nothing.
>
> -Ivan
>
> On Fri, May 31, 2013 at 08:27:44PM +0800, Yidong Cui wrote:
>> As we know, Hedwig system is a topic-based pub/sub system. One
>> subscriber in Hedwig system is simply one client, however, under the
>> environment of cloud-computing and distributed system, one subscriber may
>> not be just one client but a cluster with multiple clients in it. Besides,
>> original Hedwig system just has pub/sub model but actually many enterprises
>> ask a demand for message queue model.
>> To satisfy this, we want to modify original Hedwig system by
>> adding *message
>> queue model*. In the modified Hedwig system, not only pub/sub model but
>> also message queue model is available. The attachment of this email is
>> our design document. We want to discuss with you all, and waiting for
>> your valuable advice,thank you!
>
Re: Design new futures for Hedwig system——add "message queue model"
Posted by Ivan Kelly <iv...@apache.org>.
The mailing list seems to have blocked the attachment. Could you put
the document somewhere we could download it? Perhaps you could create
a JIRA for it on https://issues.apache.org/jira/browse/BOOKKEEPER. I
thought we did have a jira for queue semantic in hedwig, but a quick
search revealed nothing.
-Ivan
On Fri, May 31, 2013 at 08:27:44PM +0800, Yidong Cui wrote:
> As we know, Hedwig system is a topic-based pub/sub system. One
> subscriber in Hedwig system is simply one client, however, under the
> environment of cloud-computing and distributed system, one subscriber may
> not be just one client but a cluster with multiple clients in it. Besides,
> original Hedwig system just has pub/sub model but actually many enterprises
> ask a demand for message queue model.
> To satisfy this, we want to modify original Hedwig system by
> adding *message
> queue model*. In the modified Hedwig system, not only pub/sub model but
> also message queue model is available. The attachment of this email is
> our design document. We want to discuss with you all, and waiting for
> your valuable advice,thank you!
Re: Design new futures for Hedwig system——add "message queue model"
Posted by Sijie Guo <gu...@gmail.com>.
Thanks for providing the design document. Although I don't know how you
would implement this design, from my perspective, the queue model is quite
different from PUB/SUB model in Hedwig now. If you just extends the current
components, it might ends up mixing too many concepts together, which make
things complicated. So if you could put your prototype in github, it might
be easy for the community to evaluate the complexity.
Below are some comments on your document:
1. "publish process": why introduce "publish id"? publishing an message to
a topic should be independent to other clients? I don't get the point here.
2. most of the consumer design in your part is to fit a queue model into a
pub/sub model, which is not a trivial and straightforward implementation.
And it introduced bunch of other issues in your design.
2.1 the client consume behavior needs to be modified to fit in your ack
model. otherwise, it would cause bad things.
for example, client C1, C2, C3 is consuming from topic T. message 1, 2, ,3,
4, 5, 6 is delivered to them in round-robin. C1, C2 received (1, 4) and (2,
5), they consumed to 4 and 5. C3 failed, 3 and 6 would be redeliver to C1
and C2. C3 might not be sent if C1 and C2 use current consume mechanism in
hedwig client.
2.2 you had to track individual acks in both server side and client side.
missing of individual acks would cause acks or retry queue backup in server
side, which might cause bad gc activities in hub server.
2.3 the per-client flow control is bad, which you had to allocate as much
memory as the number of clients. A growing of clients is easy to attack a
hub server.
-Sijie
On Fri, May 31, 2013 at 5:27 AM, Yidong Cui <mo...@gmail.com> wrote:
> As we know, Hedwig system is a topic-based pub/sub system. One
> subscriber in Hedwig system is simply one client, however, under the
> environment of cloud-computing and distributed system, one subscriber may
> not be just one client but a cluster with multiple clients in it. Besides,
> original Hedwig system just has pub/sub model but actually many enterprises
> ask a demand for message queue model.
> To satisfy this, we want to modify original Hedwig system by adding
> *message queue model*. In the modified Hedwig system, not only pub/sub
> model but also message queue model is available. The attachment of this
> email is our design document. We want to discuss with you all, and
> waiting for your valuable advice,thank you!
>
>
>
Re: Design new futures for Hedwig system‹‹add "message queue model"
Posted by Jiannan Wang <ji...@yahoo-inc.com>.
Hi,
Yannick also ask me about the JIRA for queue model support and I've created a JIRA https://issues.apache.org/jira/browse/BOOKKEEPER-616
> The attachment of this email is our design document. We want to discuss with you all, and waiting for your valuable advice,thank you!
----
I don't see any attachment in your email, could you attach your design document in the JIRA shown above? Thanks.
- Jiannan
From: Yidong Cui <mo...@gmail.com>>
Reply-To: "bookkeeper-dev@zookeeper.apache.org<ma...@zookeeper.apache.org>" <bo...@zookeeper.apache.org>>
Date: Friday, May 31, 2013 8:27 PM
To: Hedwig Project <bo...@zookeeper.apache.org>>
Cc: hedwig 联系人 <gu...@gmail.com>>
Subject: Design new futures for Hedwig system——add "message queue model"
As we know, Hedwig system is a topic-based pub/sub system. One subscriber in Hedwig system is simply one client, however, under the environment of cloud-computing and distributed system, one subscriber may not be just one client but a cluster with multiple clients in it. Besides, original Hedwig system just has pub/sub model but actually many enterprises ask a demand for message queue model.
To satisfy this, we want to modify original Hedwig system by adding message queue model. In the modified Hedwig system, not only pub/sub model but also message queue model is available. The attachment of this email is our design document. We want to discuss with you all, and waiting for your valuable advice,thank you!