You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bookkeeper.apache.org by Yannick Legros <ya...@gmail.com> on 2012/12/20 12:35:55 UTC
hedwig question
Hello,
i currently use rabbitMQ for some messaging treatment, i read docs on
hedwig and i want try it.
But i wonder if hedwig can answer our use case.
configuration of these variables (*default_message_window_size*,*
consume_interval*,*retention_secs*) should be the answer. But i'm not sure.
i want that a message can be treat by only one subscriber.
for example
Subscriber 1 subscribe to topic A
Subscriber 2 subscribe to topic A
publisher post a message M1 on topics A.
publisher post a message M2 on topics A.
THEN
Subscriber 1 consume M1
Subscriber 2 consume M2
OR
Subscriber 1 consume M1 AND M2
Subscriber 2 consume nothing
OR
Subscriber 1 consume M2
Subscriber 2 consume M1
OR
Subscriber 1 consume nothing
Subscriber 2 consume M1 AND M2
BUT NOT
Subscriber 1 consume M1 AND M2
Subscriber 2 consume M1 AND M2
I hope i was clear.
Thx by advance for help, Yannick Legros.
Re: hedwig question
Posted by Flavio Junqueira <fp...@yahoo.com>.
I'm not entirely sure why the subscriber can't subscribe before anything else happens in your system, Yannick. Independent of your requirements, you may also want to consider using ZooKeeper to coordinate the actions of your publishers and subscribers on top of Sijie's suggestion.
-Flavio
On Jan 4, 2013, at 5:01 PM, Yannick Legros <ya...@gmail.com> wrote:
> "for currently, we don't support this semantic in Hedwig. I am doubting why
> not subscribe first before publish. is it difficult in your case?"
>
> => subscribers and publishers are waiting for external events to begin
> their publish/subscribe. i can't be sure of the order that my "Agents"
> detects the external event. So in some cases the publishers starts before
> the subscribers and i can loose few message :-(.
>
> "if to support your semantic, all the messages had to be kept until a
> subscriber arrived. if there is no subscriber, how to deal with those
> published messages? kept them forever?"
>
> => I know that hedwig is a pub/sub framwork and it is not a treatment
> queue, but keep them during a parametrized time would be great
>
> thanks,
>
> Yannick.
>
> 2012/12/29 Sijie Guo <gu...@gmail.com>
>
>> thanks, yannick. I got your idea.
>>
>> for currently, we don't support this semantic in Hedwig. I am doubting why
>> not subscribe first before publish. is it difficult in your case?
>>
>> if to support your semantic, all the messages had to be kept until a
>> subscriber arrived. if there is no subscriber, how to deal with those
>> published messages? kept them forever?
>>
>> -Sijie
>>
>>
>> On Sat, Dec 29, 2012 at 3:58 AM, Yannick Legros <
>> yannick.legros59@gmail.com> wrote:
>>
>>> thanks for your answer sijie.
>>>
>>> what i need is not exactly
>>> "subscribers be aware of messages publish before they subscribe"
>>> but
>>> "be sure that my message be consume by almost one subscriber"
>>>
>>> so if a publisher publish message on topics with no subscribers the first
>>> subscriber who will read the topics will get all published messages before
>>> his subscription.
>>>
>>> hope i was clear.
>>>
>>> Thanks, yannick.
>>>
>>> 2012/12/28 Sijie Guo <gu...@gmail.com>
>>>
>>>>> does this is the default behaviour of hedwig?
>>>>
>>>> A subscriber would receive all the messages published after its first
>>>> time subscribe until it unsubscribe.
>>>>
>>>> so go back to your example:
>>>>
>>>> 1) if the subscriber subscribe T first time, it would only receive M2.
>>>> 2) if the subscriber already subscribe before M1. so the subscribe
>>>> between M1 and M2 is an attach subscribe, so it would receive M1 and M2
>>>> eventually (if it doesn't receive M1 at its first time subscribe)
>>>>
>>>>> can we configure something (in subscription option for exemple) for
>>>> subscribers be aware of messages publish before they subscribe and not
>>>> consumed yet.
>>>>
>>>> Currently Hedwig doesn't support it. But I am just curious why a
>>>> subscriber need to be aware of messages published before they subscribed?
>>>> Is there any special case you are considering?
>>>>
>>>> -Sijie
>>>>
>>>>
>>>> On Fri, Dec 28, 2012 at 3:23 AM, Yannick Legros <
>>>> yannick.legros59@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I have just another question.
>>>>>
>>>>> in this case :
>>>>>
>>>>> publisher post a message M1 on topics T
>>>>> subscriber subscribe to T
>>>>> publisher post a message M2 on topics T
>>>>>
>>>>> when subscriber starts delivery he will get M2 but not M1 because he
>>>>> has subscribed after M1 was publish.
>>>>>
>>>>> does this is the default behaviour of hedwig?
>>>>> can we configure something (in subscription option for exemple) for
>>>>> subscribers be aware of messages publish before they subscribe and not
>>>>> consumed yet.
>>>>>
>>>>> thanks a lot.
>>>>>
>>>>
>>>>
>>>
>>
Re: hedwig question
Posted by Sijie Guo <gu...@gmail.com>.
Yannick,
Seems a producer/consumer queue model might be more suitable for your user
case, I guessed. It would be quite interesting on adapting a queue in
Hedwig or based on BookKeeper. Would look into it.
-Sijie
On Fri, Jan 4, 2013 at 9:01 AM, Yannick Legros
<ya...@gmail.com>wrote:
> "for currently, we don't support this semantic in Hedwig. I am doubting
> why not subscribe first before publish. is it difficult in your case?"
>
> => subscribers and publishers are waiting for external events to begin
> their publish/subscribe. i can't be sure of the order that my "Agents"
> detects the external event. So in some cases the publishers starts before
> the subscribers and i can loose few message :-(.
>
>
> "if to support your semantic, all the messages had to be kept until a
> subscriber arrived. if there is no subscriber, how to deal with those
> published messages? kept them forever?"
>
> => I know that hedwig is a pub/sub framwork and it is not a treatment
> queue, but keep them during a parametrized time would be great
>
> thanks,
>
> Yannick.
>
>
> 2012/12/29 Sijie Guo <gu...@gmail.com>
>
>> thanks, yannick. I got your idea.
>>
>> for currently, we don't support this semantic in Hedwig. I am doubting
>> why not subscribe first before publish. is it difficult in your case?
>>
>> if to support your semantic, all the messages had to be kept until a
>> subscriber arrived. if there is no subscriber, how to deal with those
>> published messages? kept them forever?
>>
>> -Sijie
>>
>>
>> On Sat, Dec 29, 2012 at 3:58 AM, Yannick Legros <
>> yannick.legros59@gmail.com> wrote:
>>
>>> thanks for your answer sijie.
>>>
>>> what i need is not exactly
>>> "subscribers be aware of messages publish before they subscribe"
>>> but
>>> "be sure that my message be consume by almost one subscriber"
>>>
>>> so if a publisher publish message on topics with no subscribers the
>>> first subscriber who will read the topics will get all published messages
>>> before his subscription.
>>>
>>> hope i was clear.
>>>
>>> Thanks, yannick.
>>>
>>> 2012/12/28 Sijie Guo <gu...@gmail.com>
>>>
>>>> > does this is the default behaviour of hedwig?
>>>>
>>>> A subscriber would receive all the messages published after its first
>>>> time subscribe until it unsubscribe.
>>>>
>>>> so go back to your example:
>>>>
>>>> 1) if the subscriber subscribe T first time, it would only receive M2.
>>>> 2) if the subscriber already subscribe before M1. so the subscribe
>>>> between M1 and M2 is an attach subscribe, so it would receive M1 and M2
>>>> eventually (if it doesn't receive M1 at its first time subscribe)
>>>>
>>>> > can we configure something (in subscription option for exemple) for
>>>> subscribers be aware of messages publish before they subscribe and not
>>>> consumed yet.
>>>>
>>>> Currently Hedwig doesn't support it. But I am just curious why a
>>>> subscriber need to be aware of messages published before they subscribed?
>>>> Is there any special case you are considering?
>>>>
>>>> -Sijie
>>>>
>>>>
>>>> On Fri, Dec 28, 2012 at 3:23 AM, Yannick Legros <
>>>> yannick.legros59@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I have just another question.
>>>>>
>>>>> in this case :
>>>>>
>>>>> publisher post a message M1 on topics T
>>>>> subscriber subscribe to T
>>>>> publisher post a message M2 on topics T
>>>>>
>>>>> when subscriber starts delivery he will get M2 but not M1 because he
>>>>> has subscribed after M1 was publish.
>>>>>
>>>>> does this is the default behaviour of hedwig?
>>>>> can we configure something (in subscription option for exemple) for
>>>>> subscribers be aware of messages publish before they subscribe and not
>>>>> consumed yet.
>>>>>
>>>>> thanks a lot.
>>>>>
>>>>
>>>>
>>>
>>
>
Re: hedwig question
Posted by Yannick Legros <ya...@gmail.com>.
"for currently, we don't support this semantic in Hedwig. I am doubting why
not subscribe first before publish. is it difficult in your case?"
=> subscribers and publishers are waiting for external events to begin
their publish/subscribe. i can't be sure of the order that my "Agents"
detects the external event. So in some cases the publishers starts before
the subscribers and i can loose few message :-(.
"if to support your semantic, all the messages had to be kept until a
subscriber arrived. if there is no subscriber, how to deal with those
published messages? kept them forever?"
=> I know that hedwig is a pub/sub framwork and it is not a treatment
queue, but keep them during a parametrized time would be great
thanks,
Yannick.
2012/12/29 Sijie Guo <gu...@gmail.com>
> thanks, yannick. I got your idea.
>
> for currently, we don't support this semantic in Hedwig. I am doubting why
> not subscribe first before publish. is it difficult in your case?
>
> if to support your semantic, all the messages had to be kept until a
> subscriber arrived. if there is no subscriber, how to deal with those
> published messages? kept them forever?
>
> -Sijie
>
>
> On Sat, Dec 29, 2012 at 3:58 AM, Yannick Legros <
> yannick.legros59@gmail.com> wrote:
>
>> thanks for your answer sijie.
>>
>> what i need is not exactly
>> "subscribers be aware of messages publish before they subscribe"
>> but
>> "be sure that my message be consume by almost one subscriber"
>>
>> so if a publisher publish message on topics with no subscribers the first
>> subscriber who will read the topics will get all published messages before
>> his subscription.
>>
>> hope i was clear.
>>
>> Thanks, yannick.
>>
>> 2012/12/28 Sijie Guo <gu...@gmail.com>
>>
>>> > does this is the default behaviour of hedwig?
>>>
>>> A subscriber would receive all the messages published after its first
>>> time subscribe until it unsubscribe.
>>>
>>> so go back to your example:
>>>
>>> 1) if the subscriber subscribe T first time, it would only receive M2.
>>> 2) if the subscriber already subscribe before M1. so the subscribe
>>> between M1 and M2 is an attach subscribe, so it would receive M1 and M2
>>> eventually (if it doesn't receive M1 at its first time subscribe)
>>>
>>> > can we configure something (in subscription option for exemple) for
>>> subscribers be aware of messages publish before they subscribe and not
>>> consumed yet.
>>>
>>> Currently Hedwig doesn't support it. But I am just curious why a
>>> subscriber need to be aware of messages published before they subscribed?
>>> Is there any special case you are considering?
>>>
>>> -Sijie
>>>
>>>
>>> On Fri, Dec 28, 2012 at 3:23 AM, Yannick Legros <
>>> yannick.legros59@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I have just another question.
>>>>
>>>> in this case :
>>>>
>>>> publisher post a message M1 on topics T
>>>> subscriber subscribe to T
>>>> publisher post a message M2 on topics T
>>>>
>>>> when subscriber starts delivery he will get M2 but not M1 because he
>>>> has subscribed after M1 was publish.
>>>>
>>>> does this is the default behaviour of hedwig?
>>>> can we configure something (in subscription option for exemple) for
>>>> subscribers be aware of messages publish before they subscribe and not
>>>> consumed yet.
>>>>
>>>> thanks a lot.
>>>>
>>>
>>>
>>
>
Re: hedwig question
Posted by Sijie Guo <gu...@gmail.com>.
thanks, yannick. I got your idea.
for currently, we don't support this semantic in Hedwig. I am doubting why
not subscribe first before publish. is it difficult in your case?
if to support your semantic, all the messages had to be kept until a
subscriber arrived. if there is no subscriber, how to deal with those
published messages? kept them forever?
-Sijie
On Sat, Dec 29, 2012 at 3:58 AM, Yannick Legros
<ya...@gmail.com>wrote:
> thanks for your answer sijie.
>
> what i need is not exactly
> "subscribers be aware of messages publish before they subscribe"
> but
> "be sure that my message be consume by almost one subscriber"
>
> so if a publisher publish message on topics with no subscribers the first
> subscriber who will read the topics will get all published messages before
> his subscription.
>
> hope i was clear.
>
> Thanks, yannick.
>
> 2012/12/28 Sijie Guo <gu...@gmail.com>
>
>> > does this is the default behaviour of hedwig?
>>
>> A subscriber would receive all the messages published after its first
>> time subscribe until it unsubscribe.
>>
>> so go back to your example:
>>
>> 1) if the subscriber subscribe T first time, it would only receive M2.
>> 2) if the subscriber already subscribe before M1. so the subscribe
>> between M1 and M2 is an attach subscribe, so it would receive M1 and M2
>> eventually (if it doesn't receive M1 at its first time subscribe)
>>
>> > can we configure something (in subscription option for exemple) for
>> subscribers be aware of messages publish before they subscribe and not
>> consumed yet.
>>
>> Currently Hedwig doesn't support it. But I am just curious why a
>> subscriber need to be aware of messages published before they subscribed?
>> Is there any special case you are considering?
>>
>> -Sijie
>>
>>
>> On Fri, Dec 28, 2012 at 3:23 AM, Yannick Legros <
>> yannick.legros59@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I have just another question.
>>>
>>> in this case :
>>>
>>> publisher post a message M1 on topics T
>>> subscriber subscribe to T
>>> publisher post a message M2 on topics T
>>>
>>> when subscriber starts delivery he will get M2 but not M1 because he has
>>> subscribed after M1 was publish.
>>>
>>> does this is the default behaviour of hedwig?
>>> can we configure something (in subscription option for exemple) for
>>> subscribers be aware of messages publish before they subscribe and not
>>> consumed yet.
>>>
>>> thanks a lot.
>>>
>>
>>
>
Re: hedwig question
Posted by Yannick Legros <ya...@gmail.com>.
thanks for your answer sijie.
what i need is not exactly
"subscribers be aware of messages publish before they subscribe"
but
"be sure that my message be consume by almost one subscriber"
so if a publisher publish message on topics with no subscribers the first
subscriber who will read the topics will get all published messages before
his subscription.
hope i was clear.
Thanks, yannick.
2012/12/28 Sijie Guo <gu...@gmail.com>
> > does this is the default behaviour of hedwig?
>
> A subscriber would receive all the messages published after its first time
> subscribe until it unsubscribe.
>
> so go back to your example:
>
> 1) if the subscriber subscribe T first time, it would only receive M2.
> 2) if the subscriber already subscribe before M1. so the subscribe between
> M1 and M2 is an attach subscribe, so it would receive M1 and M2 eventually
> (if it doesn't receive M1 at its first time subscribe)
>
> > can we configure something (in subscription option for exemple) for
> subscribers be aware of messages publish before they subscribe and not
> consumed yet.
>
> Currently Hedwig doesn't support it. But I am just curious why a
> subscriber need to be aware of messages published before they subscribed?
> Is there any special case you are considering?
>
> -Sijie
>
>
> On Fri, Dec 28, 2012 at 3:23 AM, Yannick Legros <
> yannick.legros59@gmail.com> wrote:
>
>> Hello,
>>
>> I have just another question.
>>
>> in this case :
>>
>> publisher post a message M1 on topics T
>> subscriber subscribe to T
>> publisher post a message M2 on topics T
>>
>> when subscriber starts delivery he will get M2 but not M1 because he has
>> subscribed after M1 was publish.
>>
>> does this is the default behaviour of hedwig?
>> can we configure something (in subscription option for exemple) for
>> subscribers be aware of messages publish before they subscribe and not
>> consumed yet.
>>
>> thanks a lot.
>>
>
>
Re: hedwig question
Posted by Sijie Guo <gu...@gmail.com>.
> does this is the default behaviour of hedwig?
A subscriber would receive all the messages published after its first time
subscribe until it unsubscribe.
so go back to your example:
1) if the subscriber subscribe T first time, it would only receive M2.
2) if the subscriber already subscribe before M1. so the subscribe between
M1 and M2 is an attach subscribe, so it would receive M1 and M2 eventually
(if it doesn't receive M1 at its first time subscribe)
> can we configure something (in subscription option for exemple) for
subscribers be aware of messages publish before they subscribe and not
consumed yet.
Currently Hedwig doesn't support it. But I am just curious why a subscriber
need to be aware of messages published before they subscribed? Is there any
special case you are considering?
-Sijie
On Fri, Dec 28, 2012 at 3:23 AM, Yannick Legros
<ya...@gmail.com>wrote:
> Hello,
>
> I have just another question.
>
> in this case :
>
> publisher post a message M1 on topics T
> subscriber subscribe to T
> publisher post a message M2 on topics T
>
> when subscriber starts delivery he will get M2 but not M1 because he has
> subscribed after M1 was publish.
>
> does this is the default behaviour of hedwig?
> can we configure something (in subscription option for exemple) for
> subscribers be aware of messages publish before they subscribe and not
> consumed yet.
>
> thanks a lot.
>
Re: hedwig question
Posted by Yannick Legros <ya...@gmail.com>.
Hello,
I have just another question.
in this case :
publisher post a message M1 on topics T
subscriber subscribe to T
publisher post a message M2 on topics T
when subscriber starts delivery he will get M2 but not M1 because he has
subscribed after M1 was publish.
does this is the default behaviour of hedwig?
can we configure something (in subscription option for exemple) for
subscribers be aware of messages publish before they subscribe and not
consumed yet.
thanks a lot.
Re: hedwig question
Posted by Yannick Legros <ya...@gmail.com>.
First thanks for your answer Zac Tse,
Im ok for get your advice.
I probably try to do that stuff with just bookeeper.
Yannick Legros.
2012/12/20 Zac Tse <de...@gmail.com>
> I don't think what you refer to is possible with current version of
> hedwig, because the messaging model here is message queue, while the model
> behind hedwig is pub/sub. As far as I am concerned, another project kafka
> from LinkedIn also use pub/sub model.
> I and my classmates are working on something with hedwig, trying to offer
> something more like what you're asking for. I am not the leader, but may be
> I can offer you some little advice.
>
>
> On Thu, Dec 20, 2012 at 7:35 PM, Yannick Legros <
> yannick.legros59@gmail.com> wrote:
>
>> Hello,
>>
>> i currently use rabbitMQ for some messaging treatment, i read docs on
>> hedwig and i want try it.
>>
>> But i wonder if hedwig can answer our use case.
>> configuration of these variables (*default_message_window_size*,*
>> consume_interval*,*retention_secs*) should be the answer. But i'm not
>> sure.
>> i want that a message can be treat by only one subscriber.
>>
>> for example
>> Subscriber 1 subscribe to topic A
>> Subscriber 2 subscribe to topic A
>> publisher post a message M1 on topics A.
>> publisher post a message M2 on topics A.
>>
>> THEN
>>
>> Subscriber 1 consume M1
>> Subscriber 2 consume M2
>>
>> OR
>>
>> Subscriber 1 consume M1 AND M2
>> Subscriber 2 consume nothing
>>
>> OR
>>
>> Subscriber 1 consume M2
>> Subscriber 2 consume M1
>>
>> OR
>>
>> Subscriber 1 consume nothing
>> Subscriber 2 consume M1 AND M2
>>
>> BUT NOT
>>
>> Subscriber 1 consume M1 AND M2
>> Subscriber 2 consume M1 AND M2
>>
>> I hope i was clear.
>>
>> Thx by advance for help, Yannick Legros.
>
>
>
>
> --
> 此致
> 祝好！
>
> 谢轶, Zac
> 手机号码：15801531574
> 邮箱：devilkirin@gmail.com
>
Re: hedwig question
Posted by Zac Tse <de...@gmail.com>.
I don't think what you refer to is possible with current version of hedwig,
because the messaging model here is message queue, while the model behind
hedwig is pub/sub. As far as I am concerned, another project kafka from
LinkedIn also use pub/sub model.
I and my classmates are working on something with hedwig, trying to offer
something more like what you're asking for. I am not the leader, but may be
I can offer you some little advice.
On Thu, Dec 20, 2012 at 7:35 PM, Yannick Legros
<ya...@gmail.com>wrote:
> Hello,
>
> i currently use rabbitMQ for some messaging treatment, i read docs on
> hedwig and i want try it.
>
> But i wonder if hedwig can answer our use case.
> configuration of these variables (*default_message_window_size*,*
> consume_interval*,*retention_secs*) should be the answer. But i'm not
> sure.
> i want that a message can be treat by only one subscriber.
>
> for example
> Subscriber 1 subscribe to topic A
> Subscriber 2 subscribe to topic A
> publisher post a message M1 on topics A.
> publisher post a message M2 on topics A.
>
> THEN
>
> Subscriber 1 consume M1
> Subscriber 2 consume M2
>
> OR
>
> Subscriber 1 consume M1 AND M2
> Subscriber 2 consume nothing
>
> OR
>
> Subscriber 1 consume M2
> Subscriber 2 consume M1
>
> OR
>
> Subscriber 1 consume nothing
> Subscriber 2 consume M1 AND M2
>
> BUT NOT
>
> Subscriber 1 consume M1 AND M2
> Subscriber 2 consume M1 AND M2
>
> I hope i was clear.
>
> Thx by advance for help, Yannick Legros.
--
此致
祝好！
谢轶, Zac
手机号码：15801531574
邮箱：devilkirin@gmail.com