You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by sowmya s <so...@gmail.com> on 2018/03/01 18:50:42 UTC
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Hello all,
Adding a few more thoughts on the problem of establishing an order of
messages across producers.
Consider the Scenario
Producer-1 produces messages a1, b1 and c1 into a queue Queue1
Producer-2 produces messages a2, b2 and c2 into queue Queue2.
If we assume that time(a1) < time(b1) < time(c1) and similarly time(a2) <
time(b2) < time(c2)
Are the following orders acceptable to the consumer?
> a1, a2, b1, b2, c1, c2
> a1, b1, c1, a2, b2, c2
> a1 b1, b2, a2, a3, b3
and an order displayed at one consumer is consistent across all consumers.
This can be achieved using Total Causal Ordering at the Producer or Queue
level, using Leslie Lamport's clock and synchronization approach.
For reference is the paper attached,
http://lamport.azurewebsites.net/pubs/time-clocks.pdf
--
Regards,
Sowmya
On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <so...@gmail.com> wrote:
> Hello,
>
> I'm trying to work on the issue ROCKETMQ-122
> <https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065> as a
> part of Google Summer of Code 2018. I've been spending some time to
> understand the system, architecture and the existing Messaging Patterns.
> I still have a few questions and would like to clarify my assumptions.
>
> - Is the current FIFO order example limited to one message queue per
> producer? Can the producer send the same message to multiple queues? Will
> the consumers of the queues be able to read messages in Order?
> - Can I assume that each producer will send messages to one queue?
> - Global Order is to be identified across all GlobalOrderedProducer (a
> new producer that is to be used for global order) instances that are
> running.
> - I think using a global clock can help establish the order between 2
> or more producers, however using some form of vector clock might also
> help identify the global order of messages between the producers.
> - A GlobalOrderedConsumer ( consumer that knows how to read globally
> ordered messages) can then compare messages across all message queues from
> the corresponding producers and extract the messages. [ this is the
> approach recommended by yukon on the issue page ]
> - We can also potentially have another layer in the Message Queue
> which accumulates all messages sent from producers and provides one ordered
> message queue for consumers to read from.
>
> Thank you for your patience and please let me know if my understanding of
> the problem and the assumptions are right.
>
> Best Regards,
> Sowmya
>
>
--
Regards,
Sowmya
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by sowmya s <so...@gmail.com>.
Hey yukon, Von,
I'm checking to see if you have had a chance to look at the proposal. It
would be very nice to get some feedback and direction before the deadline
for the final proposals.
thanks,
Sowmya
On Fri, Mar 16, 2018 at 8:07 PM, yukon <yu...@apache.org> wrote:
> Cool, let's focus on it and see whether is there anything can be polished.
>
> Regards
>
> On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <so...@gmail.com> wrote:
>
> > Hey yukon,
> > I submitted my draft on the summer of code homepage a couple of days ago,
> > also attaching the link here for reference,
> >
> > https://docs.google.com/file/d/1nXktUO_TF9-rSHSnGj5z5QZoHzMhosxm/edit?
> > usp=docslist_api&filetype=msword
> >
> > Thanks,
> > Sowmya
> >
> > On Thu, Mar 15, 2018 at 2:08 AM yukon <yu...@apache.org> wrote:
> >
> > > Hi,
> > >
> > > Of course, we can work together to finetune your design draft.
> > >
> > > Regards,
> > > yukon
> > >
> > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <so...@gmail.com>
> wrote:
> > >
> > > > Hello, yukon and Von,
> > > >
> > > > I've shared my GSOC - 18 draft of the project. I'm looking forward to
> > > > working with all of you to finetune the proposal. I will be
> allocating
> > 20
> > > > hours per week from now to the proposal acceptance phase to address
> > > > questions and dive deep into any suggestions that you provide.
> > > > I am really looking forward to work on this project.
> > > >
> > > > thanks,
> > > > Sowmya
> > > >
> > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <so...@gmail.com>
> > wrote:
> > > >
> > > > > Hello yukon,
> > > > >
> > > > > Thank you for the inputs. I was able to look at the ~/store and
> kind
> > of
> > > > > understand the storage structure. I also looked at the
> > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the
> > > > examples.
> > > > > Now I understand why you proposed a merge sort like approach for
> > > > > performing global ordering. Since the proposals are open, I am
> > > finalizing
> > > > > my draft and will have it up for review very soon.
> > > > >
> > > > > thanks,
> > > > > Sowmya
> > > > >
> > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org> wrote:
> > > > >
> > > > >> Hi Sowmya,
> > > > >>
> > > > >> ```
> > > > >> Also, it would be great if you can at a high level, help me
> > understand
> > > > >> how the messages in the message queues are stored before the
> > consumer
> > > > reads
> > > > >> them.
> > > > >> ```
> > > > >>
> > > > >>
> > > > >>
> > > > >> As shown in this figure, messages are sent to brokers by producer
> > and
> > > > >> stored in commit log[1], then messages are dispatched to
> > ConsumeQueue
> > > by
> > > > >> topic, the consumer pulls messages from the queue.
> > > > >>
> > > > >> I recommend you run a broker and send/consume some messages, then
> > > check
> > > > >> out the ~/store directory for details.
> > > > >>
> > > > >> Regards,
> > > > >> yukon
> > > > >>
> > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <so...@gmail.com>
> > > wrote:
> > > > >>
> > > > >>> Hello yukon,
> > > > >>>
> > > > >>> Currently FIFO can be achieved with a producer sending to one
> > message
> > > > >>> queue, and when global ordering is required, multiple producers
> > have
> > > to
> > > > >>> send to a single topic queue.
> > > > >>>
> > > > >>> We want to allow multiple producers to send messages on a topic
> to
> > > > >>> multiple message queues and still provide ordering guarantees to
> > the
> > > > >>> consumer, so that all consumers see the same order of data and
> also
> > > the
> > > > >>> data is delivered in an ordered fashion.
> > > > >>>
> > > > >>> 1) Your idea of using a merge sort with the assumption that the
> > first
> > > > >>> arriving message is treated as the first message to be delivered,
> > > > however,
> > > > >>> I want to propose an approach where when the producer sends a
> > message
> > > > to a
> > > > >>> message queue, it must be done in a synchronous fashion and the
> > > > response
> > > > >>> will be that the message is accepted, which means that the
> message
> > > > follows
> > > > >>> the convention that all messages delivered prior by that producer
> > > have
> > > > been
> > > > >>> stored across groups and if not the producer will need to resend
> > the
> > > > >>> message.
> > > > >>>
> > > > >>> We can use a variant of total causal ordering in the layer
> between
> > > the
> > > > >>> message queue and store.
> > > > >>>
> > > > >>> I have been busy with my class project so I couldn't make a lot
> of
> > > > >>> progress in detailing my approach. Also, it would be great if you
> > can
> > > > at a
> > > > >>> high level, help me understand how the messages in the message
> > queues
> > > > are
> > > > >>> stored before the consumer reads them.
> > > > >>>
> > > > >>> Does the consumer read directly from the message queue that the
> > > > producer
> > > > >>> sends data to? does the broker receive the queued producer
> > messages,
> > > > store
> > > > >>> them and then pushes them to another queue for the consumer to
> read
> > > > from?
> > > > >>>
> > > > >>> For reference on total causal ordering:
> > > https://www.cl.cam.ac.uk/teach
> > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
> > > > >>>
> > > > >>> thanks,
> > > > >>> Sowmya
> > > > >>>
> > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org> wrote:
> > > > >>>
> > > > >>>> Hi Sowmya,
> > > > >>>>
> > > > >>>> Sorry for the late reply, do you have any update on this
> project?
> > > > >>>>
> > > > >>>> In RocketMQ, one message queue is a FIFO queue natively, so I
> > > proposed
> > > > >>>> a simple solution that performs merge sort on multiple queues to
> > > > improve
> > > > >>>> performance and scalability.
> > > > >>>>
> > > > >>>> While the order issue across producers is difficult, we could
> > assume
> > > > >>>> that the message first arrives the broker should be consumed
> > > firstly.
> > > > >>>>
> > > > >>>> But it would be wonderful if you have a real design about the
> > order
> > > > >>>> issue across producers based on the RocketMQ design and the
> > storage
> > > > >>>> structure.
> > > > >>>>
> > > > >>>> Looking forward your design ~
> > > > >>>>
> > > > >>>> Regards,
> > > > >>>> yukon
> > > > >>>>
> > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <so...@gmail.com>
> > > > wrote:
> > > > >>>>
> > > > >>>>> Hello all,
> > > > >>>>>
> > > > >>>>> Adding a few more thoughts on the problem of establishing an
> > order
> > > of
> > > > >>>>> messages across producers.
> > > > >>>>>
> > > > >>>>> Consider the Scenario
> > > > >>>>>
> > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1
> > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2.
> > > > >>>>>
> > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and similarly
> > > > >>>>> time(a2) < time(b2) < time(c2)
> > > > >>>>>
> > > > >>>>> Are the following orders acceptable to the consumer?
> > > > >>>>>
> > > > >>>>> > a1, a2, b1, b2, c1, c2
> > > > >>>>> > a1, b1, c1, a2, b2, c2
> > > > >>>>> > a1 b1, b2, a2, a3, b3
> > > > >>>>>
> > > > >>>>> and an order displayed at one consumer is consistent across all
> > > > >>>>> consumers.
> > > > >>>>>
> > > > >>>>> This can be achieved using Total Causal Ordering at the
> Producer
> > or
> > > > >>>>> Queue level, using Leslie Lamport's clock and synchronization
> > > > approach.
> > > > >>>>>
> > > > >>>>> For reference is the paper attached,
> > > http://lamport.azurewebsites.n
> > > > >>>>> et/pubs/time-clocks.pdf
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> --
> > > > >>>>> Regards,
> > > > >>>>>
> > > > >>>>> Sowmya
> > > > >>>>>
> > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <
> sowmya9029@gmail.com>
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> Hello,
> > > > >>>>>>
> > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122
> > > > >>>>>> <
> > > https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065>
> > > > as
> > > > >>>>>> a part of Google Summer of Code 2018. I've been spending some
> > time
> > > > to
> > > > >>>>>> understand the system, architecture and the existing Messaging
> > > > Patterns.
> > > > >>>>>> I still have a few questions and would like to clarify my
> > > > assumptions.
> > > > >>>>>>
> > > > >>>>>> - Is the current FIFO order example limited to one message
> > > queue
> > > > >>>>>> per producer? Can the producer send the same message to
> > > multiple
> > > > queues?
> > > > >>>>>> Will the consumers of the queues be able to read messages
> in
> > > > Order?
> > > > >>>>>> - Can I assume that each producer will send messages to one
> > > > queue?
> > > > >>>>>> - Global Order is to be identified across all
> > > > >>>>>> GlobalOrderedProducer (a new producer that is to be used
> for
> > > > global order)
> > > > >>>>>> instances that are running.
> > > > >>>>>> - I think using a global clock can help establish the order
> > > > >>>>>> between 2 or more producers, however using some form of
> > vector
> > > > >>>>>> clock might also help identify the global order of messages
> > > > between the
> > > > >>>>>> producers.
> > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how to read
> > > > >>>>>> globally ordered messages) can then compare messages across
> > all
> > > > message
> > > > >>>>>> queues from the corresponding producers and extract the
> > > > messages. [ this is
> > > > >>>>>> the approach recommended by yukon on the issue page ]
> > > > >>>>>> - We can also potentially have another layer in the Message
> > > Queue
> > > > >>>>>> which accumulates all messages sent from producers and
> > provides
> > > > one ordered
> > > > >>>>>> message queue for consumers to read from.
> > > > >>>>>>
> > > > >>>>>> Thank you for your patience and please let me know if my
> > > > >>>>>> understanding of the problem and the assumptions are right.
> > > > >>>>>>
> > > > >>>>>> Best Regards,
> > > > >>>>>> Sowmya
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> --
> > > > >>>>> Regards,
> > > > >>>>>
> > > > >>>>> Sowmya
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Regards,
> > > > >>>
> > > > >>> Sowmya
> > > > >>>
> > > > >>>
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Sowmya
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Sowmya
> > > >
> > >
> > --
> > Regards,
> >
> > Sowmya
> >
>
--
Regards,
Sowmya
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by sowmya s <so...@gmail.com>.
Hi yukon,
I'm working on updating the design for the hot-pot issue.
Thanks for the direction, from a consumer perspective, when the context is
chosen as orderly, there is a specific Rebalance interface that orders the
messages by Topic. this is used by the DefaultMQPushConsumerImpl to send
messages to the consumer. The RebalanceImpl is the one that actually
fetches the messages by topic from brokers and organizes by topic.
Along with a How about having a custom RebalanceImpl that can be injected
into the Consumer, so that the pull/push can be created based on the global
order rebalance strategy.
In the rebalanceImpl class, there is an implementation to do rebalance
where messages from all brokers are fetched and ordered.
thanks,
Sowmya
On Fri, Mar 23, 2018 at 5:00 AM, yukon <yu...@apache.org> wrote:
> Hi,
>
> ```
> Does the fleet of broker slaves all store the various commit logs that are
> received by them or is the commit log store, replication managed by the
> broker controller?
> ```
> Yes, the slave broker will all the commit logs synced from the master
> broker.
>
> ```
> The other direction I was thinking about is adding/modifying the
> PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
> DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding
> this change there will solve the merge issue for the consumers.
> ```
>
> How about adding another consumer type?
>
> On Fri, Mar 23, 2018 at 7:48 PM, yukon <yu...@apache.org> wrote:
>
> > Hi,
> >
> > I added some comments, but it seems your google doc link isn't shared:
> >
> > ```
> > To resolve this issue, we need a global ordered messaging mechanism to
> > have a good scalability without hot-pot issue.
> >
> > A practical proposal is that sending messages to distributed
> > queue/partition with an ordered key. Messages are ordered in the same
> > queue, then implement an OrderConsumer to pull messages from these queues
> > then perform merge sort, finally deliver these ordered messages to users.
> > ```
> > As mentioned in ROCKETMQ-122, we should address the hot-pot issue.
> >
> > That means the messages can be sent to multiple distributed
> > queues/partitions and still can be consumed in order manner.
> >
> > A simple explanation of hot-pot issue:
> >
> > Assume that all the generated messages in a hot shop of taobao/amazon
> > should be consumed in order, in the current state, these massive messages
> > should be sent to a fixed queue/partition, surely to a fixed server which
> > may cause high pressure in this server. You can easily infer that we
> can't
> > resolve this issue through extending the scale of broker cluster.
> >
> > Could your arch resolve this hot-pot issue?
> >
> > Regards,
> > yukon
> >
> >
> > On Thu, Mar 22, 2018 at 2:15 PM, sowmya s <so...@gmail.com> wrote:
> >
> >> It's going great.
> >> In the DefaultMessageStore, there is a class
> >> CommitLogDispatcherBuildConsumeQueue, I think that a variant of that
> >> class
> >> or another implementation which are merged from the commitlog,
> >> Since the DispachRequest has a store timestamp, the queue can be
> populated
> >> based on that time stamp.
> >>
> >> Also, I have some more questions,
> >> Does the fleet of broker slaves all store the various commit logs that
> are
> >> received by them or is the commit log store, replication managed by the
> >> broker controller?
> >>
> >> The other direction I was thinking about is adding/modifying the
> >> PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
> >> DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding
> >> this change there will solve the merge issue for the consumers.
> >>
> >> Do you think I am on the right track?
> >>
> >> thanks,
> >> Sowmya
> >>
> >>
> >>
> >> On Mon, Mar 19, 2018 at 8:15 PM, yukon <yu...@apache.org> wrote:
> >>
> >> > Hi,
> >> >
> >> > Sorry for the late reply.
> >> >
> >> > As for:
> >> >
> >> > ```
> >> > I'm looking at the BrokerController and MessageStore implementation
> and
> >> > hooks to understand where the merge logic will best fit.
> >> > ```
> >> >
> >> > how is it going?
> >> >
> >> > Regards
> >> >
> >> > On Sat, Mar 17, 2018 at 11:30 AM, sowmya s <so...@gmail.com>
> >> wrote:
> >> >
> >> > > Thank you yukon,
> >> > >
> >> > > I'm done with my coursework for this semester and have more time now
> >> to
> >> > > improve my proposal.
> >> > > I'm looking at the BrokerController and MessageStore implementation
> >> and
> >> > > hooks to understand where the merge logic will best fit. So far I've
> >> > looked
> >> > > at the codebase from a Producer and Consumer perspective and looked
> at
> >> > the
> >> > > DefaultMQProducerImpl and DefaultMQPushConsumerImpl for
> understanding
> >> > the
> >> > > link between how Producers send and Consumers receive messages.
> >> > >
> >> > > thanks,
> >> > > Sowmya
> >> > >
> >> > > On Fri, Mar 16, 2018 at 8:07 PM, yukon <yu...@apache.org> wrote:
> >> > >
> >> > > > Cool, let's focus on it and see whether is there anything can be
> >> > > polished.
> >> > > >
> >> > > > Regards
> >> > > >
> >> > > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <so...@gmail.com>
> >> > wrote:
> >> > > >
> >> > > > > Hey yukon,
> >> > > > > I submitted my draft on the summer of code homepage a couple of
> >> days
> >> > > ago,
> >> > > > > also attaching the link here for reference,
> >> > > > >
> >> > > > > https://docs.google.com/file/d/1nXktUO_TF9-
> >> > rSHSnGj5z5QZoHzMhosxm/edit?
> >> > > > > usp=docslist_api&filetype=msword
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Sowmya
> >> > > > >
> >> > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon <yu...@apache.org> wrote:
> >> > > > >
> >> > > > > > Hi,
> >> > > > > >
> >> > > > > > Of course, we can work together to finetune your design draft.
> >> > > > > >
> >> > > > > > Regards,
> >> > > > > > yukon
> >> > > > > >
> >> > > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <
> sowmya9029@gmail.com
> >> >
> >> > > > wrote:
> >> > > > > >
> >> > > > > > > Hello, yukon and Von,
> >> > > > > > >
> >> > > > > > > I've shared my GSOC - 18 draft of the project. I'm looking
> >> > forward
> >> > > to
> >> > > > > > > working with all of you to finetune the proposal. I will be
> >> > > > allocating
> >> > > > > 20
> >> > > > > > > hours per week from now to the proposal acceptance phase to
> >> > address
> >> > > > > > > questions and dive deep into any suggestions that you
> provide.
> >> > > > > > > I am really looking forward to work on this project.
> >> > > > > > >
> >> > > > > > > thanks,
> >> > > > > > > Sowmya
> >> > > > > > >
> >> > > > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <
> >> sowmya9029@gmail.com
> >> > >
> >> > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Hello yukon,
> >> > > > > > > >
> >> > > > > > > > Thank you for the inputs. I was able to look at the
> ~/store
> >> and
> >> > > > kind
> >> > > > > of
> >> > > > > > > > understand the storage structure. I also looked at the
> >> > > > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used
> in
> >> > the
> >> > > > > > > examples.
> >> > > > > > > > Now I understand why you proposed a merge sort like
> approach
> >> > for
> >> > > > > > > > performing global ordering. Since the proposals are open,
> I
> >> am
> >> > > > > > finalizing
> >> > > > > > > > my draft and will have it up for review very soon.
> >> > > > > > > >
> >> > > > > > > > thanks,
> >> > > > > > > > Sowmya
> >> > > > > > > >
> >> > > > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org>
> >> > wrote:
> >> > > > > > > >
> >> > > > > > > >> Hi Sowmya,
> >> > > > > > > >>
> >> > > > > > > >> ```
> >> > > > > > > >> Also, it would be great if you can at a high level, help
> me
> >> > > > > understand
> >> > > > > > > >> how the messages in the message queues are stored before
> >> the
> >> > > > > consumer
> >> > > > > > > reads
> >> > > > > > > >> them.
> >> > > > > > > >> ```
> >> > > > > > > >>
> >> > > > > > > >>
> >> > > > > > > >>
> >> > > > > > > >> As shown in this figure, messages are sent to brokers by
> >> > > producer
> >> > > > > and
> >> > > > > > > >> stored in commit log[1], then messages are dispatched to
> >> > > > > ConsumeQueue
> >> > > > > > by
> >> > > > > > > >> topic, the consumer pulls messages from the queue.
> >> > > > > > > >>
> >> > > > > > > >> I recommend you run a broker and send/consume some
> >> messages,
> >> > > then
> >> > > > > > check
> >> > > > > > > >> out the ~/store directory for details.
> >> > > > > > > >>
> >> > > > > > > >> Regards,
> >> > > > > > > >> yukon
> >> > > > > > > >>
> >> > > > > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <
> >> > sowmya9029@gmail.com
> >> > > >
> >> > > > > > wrote:
> >> > > > > > > >>
> >> > > > > > > >>> Hello yukon,
> >> > > > > > > >>>
> >> > > > > > > >>> Currently FIFO can be achieved with a producer sending
> to
> >> one
> >> > > > > message
> >> > > > > > > >>> queue, and when global ordering is required, multiple
> >> > producers
> >> > > > > have
> >> > > > > > to
> >> > > > > > > >>> send to a single topic queue.
> >> > > > > > > >>>
> >> > > > > > > >>> We want to allow multiple producers to send messages on
> a
> >> > topic
> >> > > > to
> >> > > > > > > >>> multiple message queues and still provide ordering
> >> guarantees
> >> > > to
> >> > > > > the
> >> > > > > > > >>> consumer, so that all consumers see the same order of
> data
> >> > and
> >> > > > also
> >> > > > > > the
> >> > > > > > > >>> data is delivered in an ordered fashion.
> >> > > > > > > >>>
> >> > > > > > > >>> 1) Your idea of using a merge sort with the assumption
> >> that
> >> > the
> >> > > > > first
> >> > > > > > > >>> arriving message is treated as the first message to be
> >> > > delivered,
> >> > > > > > > however,
> >> > > > > > > >>> I want to propose an approach where when the producer
> >> sends a
> >> > > > > message
> >> > > > > > > to a
> >> > > > > > > >>> message queue, it must be done in a synchronous fashion
> >> and
> >> > the
> >> > > > > > > response
> >> > > > > > > >>> will be that the message is accepted, which means that
> the
> >> > > > message
> >> > > > > > > follows
> >> > > > > > > >>> the convention that all messages delivered prior by that
> >> > > producer
> >> > > > > > have
> >> > > > > > > been
> >> > > > > > > >>> stored across groups and if not the producer will need
> to
> >> > > resend
> >> > > > > the
> >> > > > > > > >>> message.
> >> > > > > > > >>>
> >> > > > > > > >>> We can use a variant of total causal ordering in the
> layer
> >> > > > between
> >> > > > > > the
> >> > > > > > > >>> message queue and store.
> >> > > > > > > >>>
> >> > > > > > > >>> I have been busy with my class project so I couldn't
> make
> >> a
> >> > lot
> >> > > > of
> >> > > > > > > >>> progress in detailing my approach. Also, it would be
> >> great if
> >> > > you
> >> > > > > can
> >> > > > > > > at a
> >> > > > > > > >>> high level, help me understand how the messages in the
> >> > message
> >> > > > > queues
> >> > > > > > > are
> >> > > > > > > >>> stored before the consumer reads them.
> >> > > > > > > >>>
> >> > > > > > > >>> Does the consumer read directly from the message queue
> >> that
> >> > the
> >> > > > > > > producer
> >> > > > > > > >>> sends data to? does the broker receive the queued
> producer
> >> > > > > messages,
> >> > > > > > > store
> >> > > > > > > >>> them and then pushes them to another queue for the
> >> consumer
> >> > to
> >> > > > read
> >> > > > > > > from?
> >> > > > > > > >>>
> >> > > > > > > >>> For reference on total causal ordering:
> >> > > > > > https://www.cl.cam.ac.uk/teach
> >> > > > > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
> >> > > > > > > >>>
> >> > > > > > > >>> thanks,
> >> > > > > > > >>> Sowmya
> >> > > > > > > >>>
> >> > > > > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yukon@apache.org
> >
> >> > > wrote:
> >> > > > > > > >>>
> >> > > > > > > >>>> Hi Sowmya,
> >> > > > > > > >>>>
> >> > > > > > > >>>> Sorry for the late reply, do you have any update on
> this
> >> > > > project?
> >> > > > > > > >>>>
> >> > > > > > > >>>> In RocketMQ, one message queue is a FIFO queue
> natively,
> >> so
> >> > I
> >> > > > > > proposed
> >> > > > > > > >>>> a simple solution that performs merge sort on multiple
> >> > queues
> >> > > to
> >> > > > > > > improve
> >> > > > > > > >>>> performance and scalability.
> >> > > > > > > >>>>
> >> > > > > > > >>>> While the order issue across producers is difficult, we
> >> > could
> >> > > > > assume
> >> > > > > > > >>>> that the message first arrives the broker should be
> >> consumed
> >> > > > > > firstly.
> >> > > > > > > >>>>
> >> > > > > > > >>>> But it would be wonderful if you have a real design
> about
> >> > the
> >> > > > > order
> >> > > > > > > >>>> issue across producers based on the RocketMQ design and
> >> the
> >> > > > > storage
> >> > > > > > > >>>> structure.
> >> > > > > > > >>>>
> >> > > > > > > >>>> Looking forward your design ~
> >> > > > > > > >>>>
> >> > > > > > > >>>> Regards,
> >> > > > > > > >>>> yukon
> >> > > > > > > >>>>
> >> > > > > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <
> >> > > sowmya9029@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > > >>>>
> >> > > > > > > >>>>> Hello all,
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> Adding a few more thoughts on the problem of
> >> establishing
> >> > an
> >> > > > > order
> >> > > > > > of
> >> > > > > > > >>>>> messages across producers.
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> Consider the Scenario
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a
> queue
> >> > > Queue1
> >> > > > > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue
> >> > Queue2.
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and
> >> > > similarly
> >> > > > > > > >>>>> time(a2) < time(b2) < time(c2)
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> Are the following orders acceptable to the consumer?
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> > a1, a2, b1, b2, c1, c2
> >> > > > > > > >>>>> > a1, b1, c1, a2, b2, c2
> >> > > > > > > >>>>> > a1 b1, b2, a2, a3, b3
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> and an order displayed at one consumer is consistent
> >> across
> >> > > all
> >> > > > > > > >>>>> consumers.
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> This can be achieved using Total Causal Ordering at
> the
> >> > > > Producer
> >> > > > > or
> >> > > > > > > >>>>> Queue level, using Leslie Lamport's clock and
> >> > synchronization
> >> > > > > > > approach.
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> For reference is the paper attached,
> >> > > > > > http://lamport.azurewebsites.n
> >> > > > > > > >>>>> et/pubs/time-clocks.pdf
> >> > > > > > > >>>>>
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> --
> >> > > > > > > >>>>> Regards,
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> Sowmya
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <
> >> > > > sowmya9029@gmail.com>
> >> > > > > > > >>>>> wrote:
> >> > > > > > > >>>>>
> >> > > > > > > >>>>>> Hello,
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122
> >> > > > > > > >>>>>> <
> >> > > > > > https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=
> >> 12343065
> >> > >
> >> > > > > > > as
> >> > > > > > > >>>>>> a part of Google Summer of Code 2018. I've been
> >> spending
> >> > > some
> >> > > > > time
> >> > > > > > > to
> >> > > > > > > >>>>>> understand the system, architecture and the existing
> >> > > Messaging
> >> > > > > > > Patterns.
> >> > > > > > > >>>>>> I still have a few questions and would like to
> clarify
> >> my
> >> > > > > > > assumptions.
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>> - Is the current FIFO order example limited to one
> >> > > message
> >> > > > > > queue
> >> > > > > > > >>>>>> per producer? Can the producer send the same
> >> message to
> >> > > > > > multiple
> >> > > > > > > queues?
> >> > > > > > > >>>>>> Will the consumers of the queues be able to read
> >> > messages
> >> > > > in
> >> > > > > > > Order?
> >> > > > > > > >>>>>> - Can I assume that each producer will send
> >> messages to
> >> > > one
> >> > > > > > > queue?
> >> > > > > > > >>>>>> - Global Order is to be identified across all
> >> > > > > > > >>>>>> GlobalOrderedProducer (a new producer that is to
> be
> >> > used
> >> > > > for
> >> > > > > > > global order)
> >> > > > > > > >>>>>> instances that are running.
> >> > > > > > > >>>>>> - I think using a global clock can help establish
> >> the
> >> > > order
> >> > > > > > > >>>>>> between 2 or more producers, however using some
> >> form of
> >> > > > > vector
> >> > > > > > > >>>>>> clock might also help identify the global order of
> >> > > messages
> >> > > > > > > between the
> >> > > > > > > >>>>>> producers.
> >> > > > > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows
> how
> >> to
> >> > > read
> >> > > > > > > >>>>>> globally ordered messages) can then compare
> messages
> >> > > across
> >> > > > > all
> >> > > > > > > message
> >> > > > > > > >>>>>> queues from the corresponding producers and
> extract
> >> the
> >> > > > > > > messages. [ this is
> >> > > > > > > >>>>>> the approach recommended by yukon on the issue
> page
> >> ]
> >> > > > > > > >>>>>> - We can also potentially have another layer in
> the
> >> > > Message
> >> > > > > > Queue
> >> > > > > > > >>>>>> which accumulates all messages sent from producers
> >> and
> >> > > > > provides
> >> > > > > > > one ordered
> >> > > > > > > >>>>>> message queue for consumers to read from.
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>> Thank you for your patience and please let me know if
> >> my
> >> > > > > > > >>>>>> understanding of the problem and the assumptions are
> >> > right.
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>> Best Regards,
> >> > > > > > > >>>>>> Sowmya
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> --
> >> > > > > > > >>>>> Regards,
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> Sowmya
> >> > > > > > > >>>>>
> >> > > > > > > >>>>>
> >> > > > > > > >>>>
> >> > > > > > > >>>
> >> > > > > > > >>>
> >> > > > > > > >>> --
> >> > > > > > > >>> Regards,
> >> > > > > > > >>>
> >> > > > > > > >>> Sowmya
> >> > > > > > > >>>
> >> > > > > > > >>>
> >> > > > > > > >>
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > --
> >> > > > > > > > Regards,
> >> > > > > > > >
> >> > > > > > > > Sowmya
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > Regards,
> >> > > > > > >
> >> > > > > > > Sowmya
> >> > > > > > >
> >> > > > > >
> >> > > > > --
> >> > > > > Regards,
> >> > > > >
> >> > > > > Sowmya
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Regards,
> >> > >
> >> > > Sowmya
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Regards,
> >>
> >> Sowmya
> >>
> >
> >
>
--
Regards,
Sowmya
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by yukon <yu...@apache.org>.
Hi,
```
Does the fleet of broker slaves all store the various commit logs that are
received by them or is the commit log store, replication managed by the
broker controller?
```
Yes, the slave broker will all the commit logs synced from the master
broker.
```
The other direction I was thinking about is adding/modifying the
PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding
this change there will solve the merge issue for the consumers.
```
How about adding another consumer type?
On Fri, Mar 23, 2018 at 7:48 PM, yukon <yu...@apache.org> wrote:
> Hi,
>
> I added some comments, but it seems your google doc link isn't shared:
>
> ```
> To resolve this issue, we need a global ordered messaging mechanism to
> have a good scalability without hot-pot issue.
>
> A practical proposal is that sending messages to distributed
> queue/partition with an ordered key. Messages are ordered in the same
> queue, then implement an OrderConsumer to pull messages from these queues
> then perform merge sort, finally deliver these ordered messages to users.
> ```
> As mentioned in ROCKETMQ-122, we should address the hot-pot issue.
>
> That means the messages can be sent to multiple distributed
> queues/partitions and still can be consumed in order manner.
>
> A simple explanation of hot-pot issue:
>
> Assume that all the generated messages in a hot shop of taobao/amazon
> should be consumed in order, in the current state, these massive messages
> should be sent to a fixed queue/partition, surely to a fixed server which
> may cause high pressure in this server. You can easily infer that we can't
> resolve this issue through extending the scale of broker cluster.
>
> Could your arch resolve this hot-pot issue?
>
> Regards,
> yukon
>
>
> On Thu, Mar 22, 2018 at 2:15 PM, sowmya s <so...@gmail.com> wrote:
>
>> It's going great.
>> In the DefaultMessageStore, there is a class
>> CommitLogDispatcherBuildConsumeQueue, I think that a variant of that
>> class
>> or another implementation which are merged from the commitlog,
>> Since the DispachRequest has a store timestamp, the queue can be populated
>> based on that time stamp.
>>
>> Also, I have some more questions,
>> Does the fleet of broker slaves all store the various commit logs that are
>> received by them or is the commit log store, replication managed by the
>> broker controller?
>>
>> The other direction I was thinking about is adding/modifying the
>> PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
>> DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding
>> this change there will solve the merge issue for the consumers.
>>
>> Do you think I am on the right track?
>>
>> thanks,
>> Sowmya
>>
>>
>>
>> On Mon, Mar 19, 2018 at 8:15 PM, yukon <yu...@apache.org> wrote:
>>
>> > Hi,
>> >
>> > Sorry for the late reply.
>> >
>> > As for:
>> >
>> > ```
>> > I'm looking at the BrokerController and MessageStore implementation and
>> > hooks to understand where the merge logic will best fit.
>> > ```
>> >
>> > how is it going?
>> >
>> > Regards
>> >
>> > On Sat, Mar 17, 2018 at 11:30 AM, sowmya s <so...@gmail.com>
>> wrote:
>> >
>> > > Thank you yukon,
>> > >
>> > > I'm done with my coursework for this semester and have more time now
>> to
>> > > improve my proposal.
>> > > I'm looking at the BrokerController and MessageStore implementation
>> and
>> > > hooks to understand where the merge logic will best fit. So far I've
>> > looked
>> > > at the codebase from a Producer and Consumer perspective and looked at
>> > the
>> > > DefaultMQProducerImpl and DefaultMQPushConsumerImpl for understanding
>> > the
>> > > link between how Producers send and Consumers receive messages.
>> > >
>> > > thanks,
>> > > Sowmya
>> > >
>> > > On Fri, Mar 16, 2018 at 8:07 PM, yukon <yu...@apache.org> wrote:
>> > >
>> > > > Cool, let's focus on it and see whether is there anything can be
>> > > polished.
>> > > >
>> > > > Regards
>> > > >
>> > > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <so...@gmail.com>
>> > wrote:
>> > > >
>> > > > > Hey yukon,
>> > > > > I submitted my draft on the summer of code homepage a couple of
>> days
>> > > ago,
>> > > > > also attaching the link here for reference,
>> > > > >
>> > > > > https://docs.google.com/file/d/1nXktUO_TF9-
>> > rSHSnGj5z5QZoHzMhosxm/edit?
>> > > > > usp=docslist_api&filetype=msword
>> > > > >
>> > > > > Thanks,
>> > > > > Sowmya
>> > > > >
>> > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon <yu...@apache.org> wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > Of course, we can work together to finetune your design draft.
>> > > > > >
>> > > > > > Regards,
>> > > > > > yukon
>> > > > > >
>> > > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <sowmya9029@gmail.com
>> >
>> > > > wrote:
>> > > > > >
>> > > > > > > Hello, yukon and Von,
>> > > > > > >
>> > > > > > > I've shared my GSOC - 18 draft of the project. I'm looking
>> > forward
>> > > to
>> > > > > > > working with all of you to finetune the proposal. I will be
>> > > > allocating
>> > > > > 20
>> > > > > > > hours per week from now to the proposal acceptance phase to
>> > address
>> > > > > > > questions and dive deep into any suggestions that you provide.
>> > > > > > > I am really looking forward to work on this project.
>> > > > > > >
>> > > > > > > thanks,
>> > > > > > > Sowmya
>> > > > > > >
>> > > > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <
>> sowmya9029@gmail.com
>> > >
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > Hello yukon,
>> > > > > > > >
>> > > > > > > > Thank you for the inputs. I was able to look at the ~/store
>> and
>> > > > kind
>> > > > > of
>> > > > > > > > understand the storage structure. I also looked at the
>> > > > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in
>> > the
>> > > > > > > examples.
>> > > > > > > > Now I understand why you proposed a merge sort like approach
>> > for
>> > > > > > > > performing global ordering. Since the proposals are open, I
>> am
>> > > > > > finalizing
>> > > > > > > > my draft and will have it up for review very soon.
>> > > > > > > >
>> > > > > > > > thanks,
>> > > > > > > > Sowmya
>> > > > > > > >
>> > > > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org>
>> > wrote:
>> > > > > > > >
>> > > > > > > >> Hi Sowmya,
>> > > > > > > >>
>> > > > > > > >> ```
>> > > > > > > >> Also, it would be great if you can at a high level, help me
>> > > > > understand
>> > > > > > > >> how the messages in the message queues are stored before
>> the
>> > > > > consumer
>> > > > > > > reads
>> > > > > > > >> them.
>> > > > > > > >> ```
>> > > > > > > >>
>> > > > > > > >>
>> > > > > > > >>
>> > > > > > > >> As shown in this figure, messages are sent to brokers by
>> > > producer
>> > > > > and
>> > > > > > > >> stored in commit log[1], then messages are dispatched to
>> > > > > ConsumeQueue
>> > > > > > by
>> > > > > > > >> topic, the consumer pulls messages from the queue.
>> > > > > > > >>
>> > > > > > > >> I recommend you run a broker and send/consume some
>> messages,
>> > > then
>> > > > > > check
>> > > > > > > >> out the ~/store directory for details.
>> > > > > > > >>
>> > > > > > > >> Regards,
>> > > > > > > >> yukon
>> > > > > > > >>
>> > > > > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <
>> > sowmya9029@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > > > >>
>> > > > > > > >>> Hello yukon,
>> > > > > > > >>>
>> > > > > > > >>> Currently FIFO can be achieved with a producer sending to
>> one
>> > > > > message
>> > > > > > > >>> queue, and when global ordering is required, multiple
>> > producers
>> > > > > have
>> > > > > > to
>> > > > > > > >>> send to a single topic queue.
>> > > > > > > >>>
>> > > > > > > >>> We want to allow multiple producers to send messages on a
>> > topic
>> > > > to
>> > > > > > > >>> multiple message queues and still provide ordering
>> guarantees
>> > > to
>> > > > > the
>> > > > > > > >>> consumer, so that all consumers see the same order of data
>> > and
>> > > > also
>> > > > > > the
>> > > > > > > >>> data is delivered in an ordered fashion.
>> > > > > > > >>>
>> > > > > > > >>> 1) Your idea of using a merge sort with the assumption
>> that
>> > the
>> > > > > first
>> > > > > > > >>> arriving message is treated as the first message to be
>> > > delivered,
>> > > > > > > however,
>> > > > > > > >>> I want to propose an approach where when the producer
>> sends a
>> > > > > message
>> > > > > > > to a
>> > > > > > > >>> message queue, it must be done in a synchronous fashion
>> and
>> > the
>> > > > > > > response
>> > > > > > > >>> will be that the message is accepted, which means that the
>> > > > message
>> > > > > > > follows
>> > > > > > > >>> the convention that all messages delivered prior by that
>> > > producer
>> > > > > > have
>> > > > > > > been
>> > > > > > > >>> stored across groups and if not the producer will need to
>> > > resend
>> > > > > the
>> > > > > > > >>> message.
>> > > > > > > >>>
>> > > > > > > >>> We can use a variant of total causal ordering in the layer
>> > > > between
>> > > > > > the
>> > > > > > > >>> message queue and store.
>> > > > > > > >>>
>> > > > > > > >>> I have been busy with my class project so I couldn't make
>> a
>> > lot
>> > > > of
>> > > > > > > >>> progress in detailing my approach. Also, it would be
>> great if
>> > > you
>> > > > > can
>> > > > > > > at a
>> > > > > > > >>> high level, help me understand how the messages in the
>> > message
>> > > > > queues
>> > > > > > > are
>> > > > > > > >>> stored before the consumer reads them.
>> > > > > > > >>>
>> > > > > > > >>> Does the consumer read directly from the message queue
>> that
>> > the
>> > > > > > > producer
>> > > > > > > >>> sends data to? does the broker receive the queued producer
>> > > > > messages,
>> > > > > > > store
>> > > > > > > >>> them and then pushes them to another queue for the
>> consumer
>> > to
>> > > > read
>> > > > > > > from?
>> > > > > > > >>>
>> > > > > > > >>> For reference on total causal ordering:
>> > > > > > https://www.cl.cam.ac.uk/teach
>> > > > > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
>> > > > > > > >>>
>> > > > > > > >>> thanks,
>> > > > > > > >>> Sowmya
>> > > > > > > >>>
>> > > > > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org>
>> > > wrote:
>> > > > > > > >>>
>> > > > > > > >>>> Hi Sowmya,
>> > > > > > > >>>>
>> > > > > > > >>>> Sorry for the late reply, do you have any update on this
>> > > > project?
>> > > > > > > >>>>
>> > > > > > > >>>> In RocketMQ, one message queue is a FIFO queue natively,
>> so
>> > I
>> > > > > > proposed
>> > > > > > > >>>> a simple solution that performs merge sort on multiple
>> > queues
>> > > to
>> > > > > > > improve
>> > > > > > > >>>> performance and scalability.
>> > > > > > > >>>>
>> > > > > > > >>>> While the order issue across producers is difficult, we
>> > could
>> > > > > assume
>> > > > > > > >>>> that the message first arrives the broker should be
>> consumed
>> > > > > > firstly.
>> > > > > > > >>>>
>> > > > > > > >>>> But it would be wonderful if you have a real design about
>> > the
>> > > > > order
>> > > > > > > >>>> issue across producers based on the RocketMQ design and
>> the
>> > > > > storage
>> > > > > > > >>>> structure.
>> > > > > > > >>>>
>> > > > > > > >>>> Looking forward your design ~
>> > > > > > > >>>>
>> > > > > > > >>>> Regards,
>> > > > > > > >>>> yukon
>> > > > > > > >>>>
>> > > > > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <
>> > > sowmya9029@gmail.com>
>> > > > > > > wrote:
>> > > > > > > >>>>
>> > > > > > > >>>>> Hello all,
>> > > > > > > >>>>>
>> > > > > > > >>>>> Adding a few more thoughts on the problem of
>> establishing
>> > an
>> > > > > order
>> > > > > > of
>> > > > > > > >>>>> messages across producers.
>> > > > > > > >>>>>
>> > > > > > > >>>>> Consider the Scenario
>> > > > > > > >>>>>
>> > > > > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue
>> > > Queue1
>> > > > > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue
>> > Queue2.
>> > > > > > > >>>>>
>> > > > > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and
>> > > similarly
>> > > > > > > >>>>> time(a2) < time(b2) < time(c2)
>> > > > > > > >>>>>
>> > > > > > > >>>>> Are the following orders acceptable to the consumer?
>> > > > > > > >>>>>
>> > > > > > > >>>>> > a1, a2, b1, b2, c1, c2
>> > > > > > > >>>>> > a1, b1, c1, a2, b2, c2
>> > > > > > > >>>>> > a1 b1, b2, a2, a3, b3
>> > > > > > > >>>>>
>> > > > > > > >>>>> and an order displayed at one consumer is consistent
>> across
>> > > all
>> > > > > > > >>>>> consumers.
>> > > > > > > >>>>>
>> > > > > > > >>>>> This can be achieved using Total Causal Ordering at the
>> > > > Producer
>> > > > > or
>> > > > > > > >>>>> Queue level, using Leslie Lamport's clock and
>> > synchronization
>> > > > > > > approach.
>> > > > > > > >>>>>
>> > > > > > > >>>>> For reference is the paper attached,
>> > > > > > http://lamport.azurewebsites.n
>> > > > > > > >>>>> et/pubs/time-clocks.pdf
>> > > > > > > >>>>>
>> > > > > > > >>>>>
>> > > > > > > >>>>> --
>> > > > > > > >>>>> Regards,
>> > > > > > > >>>>>
>> > > > > > > >>>>> Sowmya
>> > > > > > > >>>>>
>> > > > > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <
>> > > > sowmya9029@gmail.com>
>> > > > > > > >>>>> wrote:
>> > > > > > > >>>>>
>> > > > > > > >>>>>> Hello,
>> > > > > > > >>>>>>
>> > > > > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122
>> > > > > > > >>>>>> <
>> > > > > > https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=
>> 12343065
>> > >
>> > > > > > > as
>> > > > > > > >>>>>> a part of Google Summer of Code 2018. I've been
>> spending
>> > > some
>> > > > > time
>> > > > > > > to
>> > > > > > > >>>>>> understand the system, architecture and the existing
>> > > Messaging
>> > > > > > > Patterns.
>> > > > > > > >>>>>> I still have a few questions and would like to clarify
>> my
>> > > > > > > assumptions.
>> > > > > > > >>>>>>
>> > > > > > > >>>>>> - Is the current FIFO order example limited to one
>> > > message
>> > > > > > queue
>> > > > > > > >>>>>> per producer? Can the producer send the same
>> message to
>> > > > > > multiple
>> > > > > > > queues?
>> > > > > > > >>>>>> Will the consumers of the queues be able to read
>> > messages
>> > > > in
>> > > > > > > Order?
>> > > > > > > >>>>>> - Can I assume that each producer will send
>> messages to
>> > > one
>> > > > > > > queue?
>> > > > > > > >>>>>> - Global Order is to be identified across all
>> > > > > > > >>>>>> GlobalOrderedProducer (a new producer that is to be
>> > used
>> > > > for
>> > > > > > > global order)
>> > > > > > > >>>>>> instances that are running.
>> > > > > > > >>>>>> - I think using a global clock can help establish
>> the
>> > > order
>> > > > > > > >>>>>> between 2 or more producers, however using some
>> form of
>> > > > > vector
>> > > > > > > >>>>>> clock might also help identify the global order of
>> > > messages
>> > > > > > > between the
>> > > > > > > >>>>>> producers.
>> > > > > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how
>> to
>> > > read
>> > > > > > > >>>>>> globally ordered messages) can then compare messages
>> > > across
>> > > > > all
>> > > > > > > message
>> > > > > > > >>>>>> queues from the corresponding producers and extract
>> the
>> > > > > > > messages. [ this is
>> > > > > > > >>>>>> the approach recommended by yukon on the issue page
>> ]
>> > > > > > > >>>>>> - We can also potentially have another layer in the
>> > > Message
>> > > > > > Queue
>> > > > > > > >>>>>> which accumulates all messages sent from producers
>> and
>> > > > > provides
>> > > > > > > one ordered
>> > > > > > > >>>>>> message queue for consumers to read from.
>> > > > > > > >>>>>>
>> > > > > > > >>>>>> Thank you for your patience and please let me know if
>> my
>> > > > > > > >>>>>> understanding of the problem and the assumptions are
>> > right.
>> > > > > > > >>>>>>
>> > > > > > > >>>>>> Best Regards,
>> > > > > > > >>>>>> Sowmya
>> > > > > > > >>>>>>
>> > > > > > > >>>>>>
>> > > > > > > >>>>>
>> > > > > > > >>>>>
>> > > > > > > >>>>> --
>> > > > > > > >>>>> Regards,
>> > > > > > > >>>>>
>> > > > > > > >>>>> Sowmya
>> > > > > > > >>>>>
>> > > > > > > >>>>>
>> > > > > > > >>>>
>> > > > > > > >>>
>> > > > > > > >>>
>> > > > > > > >>> --
>> > > > > > > >>> Regards,
>> > > > > > > >>>
>> > > > > > > >>> Sowmya
>> > > > > > > >>>
>> > > > > > > >>>
>> > > > > > > >>
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > Regards,
>> > > > > > > >
>> > > > > > > > Sowmya
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > Regards,
>> > > > > > >
>> > > > > > > Sowmya
>> > > > > > >
>> > > > > >
>> > > > > --
>> > > > > Regards,
>> > > > >
>> > > > > Sowmya
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Regards,
>> > >
>> > > Sowmya
>> > >
>> >
>>
>>
>>
>> --
>> Regards,
>>
>> Sowmya
>>
>
>
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by yukon <yu...@apache.org>.
Cool, thanks for your efforts.
On Wed, Mar 28, 2018 at 9:02 AM, sowmya s <so...@gmail.com> wrote:
> Hello Yukon,
> I've submitted my final proposal addressing your comments as well. Hoping
> to work on this project for the summer. In the meantime i'm working on
> prototyping the global ordering approach i talked about in my proposal.
>
> thanks,
> Sowmya
>
> On Fri, Mar 23, 2018 at 4:48 AM, yukon <yu...@apache.org> wrote:
>
> > Hi,
> >
> > I added some comments, but it seems your google doc link isn't shared:
> >
> > ```
> > To resolve this issue, we need a global ordered messaging mechanism to
> have
> > a good scalability without hot-pot issue.
> >
> > A practical proposal is that sending messages to distributed
> > queue/partition with an ordered key. Messages are ordered in the same
> > queue, then implement an OrderConsumer to pull messages from these queues
> > then perform merge sort, finally deliver these ordered messages to users.
> > ```
> > As mentioned in ROCKETMQ-122, we should address the hot-pot issue.
> >
> > That means the messages can be sent to multiple distributed
> > queues/partitions and still can be consumed in order manner.
> >
> > A simple explanation of hot-pot issue:
> >
> > Assume that all the generated messages in a hot shop of taobao/amazon
> > should be consumed in order, in the current state, these massive messages
> > should be sent to a fixed queue/partition, surely to a fixed server which
> > may cause high pressure in this server. You can easily infer that we
> can't
> > resolve this issue through extending the scale of broker cluster.
> >
> > Could your arch resolve this hot-pot issue?
> >
> > Regards,
> > yukon
> >
> >
> > On Thu, Mar 22, 2018 at 2:15 PM, sowmya s <so...@gmail.com> wrote:
> >
> > > It's going great.
> > > In the DefaultMessageStore, there is a class
> > > CommitLogDispatcherBuildConsumeQueue, I think that a variant of that
> > class
> > > or another implementation which are merged from the commitlog,
> > > Since the DispachRequest has a store timestamp, the queue can be
> > populated
> > > based on that time stamp.
> > >
> > > Also, I have some more questions,
> > > Does the fleet of broker slaves all store the various commit logs that
> > are
> > > received by them or is the commit log store, replication managed by the
> > > broker controller?
> > >
> > > The other direction I was thinking about is adding/modifying the
> > > PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
> > > DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe
> adding
> > > this change there will solve the merge issue for the consumers.
> > >
> > > Do you think I am on the right track?
> > >
> > > thanks,
> > > Sowmya
> > >
> > >
> > >
> > > On Mon, Mar 19, 2018 at 8:15 PM, yukon <yu...@apache.org> wrote:
> > >
> > > > Hi,
> > > >
> > > > Sorry for the late reply.
> > > >
> > > > As for:
> > > >
> > > > ```
> > > > I'm looking at the BrokerController and MessageStore implementation
> and
> > > > hooks to understand where the merge logic will best fit.
> > > > ```
> > > >
> > > > how is it going?
> > > >
> > > > Regards
> > > >
> > > > On Sat, Mar 17, 2018 at 11:30 AM, sowmya s <so...@gmail.com>
> > wrote:
> > > >
> > > > > Thank you yukon,
> > > > >
> > > > > I'm done with my coursework for this semester and have more time
> now
> > to
> > > > > improve my proposal.
> > > > > I'm looking at the BrokerController and MessageStore implementation
> > and
> > > > > hooks to understand where the merge logic will best fit. So far
> I've
> > > > looked
> > > > > at the codebase from a Producer and Consumer perspective and looked
> > at
> > > > the
> > > > > DefaultMQProducerImpl and DefaultMQPushConsumerImpl for
> > understanding
> > > > the
> > > > > link between how Producers send and Consumers receive messages.
> > > > >
> > > > > thanks,
> > > > > Sowmya
> > > > >
> > > > > On Fri, Mar 16, 2018 at 8:07 PM, yukon <yu...@apache.org> wrote:
> > > > >
> > > > > > Cool, let's focus on it and see whether is there anything can be
> > > > > polished.
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <sowmya9029@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Hey yukon,
> > > > > > > I submitted my draft on the summer of code homepage a couple of
> > > days
> > > > > ago,
> > > > > > > also attaching the link here for reference,
> > > > > > >
> > > > > > > https://docs.google.com/file/d/1nXktUO_TF9-
> > > > rSHSnGj5z5QZoHzMhosxm/edit?
> > > > > > > usp=docslist_api&filetype=msword
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Sowmya
> > > > > > >
> > > > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon <yu...@apache.org>
> wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > Of course, we can work together to finetune your design
> draft.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > yukon
> > > > > > > >
> > > > > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <
> > sowmya9029@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello, yukon and Von,
> > > > > > > > >
> > > > > > > > > I've shared my GSOC - 18 draft of the project. I'm looking
> > > > forward
> > > > > to
> > > > > > > > > working with all of you to finetune the proposal. I will be
> > > > > > allocating
> > > > > > > 20
> > > > > > > > > hours per week from now to the proposal acceptance phase to
> > > > address
> > > > > > > > > questions and dive deep into any suggestions that you
> > provide.
> > > > > > > > > I am really looking forward to work on this project.
> > > > > > > > >
> > > > > > > > > thanks,
> > > > > > > > > Sowmya
> > > > > > > > >
> > > > > > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <
> > > sowmya9029@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello yukon,
> > > > > > > > > >
> > > > > > > > > > Thank you for the inputs. I was able to look at the
> ~/store
> > > and
> > > > > > kind
> > > > > > > of
> > > > > > > > > > understand the storage structure. I also looked at the
> > > > > > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used
> > in
> > > > the
> > > > > > > > > examples.
> > > > > > > > > > Now I understand why you proposed a merge sort like
> > approach
> > > > for
> > > > > > > > > > performing global ordering. Since the proposals are
> open, I
> > > am
> > > > > > > > finalizing
> > > > > > > > > > my draft and will have it up for review very soon.
> > > > > > > > > >
> > > > > > > > > > thanks,
> > > > > > > > > > Sowmya
> > > > > > > > > >
> > > > > > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org>
> > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi Sowmya,
> > > > > > > > > >>
> > > > > > > > > >> ```
> > > > > > > > > >> Also, it would be great if you can at a high level, help
> > me
> > > > > > > understand
> > > > > > > > > >> how the messages in the message queues are stored before
> > the
> > > > > > > consumer
> > > > > > > > > reads
> > > > > > > > > >> them.
> > > > > > > > > >> ```
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> As shown in this figure, messages are sent to brokers by
> > > > > producer
> > > > > > > and
> > > > > > > > > >> stored in commit log[1], then messages are dispatched to
> > > > > > > ConsumeQueue
> > > > > > > > by
> > > > > > > > > >> topic, the consumer pulls messages from the queue.
> > > > > > > > > >>
> > > > > > > > > >> I recommend you run a broker and send/consume some
> > messages,
> > > > > then
> > > > > > > > check
> > > > > > > > > >> out the ~/store directory for details.
> > > > > > > > > >>
> > > > > > > > > >> Regards,
> > > > > > > > > >> yukon
> > > > > > > > > >>
> > > > > > > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <
> > > > sowmya9029@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >>> Hello yukon,
> > > > > > > > > >>>
> > > > > > > > > >>> Currently FIFO can be achieved with a producer sending
> to
> > > one
> > > > > > > message
> > > > > > > > > >>> queue, and when global ordering is required, multiple
> > > > producers
> > > > > > > have
> > > > > > > > to
> > > > > > > > > >>> send to a single topic queue.
> > > > > > > > > >>>
> > > > > > > > > >>> We want to allow multiple producers to send messages
> on a
> > > > topic
> > > > > > to
> > > > > > > > > >>> multiple message queues and still provide ordering
> > > guarantees
> > > > > to
> > > > > > > the
> > > > > > > > > >>> consumer, so that all consumers see the same order of
> > data
> > > > and
> > > > > > also
> > > > > > > > the
> > > > > > > > > >>> data is delivered in an ordered fashion.
> > > > > > > > > >>>
> > > > > > > > > >>> 1) Your idea of using a merge sort with the assumption
> > that
> > > > the
> > > > > > > first
> > > > > > > > > >>> arriving message is treated as the first message to be
> > > > > delivered,
> > > > > > > > > however,
> > > > > > > > > >>> I want to propose an approach where when the producer
> > > sends a
> > > > > > > message
> > > > > > > > > to a
> > > > > > > > > >>> message queue, it must be done in a synchronous fashion
> > and
> > > > the
> > > > > > > > > response
> > > > > > > > > >>> will be that the message is accepted, which means that
> > the
> > > > > > message
> > > > > > > > > follows
> > > > > > > > > >>> the convention that all messages delivered prior by
> that
> > > > > producer
> > > > > > > > have
> > > > > > > > > been
> > > > > > > > > >>> stored across groups and if not the producer will need
> to
> > > > > resend
> > > > > > > the
> > > > > > > > > >>> message.
> > > > > > > > > >>>
> > > > > > > > > >>> We can use a variant of total causal ordering in the
> > layer
> > > > > > between
> > > > > > > > the
> > > > > > > > > >>> message queue and store.
> > > > > > > > > >>>
> > > > > > > > > >>> I have been busy with my class project so I couldn't
> > make a
> > > > lot
> > > > > > of
> > > > > > > > > >>> progress in detailing my approach. Also, it would be
> > great
> > > if
> > > > > you
> > > > > > > can
> > > > > > > > > at a
> > > > > > > > > >>> high level, help me understand how the messages in the
> > > > message
> > > > > > > queues
> > > > > > > > > are
> > > > > > > > > >>> stored before the consumer reads them.
> > > > > > > > > >>>
> > > > > > > > > >>> Does the consumer read directly from the message queue
> > that
> > > > the
> > > > > > > > > producer
> > > > > > > > > >>> sends data to? does the broker receive the queued
> > producer
> > > > > > > messages,
> > > > > > > > > store
> > > > > > > > > >>> them and then pushes them to another queue for the
> > consumer
> > > > to
> > > > > > read
> > > > > > > > > from?
> > > > > > > > > >>>
> > > > > > > > > >>> For reference on total causal ordering:
> > > > > > > > https://www.cl.cam.ac.uk/teach
> > > > > > > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
> > > > > > > > > >>>
> > > > > > > > > >>> thanks,
> > > > > > > > > >>> Sowmya
> > > > > > > > > >>>
> > > > > > > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <
> yukon@apache.org>
> > > > > wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> Hi Sowmya,
> > > > > > > > > >>>>
> > > > > > > > > >>>> Sorry for the late reply, do you have any update on
> this
> > > > > > project?
> > > > > > > > > >>>>
> > > > > > > > > >>>> In RocketMQ, one message queue is a FIFO queue
> natively,
> > > so
> > > > I
> > > > > > > > proposed
> > > > > > > > > >>>> a simple solution that performs merge sort on multiple
> > > > queues
> > > > > to
> > > > > > > > > improve
> > > > > > > > > >>>> performance and scalability.
> > > > > > > > > >>>>
> > > > > > > > > >>>> While the order issue across producers is difficult,
> we
> > > > could
> > > > > > > assume
> > > > > > > > > >>>> that the message first arrives the broker should be
> > > consumed
> > > > > > > > firstly.
> > > > > > > > > >>>>
> > > > > > > > > >>>> But it would be wonderful if you have a real design
> > about
> > > > the
> > > > > > > order
> > > > > > > > > >>>> issue across producers based on the RocketMQ design
> and
> > > the
> > > > > > > storage
> > > > > > > > > >>>> structure.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Looking forward your design ~
> > > > > > > > > >>>>
> > > > > > > > > >>>> Regards,
> > > > > > > > > >>>> yukon
> > > > > > > > > >>>>
> > > > > > > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <
> > > > > sowmya9029@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >>>>
> > > > > > > > > >>>>> Hello all,
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Adding a few more thoughts on the problem of
> > establishing
> > > > an
> > > > > > > order
> > > > > > > > of
> > > > > > > > > >>>>> messages across producers.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Consider the Scenario
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a
> queue
> > > > > Queue1
> > > > > > > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue
> > > > Queue2.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and
> > > > > similarly
> > > > > > > > > >>>>> time(a2) < time(b2) < time(c2)
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Are the following orders acceptable to the consumer?
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> > a1, a2, b1, b2, c1, c2
> > > > > > > > > >>>>> > a1, b1, c1, a2, b2, c2
> > > > > > > > > >>>>> > a1 b1, b2, a2, a3, b3
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> and an order displayed at one consumer is consistent
> > > across
> > > > > all
> > > > > > > > > >>>>> consumers.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> This can be achieved using Total Causal Ordering at
> the
> > > > > > Producer
> > > > > > > or
> > > > > > > > > >>>>> Queue level, using Leslie Lamport's clock and
> > > > synchronization
> > > > > > > > > approach.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> For reference is the paper attached,
> > > > > > > > http://lamport.azurewebsites.n
> > > > > > > > > >>>>> et/pubs/time-clocks.pdf
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> --
> > > > > > > > > >>>>> Regards,
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Sowmya
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <
> > > > > > sowmya9029@gmail.com>
> > > > > > > > > >>>>> wrote:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>> Hello,
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122
> > > > > > > > > >>>>>> <
> > > > > > > > https://issues.apache.org/jira/browse/ROCKETMQ-122?
> > > filter=12343065
> > > > >
> > > > > > > > > as
> > > > > > > > > >>>>>> a part of Google Summer of Code 2018. I've been
> > spending
> > > > > some
> > > > > > > time
> > > > > > > > > to
> > > > > > > > > >>>>>> understand the system, architecture and the existing
> > > > > Messaging
> > > > > > > > > Patterns.
> > > > > > > > > >>>>>> I still have a few questions and would like to
> clarify
> > > my
> > > > > > > > > assumptions.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> - Is the current FIFO order example limited to
> one
> > > > > message
> > > > > > > > queue
> > > > > > > > > >>>>>> per producer? Can the producer send the same
> > message
> > > to
> > > > > > > > multiple
> > > > > > > > > queues?
> > > > > > > > > >>>>>> Will the consumers of the queues be able to read
> > > > messages
> > > > > > in
> > > > > > > > > Order?
> > > > > > > > > >>>>>> - Can I assume that each producer will send
> > messages
> > > to
> > > > > one
> > > > > > > > > queue?
> > > > > > > > > >>>>>> - Global Order is to be identified across all
> > > > > > > > > >>>>>> GlobalOrderedProducer (a new producer that is to
> be
> > > > used
> > > > > > for
> > > > > > > > > global order)
> > > > > > > > > >>>>>> instances that are running.
> > > > > > > > > >>>>>> - I think using a global clock can help establish
> > the
> > > > > order
> > > > > > > > > >>>>>> between 2 or more producers, however using some
> > form
> > > of
> > > > > > > vector
> > > > > > > > > >>>>>> clock might also help identify the global order
> of
> > > > > messages
> > > > > > > > > between the
> > > > > > > > > >>>>>> producers.
> > > > > > > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows
> how
> > > to
> > > > > read
> > > > > > > > > >>>>>> globally ordered messages) can then compare
> > messages
> > > > > across
> > > > > > > all
> > > > > > > > > message
> > > > > > > > > >>>>>> queues from the corresponding producers and
> extract
> > > the
> > > > > > > > > messages. [ this is
> > > > > > > > > >>>>>> the approach recommended by yukon on the issue
> > page ]
> > > > > > > > > >>>>>> - We can also potentially have another layer in
> the
> > > > > Message
> > > > > > > > Queue
> > > > > > > > > >>>>>> which accumulates all messages sent from
> producers
> > > and
> > > > > > > provides
> > > > > > > > > one ordered
> > > > > > > > > >>>>>> message queue for consumers to read from.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Thank you for your patience and please let me know
> if
> > my
> > > > > > > > > >>>>>> understanding of the problem and the assumptions are
> > > > right.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Best Regards,
> > > > > > > > > >>>>>> Sowmya
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> --
> > > > > > > > > >>>>> Regards,
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Sowmya
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> --
> > > > > > > > > >>> Regards,
> > > > > > > > > >>>
> > > > > > > > > >>> Sowmya
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Regards,
> > > > > > > > > >
> > > > > > > > > > Sowmya
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Regards,
> > > > > > > > >
> > > > > > > > > Sowmya
> > > > > > > > >
> > > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > >
> > > > > > > Sowmya
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Sowmya
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Sowmya
> > >
> >
>
>
>
> --
> Regards,
>
> Sowmya
>
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by sowmya s <so...@gmail.com>.
Hello Yukon,
I've submitted my final proposal addressing your comments as well. Hoping
to work on this project for the summer. In the meantime i'm working on
prototyping the global ordering approach i talked about in my proposal.
thanks,
Sowmya
On Fri, Mar 23, 2018 at 4:48 AM, yukon <yu...@apache.org> wrote:
> Hi,
>
> I added some comments, but it seems your google doc link isn't shared:
>
> ```
> To resolve this issue, we need a global ordered messaging mechanism to have
> a good scalability without hot-pot issue.
>
> A practical proposal is that sending messages to distributed
> queue/partition with an ordered key. Messages are ordered in the same
> queue, then implement an OrderConsumer to pull messages from these queues
> then perform merge sort, finally deliver these ordered messages to users.
> ```
> As mentioned in ROCKETMQ-122, we should address the hot-pot issue.
>
> That means the messages can be sent to multiple distributed
> queues/partitions and still can be consumed in order manner.
>
> A simple explanation of hot-pot issue:
>
> Assume that all the generated messages in a hot shop of taobao/amazon
> should be consumed in order, in the current state, these massive messages
> should be sent to a fixed queue/partition, surely to a fixed server which
> may cause high pressure in this server. You can easily infer that we can't
> resolve this issue through extending the scale of broker cluster.
>
> Could your arch resolve this hot-pot issue?
>
> Regards,
> yukon
>
>
> On Thu, Mar 22, 2018 at 2:15 PM, sowmya s <so...@gmail.com> wrote:
>
> > It's going great.
> > In the DefaultMessageStore, there is a class
> > CommitLogDispatcherBuildConsumeQueue, I think that a variant of that
> class
> > or another implementation which are merged from the commitlog,
> > Since the DispachRequest has a store timestamp, the queue can be
> populated
> > based on that time stamp.
> >
> > Also, I have some more questions,
> > Does the fleet of broker slaves all store the various commit logs that
> are
> > received by them or is the commit log store, replication managed by the
> > broker controller?
> >
> > The other direction I was thinking about is adding/modifying the
> > PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
> > DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding
> > this change there will solve the merge issue for the consumers.
> >
> > Do you think I am on the right track?
> >
> > thanks,
> > Sowmya
> >
> >
> >
> > On Mon, Mar 19, 2018 at 8:15 PM, yukon <yu...@apache.org> wrote:
> >
> > > Hi,
> > >
> > > Sorry for the late reply.
> > >
> > > As for:
> > >
> > > ```
> > > I'm looking at the BrokerController and MessageStore implementation and
> > > hooks to understand where the merge logic will best fit.
> > > ```
> > >
> > > how is it going?
> > >
> > > Regards
> > >
> > > On Sat, Mar 17, 2018 at 11:30 AM, sowmya s <so...@gmail.com>
> wrote:
> > >
> > > > Thank you yukon,
> > > >
> > > > I'm done with my coursework for this semester and have more time now
> to
> > > > improve my proposal.
> > > > I'm looking at the BrokerController and MessageStore implementation
> and
> > > > hooks to understand where the merge logic will best fit. So far I've
> > > looked
> > > > at the codebase from a Producer and Consumer perspective and looked
> at
> > > the
> > > > DefaultMQProducerImpl and DefaultMQPushConsumerImpl for
> understanding
> > > the
> > > > link between how Producers send and Consumers receive messages.
> > > >
> > > > thanks,
> > > > Sowmya
> > > >
> > > > On Fri, Mar 16, 2018 at 8:07 PM, yukon <yu...@apache.org> wrote:
> > > >
> > > > > Cool, let's focus on it and see whether is there anything can be
> > > > polished.
> > > > >
> > > > > Regards
> > > > >
> > > > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <so...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hey yukon,
> > > > > > I submitted my draft on the summer of code homepage a couple of
> > days
> > > > ago,
> > > > > > also attaching the link here for reference,
> > > > > >
> > > > > > https://docs.google.com/file/d/1nXktUO_TF9-
> > > rSHSnGj5z5QZoHzMhosxm/edit?
> > > > > > usp=docslist_api&filetype=msword
> > > > > >
> > > > > > Thanks,
> > > > > > Sowmya
> > > > > >
> > > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon <yu...@apache.org> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Of course, we can work together to finetune your design draft.
> > > > > > >
> > > > > > > Regards,
> > > > > > > yukon
> > > > > > >
> > > > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <
> sowmya9029@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hello, yukon and Von,
> > > > > > > >
> > > > > > > > I've shared my GSOC - 18 draft of the project. I'm looking
> > > forward
> > > > to
> > > > > > > > working with all of you to finetune the proposal. I will be
> > > > > allocating
> > > > > > 20
> > > > > > > > hours per week from now to the proposal acceptance phase to
> > > address
> > > > > > > > questions and dive deep into any suggestions that you
> provide.
> > > > > > > > I am really looking forward to work on this project.
> > > > > > > >
> > > > > > > > thanks,
> > > > > > > > Sowmya
> > > > > > > >
> > > > > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <
> > sowmya9029@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello yukon,
> > > > > > > > >
> > > > > > > > > Thank you for the inputs. I was able to look at the ~/store
> > and
> > > > > kind
> > > > > > of
> > > > > > > > > understand the storage structure. I also looked at the
> > > > > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used
> in
> > > the
> > > > > > > > examples.
> > > > > > > > > Now I understand why you proposed a merge sort like
> approach
> > > for
> > > > > > > > > performing global ordering. Since the proposals are open, I
> > am
> > > > > > > finalizing
> > > > > > > > > my draft and will have it up for review very soon.
> > > > > > > > >
> > > > > > > > > thanks,
> > > > > > > > > Sowmya
> > > > > > > > >
> > > > > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org>
> > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi Sowmya,
> > > > > > > > >>
> > > > > > > > >> ```
> > > > > > > > >> Also, it would be great if you can at a high level, help
> me
> > > > > > understand
> > > > > > > > >> how the messages in the message queues are stored before
> the
> > > > > > consumer
> > > > > > > > reads
> > > > > > > > >> them.
> > > > > > > > >> ```
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> As shown in this figure, messages are sent to brokers by
> > > > producer
> > > > > > and
> > > > > > > > >> stored in commit log[1], then messages are dispatched to
> > > > > > ConsumeQueue
> > > > > > > by
> > > > > > > > >> topic, the consumer pulls messages from the queue.
> > > > > > > > >>
> > > > > > > > >> I recommend you run a broker and send/consume some
> messages,
> > > > then
> > > > > > > check
> > > > > > > > >> out the ~/store directory for details.
> > > > > > > > >>
> > > > > > > > >> Regards,
> > > > > > > > >> yukon
> > > > > > > > >>
> > > > > > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <
> > > sowmya9029@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> Hello yukon,
> > > > > > > > >>>
> > > > > > > > >>> Currently FIFO can be achieved with a producer sending to
> > one
> > > > > > message
> > > > > > > > >>> queue, and when global ordering is required, multiple
> > > producers
> > > > > > have
> > > > > > > to
> > > > > > > > >>> send to a single topic queue.
> > > > > > > > >>>
> > > > > > > > >>> We want to allow multiple producers to send messages on a
> > > topic
> > > > > to
> > > > > > > > >>> multiple message queues and still provide ordering
> > guarantees
> > > > to
> > > > > > the
> > > > > > > > >>> consumer, so that all consumers see the same order of
> data
> > > and
> > > > > also
> > > > > > > the
> > > > > > > > >>> data is delivered in an ordered fashion.
> > > > > > > > >>>
> > > > > > > > >>> 1) Your idea of using a merge sort with the assumption
> that
> > > the
> > > > > > first
> > > > > > > > >>> arriving message is treated as the first message to be
> > > > delivered,
> > > > > > > > however,
> > > > > > > > >>> I want to propose an approach where when the producer
> > sends a
> > > > > > message
> > > > > > > > to a
> > > > > > > > >>> message queue, it must be done in a synchronous fashion
> and
> > > the
> > > > > > > > response
> > > > > > > > >>> will be that the message is accepted, which means that
> the
> > > > > message
> > > > > > > > follows
> > > > > > > > >>> the convention that all messages delivered prior by that
> > > > producer
> > > > > > > have
> > > > > > > > been
> > > > > > > > >>> stored across groups and if not the producer will need to
> > > > resend
> > > > > > the
> > > > > > > > >>> message.
> > > > > > > > >>>
> > > > > > > > >>> We can use a variant of total causal ordering in the
> layer
> > > > > between
> > > > > > > the
> > > > > > > > >>> message queue and store.
> > > > > > > > >>>
> > > > > > > > >>> I have been busy with my class project so I couldn't
> make a
> > > lot
> > > > > of
> > > > > > > > >>> progress in detailing my approach. Also, it would be
> great
> > if
> > > > you
> > > > > > can
> > > > > > > > at a
> > > > > > > > >>> high level, help me understand how the messages in the
> > > message
> > > > > > queues
> > > > > > > > are
> > > > > > > > >>> stored before the consumer reads them.
> > > > > > > > >>>
> > > > > > > > >>> Does the consumer read directly from the message queue
> that
> > > the
> > > > > > > > producer
> > > > > > > > >>> sends data to? does the broker receive the queued
> producer
> > > > > > messages,
> > > > > > > > store
> > > > > > > > >>> them and then pushes them to another queue for the
> consumer
> > > to
> > > > > read
> > > > > > > > from?
> > > > > > > > >>>
> > > > > > > > >>> For reference on total causal ordering:
> > > > > > > https://www.cl.cam.ac.uk/teach
> > > > > > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
> > > > > > > > >>>
> > > > > > > > >>> thanks,
> > > > > > > > >>> Sowmya
> > > > > > > > >>>
> > > > > > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org>
> > > > wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Hi Sowmya,
> > > > > > > > >>>>
> > > > > > > > >>>> Sorry for the late reply, do you have any update on this
> > > > > project?
> > > > > > > > >>>>
> > > > > > > > >>>> In RocketMQ, one message queue is a FIFO queue natively,
> > so
> > > I
> > > > > > > proposed
> > > > > > > > >>>> a simple solution that performs merge sort on multiple
> > > queues
> > > > to
> > > > > > > > improve
> > > > > > > > >>>> performance and scalability.
> > > > > > > > >>>>
> > > > > > > > >>>> While the order issue across producers is difficult, we
> > > could
> > > > > > assume
> > > > > > > > >>>> that the message first arrives the broker should be
> > consumed
> > > > > > > firstly.
> > > > > > > > >>>>
> > > > > > > > >>>> But it would be wonderful if you have a real design
> about
> > > the
> > > > > > order
> > > > > > > > >>>> issue across producers based on the RocketMQ design and
> > the
> > > > > > storage
> > > > > > > > >>>> structure.
> > > > > > > > >>>>
> > > > > > > > >>>> Looking forward your design ~
> > > > > > > > >>>>
> > > > > > > > >>>> Regards,
> > > > > > > > >>>> yukon
> > > > > > > > >>>>
> > > > > > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <
> > > > sowmya9029@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >>>>
> > > > > > > > >>>>> Hello all,
> > > > > > > > >>>>>
> > > > > > > > >>>>> Adding a few more thoughts on the problem of
> establishing
> > > an
> > > > > > order
> > > > > > > of
> > > > > > > > >>>>> messages across producers.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Consider the Scenario
> > > > > > > > >>>>>
> > > > > > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue
> > > > Queue1
> > > > > > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue
> > > Queue2.
> > > > > > > > >>>>>
> > > > > > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and
> > > > similarly
> > > > > > > > >>>>> time(a2) < time(b2) < time(c2)
> > > > > > > > >>>>>
> > > > > > > > >>>>> Are the following orders acceptable to the consumer?
> > > > > > > > >>>>>
> > > > > > > > >>>>> > a1, a2, b1, b2, c1, c2
> > > > > > > > >>>>> > a1, b1, c1, a2, b2, c2
> > > > > > > > >>>>> > a1 b1, b2, a2, a3, b3
> > > > > > > > >>>>>
> > > > > > > > >>>>> and an order displayed at one consumer is consistent
> > across
> > > > all
> > > > > > > > >>>>> consumers.
> > > > > > > > >>>>>
> > > > > > > > >>>>> This can be achieved using Total Causal Ordering at the
> > > > > Producer
> > > > > > or
> > > > > > > > >>>>> Queue level, using Leslie Lamport's clock and
> > > synchronization
> > > > > > > > approach.
> > > > > > > > >>>>>
> > > > > > > > >>>>> For reference is the paper attached,
> > > > > > > http://lamport.azurewebsites.n
> > > > > > > > >>>>> et/pubs/time-clocks.pdf
> > > > > > > > >>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>> --
> > > > > > > > >>>>> Regards,
> > > > > > > > >>>>>
> > > > > > > > >>>>> Sowmya
> > > > > > > > >>>>>
> > > > > > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <
> > > > > sowmya9029@gmail.com>
> > > > > > > > >>>>> wrote:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> Hello,
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122
> > > > > > > > >>>>>> <
> > > > > > > https://issues.apache.org/jira/browse/ROCKETMQ-122?
> > filter=12343065
> > > >
> > > > > > > > as
> > > > > > > > >>>>>> a part of Google Summer of Code 2018. I've been
> spending
> > > > some
> > > > > > time
> > > > > > > > to
> > > > > > > > >>>>>> understand the system, architecture and the existing
> > > > Messaging
> > > > > > > > Patterns.
> > > > > > > > >>>>>> I still have a few questions and would like to clarify
> > my
> > > > > > > > assumptions.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> - Is the current FIFO order example limited to one
> > > > message
> > > > > > > queue
> > > > > > > > >>>>>> per producer? Can the producer send the same
> message
> > to
> > > > > > > multiple
> > > > > > > > queues?
> > > > > > > > >>>>>> Will the consumers of the queues be able to read
> > > messages
> > > > > in
> > > > > > > > Order?
> > > > > > > > >>>>>> - Can I assume that each producer will send
> messages
> > to
> > > > one
> > > > > > > > queue?
> > > > > > > > >>>>>> - Global Order is to be identified across all
> > > > > > > > >>>>>> GlobalOrderedProducer (a new producer that is to be
> > > used
> > > > > for
> > > > > > > > global order)
> > > > > > > > >>>>>> instances that are running.
> > > > > > > > >>>>>> - I think using a global clock can help establish
> the
> > > > order
> > > > > > > > >>>>>> between 2 or more producers, however using some
> form
> > of
> > > > > > vector
> > > > > > > > >>>>>> clock might also help identify the global order of
> > > > messages
> > > > > > > > between the
> > > > > > > > >>>>>> producers.
> > > > > > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how
> > to
> > > > read
> > > > > > > > >>>>>> globally ordered messages) can then compare
> messages
> > > > across
> > > > > > all
> > > > > > > > message
> > > > > > > > >>>>>> queues from the corresponding producers and extract
> > the
> > > > > > > > messages. [ this is
> > > > > > > > >>>>>> the approach recommended by yukon on the issue
> page ]
> > > > > > > > >>>>>> - We can also potentially have another layer in the
> > > > Message
> > > > > > > Queue
> > > > > > > > >>>>>> which accumulates all messages sent from producers
> > and
> > > > > > provides
> > > > > > > > one ordered
> > > > > > > > >>>>>> message queue for consumers to read from.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Thank you for your patience and please let me know if
> my
> > > > > > > > >>>>>> understanding of the problem and the assumptions are
> > > right.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Best Regards,
> > > > > > > > >>>>>> Sowmya
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>> --
> > > > > > > > >>>>> Regards,
> > > > > > > > >>>>>
> > > > > > > > >>>>> Sowmya
> > > > > > > > >>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> --
> > > > > > > > >>> Regards,
> > > > > > > > >>>
> > > > > > > > >>> Sowmya
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Regards,
> > > > > > > > >
> > > > > > > > > Sowmya
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Sowmya
> > > > > > > >
> > > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Sowmya
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Sowmya
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Sowmya
> >
>
--
Regards,
Sowmya
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by sowmya s <so...@gmail.com>.
Hi Yukon,
I am sorry for the confusion regarding the google doc, here is the link
again, https://drive.google.com/file/d/1nXktUO_TF9-
rSHSnGj5z5QZoHzMhosxm/view?usp=sharing
Regarding the hotpot issue, In my document, under the section Optimizing
for performance, I had a similar approach to what you mentioned, If the
Producer or the implementation of a Global Ordered Producer, attaches to
the Message or message properties, an additional attribute which is an
atomic counter that gets incremented for every message sent, in combination
with the topic and the unique producer group, the messages get to
potentially different brokers under different queues.
From the consumer perspective, having a custom RebalanceImplementation and
also a custom Implementation for a MQConsumer which can be a Pull or push
consumer that orders based on the property we set above in the Global
Ordered Producer and sort by producer group, topic all the messages.
thanks,
Sowmya
On Fri, Mar 23, 2018 at 4:48 AM, yukon <yu...@apache.org> wrote:
> Hi,
>
> I added some comments, but it seems your google doc link isn't shared:
>
> ```
> To resolve this issue, we need a global ordered messaging mechanism to have
> a good scalability without hot-pot issue.
>
> A practical proposal is that sending messages to distributed
> queue/partition with an ordered key. Messages are ordered in the same
> queue, then implement an OrderConsumer to pull messages from these queues
> then perform merge sort, finally deliver these ordered messages to users.
> ```
> As mentioned in ROCKETMQ-122, we should address the hot-pot issue.
>
> That means the messages can be sent to multiple distributed
> queues/partitions and still can be consumed in order manner.
>
> A simple explanation of hot-pot issue:
>
> Assume that all the generated messages in a hot shop of taobao/amazon
> should be consumed in order, in the current state, these massive messages
> should be sent to a fixed queue/partition, surely to a fixed server which
> may cause high pressure in this server. You can easily infer that we can't
> resolve this issue through extending the scale of broker cluster.
>
> Could your arch resolve this hot-pot issue?
>
> Regards,
> yukon
>
>
> On Thu, Mar 22, 2018 at 2:15 PM, sowmya s <so...@gmail.com> wrote:
>
> > It's going great.
> > In the DefaultMessageStore, there is a class
> > CommitLogDispatcherBuildConsumeQueue, I think that a variant of that
> class
> > or another implementation which are merged from the commitlog,
> > Since the DispachRequest has a store timestamp, the queue can be
> populated
> > based on that time stamp.
> >
> > Also, I have some more questions,
> > Does the fleet of broker slaves all store the various commit logs that
> are
> > received by them or is the commit log store, replication managed by the
> > broker controller?
> >
> > The other direction I was thinking about is adding/modifying the
> > PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
> > DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding
> > this change there will solve the merge issue for the consumers.
> >
> > Do you think I am on the right track?
> >
> > thanks,
> > Sowmya
> >
> >
> >
> > On Mon, Mar 19, 2018 at 8:15 PM, yukon <yu...@apache.org> wrote:
> >
> > > Hi,
> > >
> > > Sorry for the late reply.
> > >
> > > As for:
> > >
> > > ```
> > > I'm looking at the BrokerController and MessageStore implementation and
> > > hooks to understand where the merge logic will best fit.
> > > ```
> > >
> > > how is it going?
> > >
> > > Regards
> > >
> > > On Sat, Mar 17, 2018 at 11:30 AM, sowmya s <so...@gmail.com>
> wrote:
> > >
> > > > Thank you yukon,
> > > >
> > > > I'm done with my coursework for this semester and have more time now
> to
> > > > improve my proposal.
> > > > I'm looking at the BrokerController and MessageStore implementation
> and
> > > > hooks to understand where the merge logic will best fit. So far I've
> > > looked
> > > > at the codebase from a Producer and Consumer perspective and looked
> at
> > > the
> > > > DefaultMQProducerImpl and DefaultMQPushConsumerImpl for
> understanding
> > > the
> > > > link between how Producers send and Consumers receive messages.
> > > >
> > > > thanks,
> > > > Sowmya
> > > >
> > > > On Fri, Mar 16, 2018 at 8:07 PM, yukon <yu...@apache.org> wrote:
> > > >
> > > > > Cool, let's focus on it and see whether is there anything can be
> > > > polished.
> > > > >
> > > > > Regards
> > > > >
> > > > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <so...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hey yukon,
> > > > > > I submitted my draft on the summer of code homepage a couple of
> > days
> > > > ago,
> > > > > > also attaching the link here for reference,
> > > > > >
> > > > > > https://docs.google.com/file/d/1nXktUO_TF9-
> > > rSHSnGj5z5QZoHzMhosxm/edit?
> > > > > > usp=docslist_api&filetype=msword
> > > > > >
> > > > > > Thanks,
> > > > > > Sowmya
> > > > > >
> > > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon <yu...@apache.org> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Of course, we can work together to finetune your design draft.
> > > > > > >
> > > > > > > Regards,
> > > > > > > yukon
> > > > > > >
> > > > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <
> sowmya9029@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hello, yukon and Von,
> > > > > > > >
> > > > > > > > I've shared my GSOC - 18 draft of the project. I'm looking
> > > forward
> > > > to
> > > > > > > > working with all of you to finetune the proposal. I will be
> > > > > allocating
> > > > > > 20
> > > > > > > > hours per week from now to the proposal acceptance phase to
> > > address
> > > > > > > > questions and dive deep into any suggestions that you
> provide.
> > > > > > > > I am really looking forward to work on this project.
> > > > > > > >
> > > > > > > > thanks,
> > > > > > > > Sowmya
> > > > > > > >
> > > > > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <
> > sowmya9029@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello yukon,
> > > > > > > > >
> > > > > > > > > Thank you for the inputs. I was able to look at the ~/store
> > and
> > > > > kind
> > > > > > of
> > > > > > > > > understand the storage structure. I also looked at the
> > > > > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used
> in
> > > the
> > > > > > > > examples.
> > > > > > > > > Now I understand why you proposed a merge sort like
> approach
> > > for
> > > > > > > > > performing global ordering. Since the proposals are open, I
> > am
> > > > > > > finalizing
> > > > > > > > > my draft and will have it up for review very soon.
> > > > > > > > >
> > > > > > > > > thanks,
> > > > > > > > > Sowmya
> > > > > > > > >
> > > > > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org>
> > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi Sowmya,
> > > > > > > > >>
> > > > > > > > >> ```
> > > > > > > > >> Also, it would be great if you can at a high level, help
> me
> > > > > > understand
> > > > > > > > >> how the messages in the message queues are stored before
> the
> > > > > > consumer
> > > > > > > > reads
> > > > > > > > >> them.
> > > > > > > > >> ```
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> As shown in this figure, messages are sent to brokers by
> > > > producer
> > > > > > and
> > > > > > > > >> stored in commit log[1], then messages are dispatched to
> > > > > > ConsumeQueue
> > > > > > > by
> > > > > > > > >> topic, the consumer pulls messages from the queue.
> > > > > > > > >>
> > > > > > > > >> I recommend you run a broker and send/consume some
> messages,
> > > > then
> > > > > > > check
> > > > > > > > >> out the ~/store directory for details.
> > > > > > > > >>
> > > > > > > > >> Regards,
> > > > > > > > >> yukon
> > > > > > > > >>
> > > > > > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <
> > > sowmya9029@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> Hello yukon,
> > > > > > > > >>>
> > > > > > > > >>> Currently FIFO can be achieved with a producer sending to
> > one
> > > > > > message
> > > > > > > > >>> queue, and when global ordering is required, multiple
> > > producers
> > > > > > have
> > > > > > > to
> > > > > > > > >>> send to a single topic queue.
> > > > > > > > >>>
> > > > > > > > >>> We want to allow multiple producers to send messages on a
> > > topic
> > > > > to
> > > > > > > > >>> multiple message queues and still provide ordering
> > guarantees
> > > > to
> > > > > > the
> > > > > > > > >>> consumer, so that all consumers see the same order of
> data
> > > and
> > > > > also
> > > > > > > the
> > > > > > > > >>> data is delivered in an ordered fashion.
> > > > > > > > >>>
> > > > > > > > >>> 1) Your idea of using a merge sort with the assumption
> that
> > > the
> > > > > > first
> > > > > > > > >>> arriving message is treated as the first message to be
> > > > delivered,
> > > > > > > > however,
> > > > > > > > >>> I want to propose an approach where when the producer
> > sends a
> > > > > > message
> > > > > > > > to a
> > > > > > > > >>> message queue, it must be done in a synchronous fashion
> and
> > > the
> > > > > > > > response
> > > > > > > > >>> will be that the message is accepted, which means that
> the
> > > > > message
> > > > > > > > follows
> > > > > > > > >>> the convention that all messages delivered prior by that
> > > > producer
> > > > > > > have
> > > > > > > > been
> > > > > > > > >>> stored across groups and if not the producer will need to
> > > > resend
> > > > > > the
> > > > > > > > >>> message.
> > > > > > > > >>>
> > > > > > > > >>> We can use a variant of total causal ordering in the
> layer
> > > > > between
> > > > > > > the
> > > > > > > > >>> message queue and store.
> > > > > > > > >>>
> > > > > > > > >>> I have been busy with my class project so I couldn't
> make a
> > > lot
> > > > > of
> > > > > > > > >>> progress in detailing my approach. Also, it would be
> great
> > if
> > > > you
> > > > > > can
> > > > > > > > at a
> > > > > > > > >>> high level, help me understand how the messages in the
> > > message
> > > > > > queues
> > > > > > > > are
> > > > > > > > >>> stored before the consumer reads them.
> > > > > > > > >>>
> > > > > > > > >>> Does the consumer read directly from the message queue
> that
> > > the
> > > > > > > > producer
> > > > > > > > >>> sends data to? does the broker receive the queued
> producer
> > > > > > messages,
> > > > > > > > store
> > > > > > > > >>> them and then pushes them to another queue for the
> consumer
> > > to
> > > > > read
> > > > > > > > from?
> > > > > > > > >>>
> > > > > > > > >>> For reference on total causal ordering:
> > > > > > > https://www.cl.cam.ac.uk/teach
> > > > > > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
> > > > > > > > >>>
> > > > > > > > >>> thanks,
> > > > > > > > >>> Sowmya
> > > > > > > > >>>
> > > > > > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org>
> > > > wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Hi Sowmya,
> > > > > > > > >>>>
> > > > > > > > >>>> Sorry for the late reply, do you have any update on this
> > > > > project?
> > > > > > > > >>>>
> > > > > > > > >>>> In RocketMQ, one message queue is a FIFO queue natively,
> > so
> > > I
> > > > > > > proposed
> > > > > > > > >>>> a simple solution that performs merge sort on multiple
> > > queues
> > > > to
> > > > > > > > improve
> > > > > > > > >>>> performance and scalability.
> > > > > > > > >>>>
> > > > > > > > >>>> While the order issue across producers is difficult, we
> > > could
> > > > > > assume
> > > > > > > > >>>> that the message first arrives the broker should be
> > consumed
> > > > > > > firstly.
> > > > > > > > >>>>
> > > > > > > > >>>> But it would be wonderful if you have a real design
> about
> > > the
> > > > > > order
> > > > > > > > >>>> issue across producers based on the RocketMQ design and
> > the
> > > > > > storage
> > > > > > > > >>>> structure.
> > > > > > > > >>>>
> > > > > > > > >>>> Looking forward your design ~
> > > > > > > > >>>>
> > > > > > > > >>>> Regards,
> > > > > > > > >>>> yukon
> > > > > > > > >>>>
> > > > > > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <
> > > > sowmya9029@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >>>>
> > > > > > > > >>>>> Hello all,
> > > > > > > > >>>>>
> > > > > > > > >>>>> Adding a few more thoughts on the problem of
> establishing
> > > an
> > > > > > order
> > > > > > > of
> > > > > > > > >>>>> messages across producers.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Consider the Scenario
> > > > > > > > >>>>>
> > > > > > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue
> > > > Queue1
> > > > > > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue
> > > Queue2.
> > > > > > > > >>>>>
> > > > > > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and
> > > > similarly
> > > > > > > > >>>>> time(a2) < time(b2) < time(c2)
> > > > > > > > >>>>>
> > > > > > > > >>>>> Are the following orders acceptable to the consumer?
> > > > > > > > >>>>>
> > > > > > > > >>>>> > a1, a2, b1, b2, c1, c2
> > > > > > > > >>>>> > a1, b1, c1, a2, b2, c2
> > > > > > > > >>>>> > a1 b1, b2, a2, a3, b3
> > > > > > > > >>>>>
> > > > > > > > >>>>> and an order displayed at one consumer is consistent
> > across
> > > > all
> > > > > > > > >>>>> consumers.
> > > > > > > > >>>>>
> > > > > > > > >>>>> This can be achieved using Total Causal Ordering at the
> > > > > Producer
> > > > > > or
> > > > > > > > >>>>> Queue level, using Leslie Lamport's clock and
> > > synchronization
> > > > > > > > approach.
> > > > > > > > >>>>>
> > > > > > > > >>>>> For reference is the paper attached,
> > > > > > > http://lamport.azurewebsites.n
> > > > > > > > >>>>> et/pubs/time-clocks.pdf
> > > > > > > > >>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>> --
> > > > > > > > >>>>> Regards,
> > > > > > > > >>>>>
> > > > > > > > >>>>> Sowmya
> > > > > > > > >>>>>
> > > > > > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <
> > > > > sowmya9029@gmail.com>
> > > > > > > > >>>>> wrote:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> Hello,
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122
> > > > > > > > >>>>>> <
> > > > > > > https://issues.apache.org/jira/browse/ROCKETMQ-122?
> > filter=12343065
> > > >
> > > > > > > > as
> > > > > > > > >>>>>> a part of Google Summer of Code 2018. I've been
> spending
> > > > some
> > > > > > time
> > > > > > > > to
> > > > > > > > >>>>>> understand the system, architecture and the existing
> > > > Messaging
> > > > > > > > Patterns.
> > > > > > > > >>>>>> I still have a few questions and would like to clarify
> > my
> > > > > > > > assumptions.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> - Is the current FIFO order example limited to one
> > > > message
> > > > > > > queue
> > > > > > > > >>>>>> per producer? Can the producer send the same
> message
> > to
> > > > > > > multiple
> > > > > > > > queues?
> > > > > > > > >>>>>> Will the consumers of the queues be able to read
> > > messages
> > > > > in
> > > > > > > > Order?
> > > > > > > > >>>>>> - Can I assume that each producer will send
> messages
> > to
> > > > one
> > > > > > > > queue?
> > > > > > > > >>>>>> - Global Order is to be identified across all
> > > > > > > > >>>>>> GlobalOrderedProducer (a new producer that is to be
> > > used
> > > > > for
> > > > > > > > global order)
> > > > > > > > >>>>>> instances that are running.
> > > > > > > > >>>>>> - I think using a global clock can help establish
> the
> > > > order
> > > > > > > > >>>>>> between 2 or more producers, however using some
> form
> > of
> > > > > > vector
> > > > > > > > >>>>>> clock might also help identify the global order of
> > > > messages
> > > > > > > > between the
> > > > > > > > >>>>>> producers.
> > > > > > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how
> > to
> > > > read
> > > > > > > > >>>>>> globally ordered messages) can then compare
> messages
> > > > across
> > > > > > all
> > > > > > > > message
> > > > > > > > >>>>>> queues from the corresponding producers and extract
> > the
> > > > > > > > messages. [ this is
> > > > > > > > >>>>>> the approach recommended by yukon on the issue
> page ]
> > > > > > > > >>>>>> - We can also potentially have another layer in the
> > > > Message
> > > > > > > Queue
> > > > > > > > >>>>>> which accumulates all messages sent from producers
> > and
> > > > > > provides
> > > > > > > > one ordered
> > > > > > > > >>>>>> message queue for consumers to read from.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Thank you for your patience and please let me know if
> my
> > > > > > > > >>>>>> understanding of the problem and the assumptions are
> > > right.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Best Regards,
> > > > > > > > >>>>>> Sowmya
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>> --
> > > > > > > > >>>>> Regards,
> > > > > > > > >>>>>
> > > > > > > > >>>>> Sowmya
> > > > > > > > >>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> --
> > > > > > > > >>> Regards,
> > > > > > > > >>>
> > > > > > > > >>> Sowmya
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Regards,
> > > > > > > > >
> > > > > > > > > Sowmya
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Sowmya
> > > > > > > >
> > > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Sowmya
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Sowmya
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Sowmya
> >
>
--
Regards,
Sowmya
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by yukon <yu...@apache.org>.
Hi,
I added some comments, but it seems your google doc link isn't shared:
```
To resolve this issue, we need a global ordered messaging mechanism to have
a good scalability without hot-pot issue.
A practical proposal is that sending messages to distributed
queue/partition with an ordered key. Messages are ordered in the same
queue, then implement an OrderConsumer to pull messages from these queues
then perform merge sort, finally deliver these ordered messages to users.
```
As mentioned in ROCKETMQ-122, we should address the hot-pot issue.
That means the messages can be sent to multiple distributed
queues/partitions and still can be consumed in order manner.
A simple explanation of hot-pot issue:
Assume that all the generated messages in a hot shop of taobao/amazon
should be consumed in order, in the current state, these massive messages
should be sent to a fixed queue/partition, surely to a fixed server which
may cause high pressure in this server. You can easily infer that we can't
resolve this issue through extending the scale of broker cluster.
Could your arch resolve this hot-pot issue?
Regards,
yukon
On Thu, Mar 22, 2018 at 2:15 PM, sowmya s <so...@gmail.com> wrote:
> It's going great.
> In the DefaultMessageStore, there is a class
> CommitLogDispatcherBuildConsumeQueue, I think that a variant of that class
> or another implementation which are merged from the commitlog,
> Since the DispachRequest has a store timestamp, the queue can be populated
> based on that time stamp.
>
> Also, I have some more questions,
> Does the fleet of broker slaves all store the various commit logs that are
> received by them or is the commit log store, replication managed by the
> broker controller?
>
> The other direction I was thinking about is adding/modifying the
> PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
> DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding
> this change there will solve the merge issue for the consumers.
>
> Do you think I am on the right track?
>
> thanks,
> Sowmya
>
>
>
> On Mon, Mar 19, 2018 at 8:15 PM, yukon <yu...@apache.org> wrote:
>
> > Hi,
> >
> > Sorry for the late reply.
> >
> > As for:
> >
> > ```
> > I'm looking at the BrokerController and MessageStore implementation and
> > hooks to understand where the merge logic will best fit.
> > ```
> >
> > how is it going?
> >
> > Regards
> >
> > On Sat, Mar 17, 2018 at 11:30 AM, sowmya s <so...@gmail.com> wrote:
> >
> > > Thank you yukon,
> > >
> > > I'm done with my coursework for this semester and have more time now to
> > > improve my proposal.
> > > I'm looking at the BrokerController and MessageStore implementation and
> > > hooks to understand where the merge logic will best fit. So far I've
> > looked
> > > at the codebase from a Producer and Consumer perspective and looked at
> > the
> > > DefaultMQProducerImpl and DefaultMQPushConsumerImpl for understanding
> > the
> > > link between how Producers send and Consumers receive messages.
> > >
> > > thanks,
> > > Sowmya
> > >
> > > On Fri, Mar 16, 2018 at 8:07 PM, yukon <yu...@apache.org> wrote:
> > >
> > > > Cool, let's focus on it and see whether is there anything can be
> > > polished.
> > > >
> > > > Regards
> > > >
> > > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <so...@gmail.com>
> > wrote:
> > > >
> > > > > Hey yukon,
> > > > > I submitted my draft on the summer of code homepage a couple of
> days
> > > ago,
> > > > > also attaching the link here for reference,
> > > > >
> > > > > https://docs.google.com/file/d/1nXktUO_TF9-
> > rSHSnGj5z5QZoHzMhosxm/edit?
> > > > > usp=docslist_api&filetype=msword
> > > > >
> > > > > Thanks,
> > > > > Sowmya
> > > > >
> > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon <yu...@apache.org> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Of course, we can work together to finetune your design draft.
> > > > > >
> > > > > > Regards,
> > > > > > yukon
> > > > > >
> > > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <so...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hello, yukon and Von,
> > > > > > >
> > > > > > > I've shared my GSOC - 18 draft of the project. I'm looking
> > forward
> > > to
> > > > > > > working with all of you to finetune the proposal. I will be
> > > > allocating
> > > > > 20
> > > > > > > hours per week from now to the proposal acceptance phase to
> > address
> > > > > > > questions and dive deep into any suggestions that you provide.
> > > > > > > I am really looking forward to work on this project.
> > > > > > >
> > > > > > > thanks,
> > > > > > > Sowmya
> > > > > > >
> > > > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <
> sowmya9029@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hello yukon,
> > > > > > > >
> > > > > > > > Thank you for the inputs. I was able to look at the ~/store
> and
> > > > kind
> > > > > of
> > > > > > > > understand the storage structure. I also looked at the
> > > > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in
> > the
> > > > > > > examples.
> > > > > > > > Now I understand why you proposed a merge sort like approach
> > for
> > > > > > > > performing global ordering. Since the proposals are open, I
> am
> > > > > > finalizing
> > > > > > > > my draft and will have it up for review very soon.
> > > > > > > >
> > > > > > > > thanks,
> > > > > > > > Sowmya
> > > > > > > >
> > > > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org>
> > wrote:
> > > > > > > >
> > > > > > > >> Hi Sowmya,
> > > > > > > >>
> > > > > > > >> ```
> > > > > > > >> Also, it would be great if you can at a high level, help me
> > > > > understand
> > > > > > > >> how the messages in the message queues are stored before the
> > > > > consumer
> > > > > > > reads
> > > > > > > >> them.
> > > > > > > >> ```
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> As shown in this figure, messages are sent to brokers by
> > > producer
> > > > > and
> > > > > > > >> stored in commit log[1], then messages are dispatched to
> > > > > ConsumeQueue
> > > > > > by
> > > > > > > >> topic, the consumer pulls messages from the queue.
> > > > > > > >>
> > > > > > > >> I recommend you run a broker and send/consume some messages,
> > > then
> > > > > > check
> > > > > > > >> out the ~/store directory for details.
> > > > > > > >>
> > > > > > > >> Regards,
> > > > > > > >> yukon
> > > > > > > >>
> > > > > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <
> > sowmya9029@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >>
> > > > > > > >>> Hello yukon,
> > > > > > > >>>
> > > > > > > >>> Currently FIFO can be achieved with a producer sending to
> one
> > > > > message
> > > > > > > >>> queue, and when global ordering is required, multiple
> > producers
> > > > > have
> > > > > > to
> > > > > > > >>> send to a single topic queue.
> > > > > > > >>>
> > > > > > > >>> We want to allow multiple producers to send messages on a
> > topic
> > > > to
> > > > > > > >>> multiple message queues and still provide ordering
> guarantees
> > > to
> > > > > the
> > > > > > > >>> consumer, so that all consumers see the same order of data
> > and
> > > > also
> > > > > > the
> > > > > > > >>> data is delivered in an ordered fashion.
> > > > > > > >>>
> > > > > > > >>> 1) Your idea of using a merge sort with the assumption that
> > the
> > > > > first
> > > > > > > >>> arriving message is treated as the first message to be
> > > delivered,
> > > > > > > however,
> > > > > > > >>> I want to propose an approach where when the producer
> sends a
> > > > > message
> > > > > > > to a
> > > > > > > >>> message queue, it must be done in a synchronous fashion and
> > the
> > > > > > > response
> > > > > > > >>> will be that the message is accepted, which means that the
> > > > message
> > > > > > > follows
> > > > > > > >>> the convention that all messages delivered prior by that
> > > producer
> > > > > > have
> > > > > > > been
> > > > > > > >>> stored across groups and if not the producer will need to
> > > resend
> > > > > the
> > > > > > > >>> message.
> > > > > > > >>>
> > > > > > > >>> We can use a variant of total causal ordering in the layer
> > > > between
> > > > > > the
> > > > > > > >>> message queue and store.
> > > > > > > >>>
> > > > > > > >>> I have been busy with my class project so I couldn't make a
> > lot
> > > > of
> > > > > > > >>> progress in detailing my approach. Also, it would be great
> if
> > > you
> > > > > can
> > > > > > > at a
> > > > > > > >>> high level, help me understand how the messages in the
> > message
> > > > > queues
> > > > > > > are
> > > > > > > >>> stored before the consumer reads them.
> > > > > > > >>>
> > > > > > > >>> Does the consumer read directly from the message queue that
> > the
> > > > > > > producer
> > > > > > > >>> sends data to? does the broker receive the queued producer
> > > > > messages,
> > > > > > > store
> > > > > > > >>> them and then pushes them to another queue for the consumer
> > to
> > > > read
> > > > > > > from?
> > > > > > > >>>
> > > > > > > >>> For reference on total causal ordering:
> > > > > > https://www.cl.cam.ac.uk/teach
> > > > > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
> > > > > > > >>>
> > > > > > > >>> thanks,
> > > > > > > >>> Sowmya
> > > > > > > >>>
> > > > > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org>
> > > wrote:
> > > > > > > >>>
> > > > > > > >>>> Hi Sowmya,
> > > > > > > >>>>
> > > > > > > >>>> Sorry for the late reply, do you have any update on this
> > > > project?
> > > > > > > >>>>
> > > > > > > >>>> In RocketMQ, one message queue is a FIFO queue natively,
> so
> > I
> > > > > > proposed
> > > > > > > >>>> a simple solution that performs merge sort on multiple
> > queues
> > > to
> > > > > > > improve
> > > > > > > >>>> performance and scalability.
> > > > > > > >>>>
> > > > > > > >>>> While the order issue across producers is difficult, we
> > could
> > > > > assume
> > > > > > > >>>> that the message first arrives the broker should be
> consumed
> > > > > > firstly.
> > > > > > > >>>>
> > > > > > > >>>> But it would be wonderful if you have a real design about
> > the
> > > > > order
> > > > > > > >>>> issue across producers based on the RocketMQ design and
> the
> > > > > storage
> > > > > > > >>>> structure.
> > > > > > > >>>>
> > > > > > > >>>> Looking forward your design ~
> > > > > > > >>>>
> > > > > > > >>>> Regards,
> > > > > > > >>>> yukon
> > > > > > > >>>>
> > > > > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <
> > > sowmya9029@gmail.com>
> > > > > > > wrote:
> > > > > > > >>>>
> > > > > > > >>>>> Hello all,
> > > > > > > >>>>>
> > > > > > > >>>>> Adding a few more thoughts on the problem of establishing
> > an
> > > > > order
> > > > > > of
> > > > > > > >>>>> messages across producers.
> > > > > > > >>>>>
> > > > > > > >>>>> Consider the Scenario
> > > > > > > >>>>>
> > > > > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue
> > > Queue1
> > > > > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue
> > Queue2.
> > > > > > > >>>>>
> > > > > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and
> > > similarly
> > > > > > > >>>>> time(a2) < time(b2) < time(c2)
> > > > > > > >>>>>
> > > > > > > >>>>> Are the following orders acceptable to the consumer?
> > > > > > > >>>>>
> > > > > > > >>>>> > a1, a2, b1, b2, c1, c2
> > > > > > > >>>>> > a1, b1, c1, a2, b2, c2
> > > > > > > >>>>> > a1 b1, b2, a2, a3, b3
> > > > > > > >>>>>
> > > > > > > >>>>> and an order displayed at one consumer is consistent
> across
> > > all
> > > > > > > >>>>> consumers.
> > > > > > > >>>>>
> > > > > > > >>>>> This can be achieved using Total Causal Ordering at the
> > > > Producer
> > > > > or
> > > > > > > >>>>> Queue level, using Leslie Lamport's clock and
> > synchronization
> > > > > > > approach.
> > > > > > > >>>>>
> > > > > > > >>>>> For reference is the paper attached,
> > > > > > http://lamport.azurewebsites.n
> > > > > > > >>>>> et/pubs/time-clocks.pdf
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > > >>>>> --
> > > > > > > >>>>> Regards,
> > > > > > > >>>>>
> > > > > > > >>>>> Sowmya
> > > > > > > >>>>>
> > > > > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <
> > > > sowmya9029@gmail.com>
> > > > > > > >>>>> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>>> Hello,
> > > > > > > >>>>>>
> > > > > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122
> > > > > > > >>>>>> <
> > > > > > https://issues.apache.org/jira/browse/ROCKETMQ-122?
> filter=12343065
> > >
> > > > > > > as
> > > > > > > >>>>>> a part of Google Summer of Code 2018. I've been spending
> > > some
> > > > > time
> > > > > > > to
> > > > > > > >>>>>> understand the system, architecture and the existing
> > > Messaging
> > > > > > > Patterns.
> > > > > > > >>>>>> I still have a few questions and would like to clarify
> my
> > > > > > > assumptions.
> > > > > > > >>>>>>
> > > > > > > >>>>>> - Is the current FIFO order example limited to one
> > > message
> > > > > > queue
> > > > > > > >>>>>> per producer? Can the producer send the same message
> to
> > > > > > multiple
> > > > > > > queues?
> > > > > > > >>>>>> Will the consumers of the queues be able to read
> > messages
> > > > in
> > > > > > > Order?
> > > > > > > >>>>>> - Can I assume that each producer will send messages
> to
> > > one
> > > > > > > queue?
> > > > > > > >>>>>> - Global Order is to be identified across all
> > > > > > > >>>>>> GlobalOrderedProducer (a new producer that is to be
> > used
> > > > for
> > > > > > > global order)
> > > > > > > >>>>>> instances that are running.
> > > > > > > >>>>>> - I think using a global clock can help establish the
> > > order
> > > > > > > >>>>>> between 2 or more producers, however using some form
> of
> > > > > vector
> > > > > > > >>>>>> clock might also help identify the global order of
> > > messages
> > > > > > > between the
> > > > > > > >>>>>> producers.
> > > > > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how
> to
> > > read
> > > > > > > >>>>>> globally ordered messages) can then compare messages
> > > across
> > > > > all
> > > > > > > message
> > > > > > > >>>>>> queues from the corresponding producers and extract
> the
> > > > > > > messages. [ this is
> > > > > > > >>>>>> the approach recommended by yukon on the issue page ]
> > > > > > > >>>>>> - We can also potentially have another layer in the
> > > Message
> > > > > > Queue
> > > > > > > >>>>>> which accumulates all messages sent from producers
> and
> > > > > provides
> > > > > > > one ordered
> > > > > > > >>>>>> message queue for consumers to read from.
> > > > > > > >>>>>>
> > > > > > > >>>>>> Thank you for your patience and please let me know if my
> > > > > > > >>>>>> understanding of the problem and the assumptions are
> > right.
> > > > > > > >>>>>>
> > > > > > > >>>>>> Best Regards,
> > > > > > > >>>>>> Sowmya
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > > >>>>> --
> > > > > > > >>>>> Regards,
> > > > > > > >>>>>
> > > > > > > >>>>> Sowmya
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> --
> > > > > > > >>> Regards,
> > > > > > > >>>
> > > > > > > >>> Sowmya
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Sowmya
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > >
> > > > > > > Sowmya
> > > > > > >
> > > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Sowmya
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Sowmya
> > >
> >
>
>
>
> --
> Regards,
>
> Sowmya
>
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by sowmya s <so...@gmail.com>.
It's going great.
In the DefaultMessageStore, there is a class
CommitLogDispatcherBuildConsumeQueue, I think that a variant of that class
or another implementation which are merged from the commitlog,
Since the DispachRequest has a store timestamp, the queue can be populated
based on that time stamp.
Also, I have some more questions,
Does the fleet of broker slaves all store the various commit logs that are
received by them or is the commit log store, replication managed by the
broker controller?
The other direction I was thinking about is adding/modifying the
PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding
this change there will solve the merge issue for the consumers.
Do you think I am on the right track?
thanks,
Sowmya
On Mon, Mar 19, 2018 at 8:15 PM, yukon <yu...@apache.org> wrote:
> Hi,
>
> Sorry for the late reply.
>
> As for:
>
> ```
> I'm looking at the BrokerController and MessageStore implementation and
> hooks to understand where the merge logic will best fit.
> ```
>
> how is it going?
>
> Regards
>
> On Sat, Mar 17, 2018 at 11:30 AM, sowmya s <so...@gmail.com> wrote:
>
> > Thank you yukon,
> >
> > I'm done with my coursework for this semester and have more time now to
> > improve my proposal.
> > I'm looking at the BrokerController and MessageStore implementation and
> > hooks to understand where the merge logic will best fit. So far I've
> looked
> > at the codebase from a Producer and Consumer perspective and looked at
> the
> > DefaultMQProducerImpl and DefaultMQPushConsumerImpl for understanding
> the
> > link between how Producers send and Consumers receive messages.
> >
> > thanks,
> > Sowmya
> >
> > On Fri, Mar 16, 2018 at 8:07 PM, yukon <yu...@apache.org> wrote:
> >
> > > Cool, let's focus on it and see whether is there anything can be
> > polished.
> > >
> > > Regards
> > >
> > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <so...@gmail.com>
> wrote:
> > >
> > > > Hey yukon,
> > > > I submitted my draft on the summer of code homepage a couple of days
> > ago,
> > > > also attaching the link here for reference,
> > > >
> > > > https://docs.google.com/file/d/1nXktUO_TF9-
> rSHSnGj5z5QZoHzMhosxm/edit?
> > > > usp=docslist_api&filetype=msword
> > > >
> > > > Thanks,
> > > > Sowmya
> > > >
> > > > On Thu, Mar 15, 2018 at 2:08 AM yukon <yu...@apache.org> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Of course, we can work together to finetune your design draft.
> > > > >
> > > > > Regards,
> > > > > yukon
> > > > >
> > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <so...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hello, yukon and Von,
> > > > > >
> > > > > > I've shared my GSOC - 18 draft of the project. I'm looking
> forward
> > to
> > > > > > working with all of you to finetune the proposal. I will be
> > > allocating
> > > > 20
> > > > > > hours per week from now to the proposal acceptance phase to
> address
> > > > > > questions and dive deep into any suggestions that you provide.
> > > > > > I am really looking forward to work on this project.
> > > > > >
> > > > > > thanks,
> > > > > > Sowmya
> > > > > >
> > > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <sowmya9029@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Hello yukon,
> > > > > > >
> > > > > > > Thank you for the inputs. I was able to look at the ~/store and
> > > kind
> > > > of
> > > > > > > understand the storage structure. I also looked at the
> > > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in
> the
> > > > > > examples.
> > > > > > > Now I understand why you proposed a merge sort like approach
> for
> > > > > > > performing global ordering. Since the proposals are open, I am
> > > > > finalizing
> > > > > > > my draft and will have it up for review very soon.
> > > > > > >
> > > > > > > thanks,
> > > > > > > Sowmya
> > > > > > >
> > > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org>
> wrote:
> > > > > > >
> > > > > > >> Hi Sowmya,
> > > > > > >>
> > > > > > >> ```
> > > > > > >> Also, it would be great if you can at a high level, help me
> > > > understand
> > > > > > >> how the messages in the message queues are stored before the
> > > > consumer
> > > > > > reads
> > > > > > >> them.
> > > > > > >> ```
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> As shown in this figure, messages are sent to brokers by
> > producer
> > > > and
> > > > > > >> stored in commit log[1], then messages are dispatched to
> > > > ConsumeQueue
> > > > > by
> > > > > > >> topic, the consumer pulls messages from the queue.
> > > > > > >>
> > > > > > >> I recommend you run a broker and send/consume some messages,
> > then
> > > > > check
> > > > > > >> out the ~/store directory for details.
> > > > > > >>
> > > > > > >> Regards,
> > > > > > >> yukon
> > > > > > >>
> > > > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <
> sowmya9029@gmail.com
> > >
> > > > > wrote:
> > > > > > >>
> > > > > > >>> Hello yukon,
> > > > > > >>>
> > > > > > >>> Currently FIFO can be achieved with a producer sending to one
> > > > message
> > > > > > >>> queue, and when global ordering is required, multiple
> producers
> > > > have
> > > > > to
> > > > > > >>> send to a single topic queue.
> > > > > > >>>
> > > > > > >>> We want to allow multiple producers to send messages on a
> topic
> > > to
> > > > > > >>> multiple message queues and still provide ordering guarantees
> > to
> > > > the
> > > > > > >>> consumer, so that all consumers see the same order of data
> and
> > > also
> > > > > the
> > > > > > >>> data is delivered in an ordered fashion.
> > > > > > >>>
> > > > > > >>> 1) Your idea of using a merge sort with the assumption that
> the
> > > > first
> > > > > > >>> arriving message is treated as the first message to be
> > delivered,
> > > > > > however,
> > > > > > >>> I want to propose an approach where when the producer sends a
> > > > message
> > > > > > to a
> > > > > > >>> message queue, it must be done in a synchronous fashion and
> the
> > > > > > response
> > > > > > >>> will be that the message is accepted, which means that the
> > > message
> > > > > > follows
> > > > > > >>> the convention that all messages delivered prior by that
> > producer
> > > > > have
> > > > > > been
> > > > > > >>> stored across groups and if not the producer will need to
> > resend
> > > > the
> > > > > > >>> message.
> > > > > > >>>
> > > > > > >>> We can use a variant of total causal ordering in the layer
> > > between
> > > > > the
> > > > > > >>> message queue and store.
> > > > > > >>>
> > > > > > >>> I have been busy with my class project so I couldn't make a
> lot
> > > of
> > > > > > >>> progress in detailing my approach. Also, it would be great if
> > you
> > > > can
> > > > > > at a
> > > > > > >>> high level, help me understand how the messages in the
> message
> > > > queues
> > > > > > are
> > > > > > >>> stored before the consumer reads them.
> > > > > > >>>
> > > > > > >>> Does the consumer read directly from the message queue that
> the
> > > > > > producer
> > > > > > >>> sends data to? does the broker receive the queued producer
> > > > messages,
> > > > > > store
> > > > > > >>> them and then pushes them to another queue for the consumer
> to
> > > read
> > > > > > from?
> > > > > > >>>
> > > > > > >>> For reference on total causal ordering:
> > > > > https://www.cl.cam.ac.uk/teach
> > > > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
> > > > > > >>>
> > > > > > >>> thanks,
> > > > > > >>> Sowmya
> > > > > > >>>
> > > > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org>
> > wrote:
> > > > > > >>>
> > > > > > >>>> Hi Sowmya,
> > > > > > >>>>
> > > > > > >>>> Sorry for the late reply, do you have any update on this
> > > project?
> > > > > > >>>>
> > > > > > >>>> In RocketMQ, one message queue is a FIFO queue natively, so
> I
> > > > > proposed
> > > > > > >>>> a simple solution that performs merge sort on multiple
> queues
> > to
> > > > > > improve
> > > > > > >>>> performance and scalability.
> > > > > > >>>>
> > > > > > >>>> While the order issue across producers is difficult, we
> could
> > > > assume
> > > > > > >>>> that the message first arrives the broker should be consumed
> > > > > firstly.
> > > > > > >>>>
> > > > > > >>>> But it would be wonderful if you have a real design about
> the
> > > > order
> > > > > > >>>> issue across producers based on the RocketMQ design and the
> > > > storage
> > > > > > >>>> structure.
> > > > > > >>>>
> > > > > > >>>> Looking forward your design ~
> > > > > > >>>>
> > > > > > >>>> Regards,
> > > > > > >>>> yukon
> > > > > > >>>>
> > > > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <
> > sowmya9029@gmail.com>
> > > > > > wrote:
> > > > > > >>>>
> > > > > > >>>>> Hello all,
> > > > > > >>>>>
> > > > > > >>>>> Adding a few more thoughts on the problem of establishing
> an
> > > > order
> > > > > of
> > > > > > >>>>> messages across producers.
> > > > > > >>>>>
> > > > > > >>>>> Consider the Scenario
> > > > > > >>>>>
> > > > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue
> > Queue1
> > > > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue
> Queue2.
> > > > > > >>>>>
> > > > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and
> > similarly
> > > > > > >>>>> time(a2) < time(b2) < time(c2)
> > > > > > >>>>>
> > > > > > >>>>> Are the following orders acceptable to the consumer?
> > > > > > >>>>>
> > > > > > >>>>> > a1, a2, b1, b2, c1, c2
> > > > > > >>>>> > a1, b1, c1, a2, b2, c2
> > > > > > >>>>> > a1 b1, b2, a2, a3, b3
> > > > > > >>>>>
> > > > > > >>>>> and an order displayed at one consumer is consistent across
> > all
> > > > > > >>>>> consumers.
> > > > > > >>>>>
> > > > > > >>>>> This can be achieved using Total Causal Ordering at the
> > > Producer
> > > > or
> > > > > > >>>>> Queue level, using Leslie Lamport's clock and
> synchronization
> > > > > > approach.
> > > > > > >>>>>
> > > > > > >>>>> For reference is the paper attached,
> > > > > http://lamport.azurewebsites.n
> > > > > > >>>>> et/pubs/time-clocks.pdf
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>> --
> > > > > > >>>>> Regards,
> > > > > > >>>>>
> > > > > > >>>>> Sowmya
> > > > > > >>>>>
> > > > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <
> > > sowmya9029@gmail.com>
> > > > > > >>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>>> Hello,
> > > > > > >>>>>>
> > > > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122
> > > > > > >>>>>> <
> > > > > https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065
> >
> > > > > > as
> > > > > > >>>>>> a part of Google Summer of Code 2018. I've been spending
> > some
> > > > time
> > > > > > to
> > > > > > >>>>>> understand the system, architecture and the existing
> > Messaging
> > > > > > Patterns.
> > > > > > >>>>>> I still have a few questions and would like to clarify my
> > > > > > assumptions.
> > > > > > >>>>>>
> > > > > > >>>>>> - Is the current FIFO order example limited to one
> > message
> > > > > queue
> > > > > > >>>>>> per producer? Can the producer send the same message to
> > > > > multiple
> > > > > > queues?
> > > > > > >>>>>> Will the consumers of the queues be able to read
> messages
> > > in
> > > > > > Order?
> > > > > > >>>>>> - Can I assume that each producer will send messages to
> > one
> > > > > > queue?
> > > > > > >>>>>> - Global Order is to be identified across all
> > > > > > >>>>>> GlobalOrderedProducer (a new producer that is to be
> used
> > > for
> > > > > > global order)
> > > > > > >>>>>> instances that are running.
> > > > > > >>>>>> - I think using a global clock can help establish the
> > order
> > > > > > >>>>>> between 2 or more producers, however using some form of
> > > > vector
> > > > > > >>>>>> clock might also help identify the global order of
> > messages
> > > > > > between the
> > > > > > >>>>>> producers.
> > > > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how to
> > read
> > > > > > >>>>>> globally ordered messages) can then compare messages
> > across
> > > > all
> > > > > > message
> > > > > > >>>>>> queues from the corresponding producers and extract the
> > > > > > messages. [ this is
> > > > > > >>>>>> the approach recommended by yukon on the issue page ]
> > > > > > >>>>>> - We can also potentially have another layer in the
> > Message
> > > > > Queue
> > > > > > >>>>>> which accumulates all messages sent from producers and
> > > > provides
> > > > > > one ordered
> > > > > > >>>>>> message queue for consumers to read from.
> > > > > > >>>>>>
> > > > > > >>>>>> Thank you for your patience and please let me know if my
> > > > > > >>>>>> understanding of the problem and the assumptions are
> right.
> > > > > > >>>>>>
> > > > > > >>>>>> Best Regards,
> > > > > > >>>>>> Sowmya
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>> --
> > > > > > >>>>> Regards,
> > > > > > >>>>>
> > > > > > >>>>> Sowmya
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> --
> > > > > > >>> Regards,
> > > > > > >>>
> > > > > > >>> Sowmya
> > > > > > >>>
> > > > > > >>>
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > >
> > > > > > > Sowmya
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Sowmya
> > > > > >
> > > > >
> > > > --
> > > > Regards,
> > > >
> > > > Sowmya
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Sowmya
> >
>
--
Regards,
Sowmya
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by yukon <yu...@apache.org>.
Hi,
Sorry for the late reply.
As for:
```
I'm looking at the BrokerController and MessageStore implementation and
hooks to understand where the merge logic will best fit.
```
how is it going?
Regards
On Sat, Mar 17, 2018 at 11:30 AM, sowmya s <so...@gmail.com> wrote:
> Thank you yukon,
>
> I'm done with my coursework for this semester and have more time now to
> improve my proposal.
> I'm looking at the BrokerController and MessageStore implementation and
> hooks to understand where the merge logic will best fit. So far I've looked
> at the codebase from a Producer and Consumer perspective and looked at the
> DefaultMQProducerImpl and DefaultMQPushConsumerImpl for understanding the
> link between how Producers send and Consumers receive messages.
>
> thanks,
> Sowmya
>
> On Fri, Mar 16, 2018 at 8:07 PM, yukon <yu...@apache.org> wrote:
>
> > Cool, let's focus on it and see whether is there anything can be
> polished.
> >
> > Regards
> >
> > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <so...@gmail.com> wrote:
> >
> > > Hey yukon,
> > > I submitted my draft on the summer of code homepage a couple of days
> ago,
> > > also attaching the link here for reference,
> > >
> > > https://docs.google.com/file/d/1nXktUO_TF9-rSHSnGj5z5QZoHzMhosxm/edit?
> > > usp=docslist_api&filetype=msword
> > >
> > > Thanks,
> > > Sowmya
> > >
> > > On Thu, Mar 15, 2018 at 2:08 AM yukon <yu...@apache.org> wrote:
> > >
> > > > Hi,
> > > >
> > > > Of course, we can work together to finetune your design draft.
> > > >
> > > > Regards,
> > > > yukon
> > > >
> > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <so...@gmail.com>
> > wrote:
> > > >
> > > > > Hello, yukon and Von,
> > > > >
> > > > > I've shared my GSOC - 18 draft of the project. I'm looking forward
> to
> > > > > working with all of you to finetune the proposal. I will be
> > allocating
> > > 20
> > > > > hours per week from now to the proposal acceptance phase to address
> > > > > questions and dive deep into any suggestions that you provide.
> > > > > I am really looking forward to work on this project.
> > > > >
> > > > > thanks,
> > > > > Sowmya
> > > > >
> > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <so...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hello yukon,
> > > > > >
> > > > > > Thank you for the inputs. I was able to look at the ~/store and
> > kind
> > > of
> > > > > > understand the storage structure. I also looked at the
> > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the
> > > > > examples.
> > > > > > Now I understand why you proposed a merge sort like approach for
> > > > > > performing global ordering. Since the proposals are open, I am
> > > > finalizing
> > > > > > my draft and will have it up for review very soon.
> > > > > >
> > > > > > thanks,
> > > > > > Sowmya
> > > > > >
> > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org> wrote:
> > > > > >
> > > > > >> Hi Sowmya,
> > > > > >>
> > > > > >> ```
> > > > > >> Also, it would be great if you can at a high level, help me
> > > understand
> > > > > >> how the messages in the message queues are stored before the
> > > consumer
> > > > > reads
> > > > > >> them.
> > > > > >> ```
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> As shown in this figure, messages are sent to brokers by
> producer
> > > and
> > > > > >> stored in commit log[1], then messages are dispatched to
> > > ConsumeQueue
> > > > by
> > > > > >> topic, the consumer pulls messages from the queue.
> > > > > >>
> > > > > >> I recommend you run a broker and send/consume some messages,
> then
> > > > check
> > > > > >> out the ~/store directory for details.
> > > > > >>
> > > > > >> Regards,
> > > > > >> yukon
> > > > > >>
> > > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <sowmya9029@gmail.com
> >
> > > > wrote:
> > > > > >>
> > > > > >>> Hello yukon,
> > > > > >>>
> > > > > >>> Currently FIFO can be achieved with a producer sending to one
> > > message
> > > > > >>> queue, and when global ordering is required, multiple producers
> > > have
> > > > to
> > > > > >>> send to a single topic queue.
> > > > > >>>
> > > > > >>> We want to allow multiple producers to send messages on a topic
> > to
> > > > > >>> multiple message queues and still provide ordering guarantees
> to
> > > the
> > > > > >>> consumer, so that all consumers see the same order of data and
> > also
> > > > the
> > > > > >>> data is delivered in an ordered fashion.
> > > > > >>>
> > > > > >>> 1) Your idea of using a merge sort with the assumption that the
> > > first
> > > > > >>> arriving message is treated as the first message to be
> delivered,
> > > > > however,
> > > > > >>> I want to propose an approach where when the producer sends a
> > > message
> > > > > to a
> > > > > >>> message queue, it must be done in a synchronous fashion and the
> > > > > response
> > > > > >>> will be that the message is accepted, which means that the
> > message
> > > > > follows
> > > > > >>> the convention that all messages delivered prior by that
> producer
> > > > have
> > > > > been
> > > > > >>> stored across groups and if not the producer will need to
> resend
> > > the
> > > > > >>> message.
> > > > > >>>
> > > > > >>> We can use a variant of total causal ordering in the layer
> > between
> > > > the
> > > > > >>> message queue and store.
> > > > > >>>
> > > > > >>> I have been busy with my class project so I couldn't make a lot
> > of
> > > > > >>> progress in detailing my approach. Also, it would be great if
> you
> > > can
> > > > > at a
> > > > > >>> high level, help me understand how the messages in the message
> > > queues
> > > > > are
> > > > > >>> stored before the consumer reads them.
> > > > > >>>
> > > > > >>> Does the consumer read directly from the message queue that the
> > > > > producer
> > > > > >>> sends data to? does the broker receive the queued producer
> > > messages,
> > > > > store
> > > > > >>> them and then pushes them to another queue for the consumer to
> > read
> > > > > from?
> > > > > >>>
> > > > > >>> For reference on total causal ordering:
> > > > https://www.cl.cam.ac.uk/teach
> > > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
> > > > > >>>
> > > > > >>> thanks,
> > > > > >>> Sowmya
> > > > > >>>
> > > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org>
> wrote:
> > > > > >>>
> > > > > >>>> Hi Sowmya,
> > > > > >>>>
> > > > > >>>> Sorry for the late reply, do you have any update on this
> > project?
> > > > > >>>>
> > > > > >>>> In RocketMQ, one message queue is a FIFO queue natively, so I
> > > > proposed
> > > > > >>>> a simple solution that performs merge sort on multiple queues
> to
> > > > > improve
> > > > > >>>> performance and scalability.
> > > > > >>>>
> > > > > >>>> While the order issue across producers is difficult, we could
> > > assume
> > > > > >>>> that the message first arrives the broker should be consumed
> > > > firstly.
> > > > > >>>>
> > > > > >>>> But it would be wonderful if you have a real design about the
> > > order
> > > > > >>>> issue across producers based on the RocketMQ design and the
> > > storage
> > > > > >>>> structure.
> > > > > >>>>
> > > > > >>>> Looking forward your design ~
> > > > > >>>>
> > > > > >>>> Regards,
> > > > > >>>> yukon
> > > > > >>>>
> > > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <
> sowmya9029@gmail.com>
> > > > > wrote:
> > > > > >>>>
> > > > > >>>>> Hello all,
> > > > > >>>>>
> > > > > >>>>> Adding a few more thoughts on the problem of establishing an
> > > order
> > > > of
> > > > > >>>>> messages across producers.
> > > > > >>>>>
> > > > > >>>>> Consider the Scenario
> > > > > >>>>>
> > > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue
> Queue1
> > > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2.
> > > > > >>>>>
> > > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and
> similarly
> > > > > >>>>> time(a2) < time(b2) < time(c2)
> > > > > >>>>>
> > > > > >>>>> Are the following orders acceptable to the consumer?
> > > > > >>>>>
> > > > > >>>>> > a1, a2, b1, b2, c1, c2
> > > > > >>>>> > a1, b1, c1, a2, b2, c2
> > > > > >>>>> > a1 b1, b2, a2, a3, b3
> > > > > >>>>>
> > > > > >>>>> and an order displayed at one consumer is consistent across
> all
> > > > > >>>>> consumers.
> > > > > >>>>>
> > > > > >>>>> This can be achieved using Total Causal Ordering at the
> > Producer
> > > or
> > > > > >>>>> Queue level, using Leslie Lamport's clock and synchronization
> > > > > approach.
> > > > > >>>>>
> > > > > >>>>> For reference is the paper attached,
> > > > http://lamport.azurewebsites.n
> > > > > >>>>> et/pubs/time-clocks.pdf
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> --
> > > > > >>>>> Regards,
> > > > > >>>>>
> > > > > >>>>> Sowmya
> > > > > >>>>>
> > > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <
> > sowmya9029@gmail.com>
> > > > > >>>>> wrote:
> > > > > >>>>>
> > > > > >>>>>> Hello,
> > > > > >>>>>>
> > > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122
> > > > > >>>>>> <
> > > > https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065>
> > > > > as
> > > > > >>>>>> a part of Google Summer of Code 2018. I've been spending
> some
> > > time
> > > > > to
> > > > > >>>>>> understand the system, architecture and the existing
> Messaging
> > > > > Patterns.
> > > > > >>>>>> I still have a few questions and would like to clarify my
> > > > > assumptions.
> > > > > >>>>>>
> > > > > >>>>>> - Is the current FIFO order example limited to one
> message
> > > > queue
> > > > > >>>>>> per producer? Can the producer send the same message to
> > > > multiple
> > > > > queues?
> > > > > >>>>>> Will the consumers of the queues be able to read messages
> > in
> > > > > Order?
> > > > > >>>>>> - Can I assume that each producer will send messages to
> one
> > > > > queue?
> > > > > >>>>>> - Global Order is to be identified across all
> > > > > >>>>>> GlobalOrderedProducer (a new producer that is to be used
> > for
> > > > > global order)
> > > > > >>>>>> instances that are running.
> > > > > >>>>>> - I think using a global clock can help establish the
> order
> > > > > >>>>>> between 2 or more producers, however using some form of
> > > vector
> > > > > >>>>>> clock might also help identify the global order of
> messages
> > > > > between the
> > > > > >>>>>> producers.
> > > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how to
> read
> > > > > >>>>>> globally ordered messages) can then compare messages
> across
> > > all
> > > > > message
> > > > > >>>>>> queues from the corresponding producers and extract the
> > > > > messages. [ this is
> > > > > >>>>>> the approach recommended by yukon on the issue page ]
> > > > > >>>>>> - We can also potentially have another layer in the
> Message
> > > > Queue
> > > > > >>>>>> which accumulates all messages sent from producers and
> > > provides
> > > > > one ordered
> > > > > >>>>>> message queue for consumers to read from.
> > > > > >>>>>>
> > > > > >>>>>> Thank you for your patience and please let me know if my
> > > > > >>>>>> understanding of the problem and the assumptions are right.
> > > > > >>>>>>
> > > > > >>>>>> Best Regards,
> > > > > >>>>>> Sowmya
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> --
> > > > > >>>>> Regards,
> > > > > >>>>>
> > > > > >>>>> Sowmya
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>>
> > > > > >>> --
> > > > > >>> Regards,
> > > > > >>>
> > > > > >>> Sowmya
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Sowmya
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Sowmya
> > > > >
> > > >
> > > --
> > > Regards,
> > >
> > > Sowmya
> > >
> >
>
>
>
> --
> Regards,
>
> Sowmya
>
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by sowmya s <so...@gmail.com>.
Thank you yukon,
I'm done with my coursework for this semester and have more time now to
improve my proposal.
I'm looking at the BrokerController and MessageStore implementation and
hooks to understand where the merge logic will best fit. So far I've looked
at the codebase from a Producer and Consumer perspective and looked at the
DefaultMQProducerImpl and DefaultMQPushConsumerImpl for understanding the
link between how Producers send and Consumers receive messages.
thanks,
Sowmya
On Fri, Mar 16, 2018 at 8:07 PM, yukon <yu...@apache.org> wrote:
> Cool, let's focus on it and see whether is there anything can be polished.
>
> Regards
>
> On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <so...@gmail.com> wrote:
>
> > Hey yukon,
> > I submitted my draft on the summer of code homepage a couple of days ago,
> > also attaching the link here for reference,
> >
> > https://docs.google.com/file/d/1nXktUO_TF9-rSHSnGj5z5QZoHzMhosxm/edit?
> > usp=docslist_api&filetype=msword
> >
> > Thanks,
> > Sowmya
> >
> > On Thu, Mar 15, 2018 at 2:08 AM yukon <yu...@apache.org> wrote:
> >
> > > Hi,
> > >
> > > Of course, we can work together to finetune your design draft.
> > >
> > > Regards,
> > > yukon
> > >
> > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <so...@gmail.com>
> wrote:
> > >
> > > > Hello, yukon and Von,
> > > >
> > > > I've shared my GSOC - 18 draft of the project. I'm looking forward to
> > > > working with all of you to finetune the proposal. I will be
> allocating
> > 20
> > > > hours per week from now to the proposal acceptance phase to address
> > > > questions and dive deep into any suggestions that you provide.
> > > > I am really looking forward to work on this project.
> > > >
> > > > thanks,
> > > > Sowmya
> > > >
> > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <so...@gmail.com>
> > wrote:
> > > >
> > > > > Hello yukon,
> > > > >
> > > > > Thank you for the inputs. I was able to look at the ~/store and
> kind
> > of
> > > > > understand the storage structure. I also looked at the
> > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the
> > > > examples.
> > > > > Now I understand why you proposed a merge sort like approach for
> > > > > performing global ordering. Since the proposals are open, I am
> > > finalizing
> > > > > my draft and will have it up for review very soon.
> > > > >
> > > > > thanks,
> > > > > Sowmya
> > > > >
> > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org> wrote:
> > > > >
> > > > >> Hi Sowmya,
> > > > >>
> > > > >> ```
> > > > >> Also, it would be great if you can at a high level, help me
> > understand
> > > > >> how the messages in the message queues are stored before the
> > consumer
> > > > reads
> > > > >> them.
> > > > >> ```
> > > > >>
> > > > >>
> > > > >>
> > > > >> As shown in this figure, messages are sent to brokers by producer
> > and
> > > > >> stored in commit log[1], then messages are dispatched to
> > ConsumeQueue
> > > by
> > > > >> topic, the consumer pulls messages from the queue.
> > > > >>
> > > > >> I recommend you run a broker and send/consume some messages, then
> > > check
> > > > >> out the ~/store directory for details.
> > > > >>
> > > > >> Regards,
> > > > >> yukon
> > > > >>
> > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <so...@gmail.com>
> > > wrote:
> > > > >>
> > > > >>> Hello yukon,
> > > > >>>
> > > > >>> Currently FIFO can be achieved with a producer sending to one
> > message
> > > > >>> queue, and when global ordering is required, multiple producers
> > have
> > > to
> > > > >>> send to a single topic queue.
> > > > >>>
> > > > >>> We want to allow multiple producers to send messages on a topic
> to
> > > > >>> multiple message queues and still provide ordering guarantees to
> > the
> > > > >>> consumer, so that all consumers see the same order of data and
> also
> > > the
> > > > >>> data is delivered in an ordered fashion.
> > > > >>>
> > > > >>> 1) Your idea of using a merge sort with the assumption that the
> > first
> > > > >>> arriving message is treated as the first message to be delivered,
> > > > however,
> > > > >>> I want to propose an approach where when the producer sends a
> > message
> > > > to a
> > > > >>> message queue, it must be done in a synchronous fashion and the
> > > > response
> > > > >>> will be that the message is accepted, which means that the
> message
> > > > follows
> > > > >>> the convention that all messages delivered prior by that producer
> > > have
> > > > been
> > > > >>> stored across groups and if not the producer will need to resend
> > the
> > > > >>> message.
> > > > >>>
> > > > >>> We can use a variant of total causal ordering in the layer
> between
> > > the
> > > > >>> message queue and store.
> > > > >>>
> > > > >>> I have been busy with my class project so I couldn't make a lot
> of
> > > > >>> progress in detailing my approach. Also, it would be great if you
> > can
> > > > at a
> > > > >>> high level, help me understand how the messages in the message
> > queues
> > > > are
> > > > >>> stored before the consumer reads them.
> > > > >>>
> > > > >>> Does the consumer read directly from the message queue that the
> > > > producer
> > > > >>> sends data to? does the broker receive the queued producer
> > messages,
> > > > store
> > > > >>> them and then pushes them to another queue for the consumer to
> read
> > > > from?
> > > > >>>
> > > > >>> For reference on total causal ordering:
> > > https://www.cl.cam.ac.uk/teach
> > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
> > > > >>>
> > > > >>> thanks,
> > > > >>> Sowmya
> > > > >>>
> > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org> wrote:
> > > > >>>
> > > > >>>> Hi Sowmya,
> > > > >>>>
> > > > >>>> Sorry for the late reply, do you have any update on this
> project?
> > > > >>>>
> > > > >>>> In RocketMQ, one message queue is a FIFO queue natively, so I
> > > proposed
> > > > >>>> a simple solution that performs merge sort on multiple queues to
> > > > improve
> > > > >>>> performance and scalability.
> > > > >>>>
> > > > >>>> While the order issue across producers is difficult, we could
> > assume
> > > > >>>> that the message first arrives the broker should be consumed
> > > firstly.
> > > > >>>>
> > > > >>>> But it would be wonderful if you have a real design about the
> > order
> > > > >>>> issue across producers based on the RocketMQ design and the
> > storage
> > > > >>>> structure.
> > > > >>>>
> > > > >>>> Looking forward your design ~
> > > > >>>>
> > > > >>>> Regards,
> > > > >>>> yukon
> > > > >>>>
> > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <so...@gmail.com>
> > > > wrote:
> > > > >>>>
> > > > >>>>> Hello all,
> > > > >>>>>
> > > > >>>>> Adding a few more thoughts on the problem of establishing an
> > order
> > > of
> > > > >>>>> messages across producers.
> > > > >>>>>
> > > > >>>>> Consider the Scenario
> > > > >>>>>
> > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1
> > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2.
> > > > >>>>>
> > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and similarly
> > > > >>>>> time(a2) < time(b2) < time(c2)
> > > > >>>>>
> > > > >>>>> Are the following orders acceptable to the consumer?
> > > > >>>>>
> > > > >>>>> > a1, a2, b1, b2, c1, c2
> > > > >>>>> > a1, b1, c1, a2, b2, c2
> > > > >>>>> > a1 b1, b2, a2, a3, b3
> > > > >>>>>
> > > > >>>>> and an order displayed at one consumer is consistent across all
> > > > >>>>> consumers.
> > > > >>>>>
> > > > >>>>> This can be achieved using Total Causal Ordering at the
> Producer
> > or
> > > > >>>>> Queue level, using Leslie Lamport's clock and synchronization
> > > > approach.
> > > > >>>>>
> > > > >>>>> For reference is the paper attached,
> > > http://lamport.azurewebsites.n
> > > > >>>>> et/pubs/time-clocks.pdf
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> --
> > > > >>>>> Regards,
> > > > >>>>>
> > > > >>>>> Sowmya
> > > > >>>>>
> > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <
> sowmya9029@gmail.com>
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> Hello,
> > > > >>>>>>
> > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122
> > > > >>>>>> <
> > > https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065>
> > > > as
> > > > >>>>>> a part of Google Summer of Code 2018. I've been spending some
> > time
> > > > to
> > > > >>>>>> understand the system, architecture and the existing Messaging
> > > > Patterns.
> > > > >>>>>> I still have a few questions and would like to clarify my
> > > > assumptions.
> > > > >>>>>>
> > > > >>>>>> - Is the current FIFO order example limited to one message
> > > queue
> > > > >>>>>> per producer? Can the producer send the same message to
> > > multiple
> > > > queues?
> > > > >>>>>> Will the consumers of the queues be able to read messages
> in
> > > > Order?
> > > > >>>>>> - Can I assume that each producer will send messages to one
> > > > queue?
> > > > >>>>>> - Global Order is to be identified across all
> > > > >>>>>> GlobalOrderedProducer (a new producer that is to be used
> for
> > > > global order)
> > > > >>>>>> instances that are running.
> > > > >>>>>> - I think using a global clock can help establish the order
> > > > >>>>>> between 2 or more producers, however using some form of
> > vector
> > > > >>>>>> clock might also help identify the global order of messages
> > > > between the
> > > > >>>>>> producers.
> > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how to read
> > > > >>>>>> globally ordered messages) can then compare messages across
> > all
> > > > message
> > > > >>>>>> queues from the corresponding producers and extract the
> > > > messages. [ this is
> > > > >>>>>> the approach recommended by yukon on the issue page ]
> > > > >>>>>> - We can also potentially have another layer in the Message
> > > Queue
> > > > >>>>>> which accumulates all messages sent from producers and
> > provides
> > > > one ordered
> > > > >>>>>> message queue for consumers to read from.
> > > > >>>>>>
> > > > >>>>>> Thank you for your patience and please let me know if my
> > > > >>>>>> understanding of the problem and the assumptions are right.
> > > > >>>>>>
> > > > >>>>>> Best Regards,
> > > > >>>>>> Sowmya
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> --
> > > > >>>>> Regards,
> > > > >>>>>
> > > > >>>>> Sowmya
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Regards,
> > > > >>>
> > > > >>> Sowmya
> > > > >>>
> > > > >>>
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Sowmya
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Sowmya
> > > >
> > >
> > --
> > Regards,
> >
> > Sowmya
> >
>
--
Regards,
Sowmya
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by yukon <yu...@apache.org>.
Cool, let's focus on it and see whether is there anything can be polished.
Regards
On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <so...@gmail.com> wrote:
> Hey yukon,
> I submitted my draft on the summer of code homepage a couple of days ago,
> also attaching the link here for reference,
>
> https://docs.google.com/file/d/1nXktUO_TF9-rSHSnGj5z5QZoHzMhosxm/edit?
> usp=docslist_api&filetype=msword
>
> Thanks,
> Sowmya
>
> On Thu, Mar 15, 2018 at 2:08 AM yukon <yu...@apache.org> wrote:
>
> > Hi,
> >
> > Of course, we can work together to finetune your design draft.
> >
> > Regards,
> > yukon
> >
> > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <so...@gmail.com> wrote:
> >
> > > Hello, yukon and Von,
> > >
> > > I've shared my GSOC - 18 draft of the project. I'm looking forward to
> > > working with all of you to finetune the proposal. I will be allocating
> 20
> > > hours per week from now to the proposal acceptance phase to address
> > > questions and dive deep into any suggestions that you provide.
> > > I am really looking forward to work on this project.
> > >
> > > thanks,
> > > Sowmya
> > >
> > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <so...@gmail.com>
> wrote:
> > >
> > > > Hello yukon,
> > > >
> > > > Thank you for the inputs. I was able to look at the ~/store and kind
> of
> > > > understand the storage structure. I also looked at the
> > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the
> > > examples.
> > > > Now I understand why you proposed a merge sort like approach for
> > > > performing global ordering. Since the proposals are open, I am
> > finalizing
> > > > my draft and will have it up for review very soon.
> > > >
> > > > thanks,
> > > > Sowmya
> > > >
> > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org> wrote:
> > > >
> > > >> Hi Sowmya,
> > > >>
> > > >> ```
> > > >> Also, it would be great if you can at a high level, help me
> understand
> > > >> how the messages in the message queues are stored before the
> consumer
> > > reads
> > > >> them.
> > > >> ```
> > > >>
> > > >>
> > > >>
> > > >> As shown in this figure, messages are sent to brokers by producer
> and
> > > >> stored in commit log[1], then messages are dispatched to
> ConsumeQueue
> > by
> > > >> topic, the consumer pulls messages from the queue.
> > > >>
> > > >> I recommend you run a broker and send/consume some messages, then
> > check
> > > >> out the ~/store directory for details.
> > > >>
> > > >> Regards,
> > > >> yukon
> > > >>
> > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <so...@gmail.com>
> > wrote:
> > > >>
> > > >>> Hello yukon,
> > > >>>
> > > >>> Currently FIFO can be achieved with a producer sending to one
> message
> > > >>> queue, and when global ordering is required, multiple producers
> have
> > to
> > > >>> send to a single topic queue.
> > > >>>
> > > >>> We want to allow multiple producers to send messages on a topic to
> > > >>> multiple message queues and still provide ordering guarantees to
> the
> > > >>> consumer, so that all consumers see the same order of data and also
> > the
> > > >>> data is delivered in an ordered fashion.
> > > >>>
> > > >>> 1) Your idea of using a merge sort with the assumption that the
> first
> > > >>> arriving message is treated as the first message to be delivered,
> > > however,
> > > >>> I want to propose an approach where when the producer sends a
> message
> > > to a
> > > >>> message queue, it must be done in a synchronous fashion and the
> > > response
> > > >>> will be that the message is accepted, which means that the message
> > > follows
> > > >>> the convention that all messages delivered prior by that producer
> > have
> > > been
> > > >>> stored across groups and if not the producer will need to resend
> the
> > > >>> message.
> > > >>>
> > > >>> We can use a variant of total causal ordering in the layer between
> > the
> > > >>> message queue and store.
> > > >>>
> > > >>> I have been busy with my class project so I couldn't make a lot of
> > > >>> progress in detailing my approach. Also, it would be great if you
> can
> > > at a
> > > >>> high level, help me understand how the messages in the message
> queues
> > > are
> > > >>> stored before the consumer reads them.
> > > >>>
> > > >>> Does the consumer read directly from the message queue that the
> > > producer
> > > >>> sends data to? does the broker receive the queued producer
> messages,
> > > store
> > > >>> them and then pushes them to another queue for the consumer to read
> > > from?
> > > >>>
> > > >>> For reference on total causal ordering:
> > https://www.cl.cam.ac.uk/teach
> > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
> > > >>>
> > > >>> thanks,
> > > >>> Sowmya
> > > >>>
> > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org> wrote:
> > > >>>
> > > >>>> Hi Sowmya,
> > > >>>>
> > > >>>> Sorry for the late reply, do you have any update on this project?
> > > >>>>
> > > >>>> In RocketMQ, one message queue is a FIFO queue natively, so I
> > proposed
> > > >>>> a simple solution that performs merge sort on multiple queues to
> > > improve
> > > >>>> performance and scalability.
> > > >>>>
> > > >>>> While the order issue across producers is difficult, we could
> assume
> > > >>>> that the message first arrives the broker should be consumed
> > firstly.
> > > >>>>
> > > >>>> But it would be wonderful if you have a real design about the
> order
> > > >>>> issue across producers based on the RocketMQ design and the
> storage
> > > >>>> structure.
> > > >>>>
> > > >>>> Looking forward your design ~
> > > >>>>
> > > >>>> Regards,
> > > >>>> yukon
> > > >>>>
> > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <so...@gmail.com>
> > > wrote:
> > > >>>>
> > > >>>>> Hello all,
> > > >>>>>
> > > >>>>> Adding a few more thoughts on the problem of establishing an
> order
> > of
> > > >>>>> messages across producers.
> > > >>>>>
> > > >>>>> Consider the Scenario
> > > >>>>>
> > > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1
> > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2.
> > > >>>>>
> > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and similarly
> > > >>>>> time(a2) < time(b2) < time(c2)
> > > >>>>>
> > > >>>>> Are the following orders acceptable to the consumer?
> > > >>>>>
> > > >>>>> > a1, a2, b1, b2, c1, c2
> > > >>>>> > a1, b1, c1, a2, b2, c2
> > > >>>>> > a1 b1, b2, a2, a3, b3
> > > >>>>>
> > > >>>>> and an order displayed at one consumer is consistent across all
> > > >>>>> consumers.
> > > >>>>>
> > > >>>>> This can be achieved using Total Causal Ordering at the Producer
> or
> > > >>>>> Queue level, using Leslie Lamport's clock and synchronization
> > > approach.
> > > >>>>>
> > > >>>>> For reference is the paper attached,
> > http://lamport.azurewebsites.n
> > > >>>>> et/pubs/time-clocks.pdf
> > > >>>>>
> > > >>>>>
> > > >>>>> --
> > > >>>>> Regards,
> > > >>>>>
> > > >>>>> Sowmya
> > > >>>>>
> > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <so...@gmail.com>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Hello,
> > > >>>>>>
> > > >>>>>> I'm trying to work on the issue ROCKETMQ-122
> > > >>>>>> <
> > https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065>
> > > as
> > > >>>>>> a part of Google Summer of Code 2018. I've been spending some
> time
> > > to
> > > >>>>>> understand the system, architecture and the existing Messaging
> > > Patterns.
> > > >>>>>> I still have a few questions and would like to clarify my
> > > assumptions.
> > > >>>>>>
> > > >>>>>> - Is the current FIFO order example limited to one message
> > queue
> > > >>>>>> per producer? Can the producer send the same message to
> > multiple
> > > queues?
> > > >>>>>> Will the consumers of the queues be able to read messages in
> > > Order?
> > > >>>>>> - Can I assume that each producer will send messages to one
> > > queue?
> > > >>>>>> - Global Order is to be identified across all
> > > >>>>>> GlobalOrderedProducer (a new producer that is to be used for
> > > global order)
> > > >>>>>> instances that are running.
> > > >>>>>> - I think using a global clock can help establish the order
> > > >>>>>> between 2 or more producers, however using some form of
> vector
> > > >>>>>> clock might also help identify the global order of messages
> > > between the
> > > >>>>>> producers.
> > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how to read
> > > >>>>>> globally ordered messages) can then compare messages across
> all
> > > message
> > > >>>>>> queues from the corresponding producers and extract the
> > > messages. [ this is
> > > >>>>>> the approach recommended by yukon on the issue page ]
> > > >>>>>> - We can also potentially have another layer in the Message
> > Queue
> > > >>>>>> which accumulates all messages sent from producers and
> provides
> > > one ordered
> > > >>>>>> message queue for consumers to read from.
> > > >>>>>>
> > > >>>>>> Thank you for your patience and please let me know if my
> > > >>>>>> understanding of the problem and the assumptions are right.
> > > >>>>>>
> > > >>>>>> Best Regards,
> > > >>>>>> Sowmya
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> --
> > > >>>>> Regards,
> > > >>>>>
> > > >>>>> Sowmya
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Regards,
> > > >>>
> > > >>> Sowmya
> > > >>>
> > > >>>
> > > >>
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Sowmya
> > > >
> > > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Sowmya
> > >
> >
> --
> Regards,
>
> Sowmya
>
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by sowmya s <so...@gmail.com>.
Hey yukon,
I submitted my draft on the summer of code homepage a couple of days ago,
also attaching the link here for reference,
https://docs.google.com/file/d/1nXktUO_TF9-rSHSnGj5z5QZoHzMhosxm/edit?usp=docslist_api&filetype=msword
Thanks,
Sowmya
On Thu, Mar 15, 2018 at 2:08 AM yukon <yu...@apache.org> wrote:
> Hi,
>
> Of course, we can work together to finetune your design draft.
>
> Regards,
> yukon
>
> On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <so...@gmail.com> wrote:
>
> > Hello, yukon and Von,
> >
> > I've shared my GSOC - 18 draft of the project. I'm looking forward to
> > working with all of you to finetune the proposal. I will be allocating 20
> > hours per week from now to the proposal acceptance phase to address
> > questions and dive deep into any suggestions that you provide.
> > I am really looking forward to work on this project.
> >
> > thanks,
> > Sowmya
> >
> > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <so...@gmail.com> wrote:
> >
> > > Hello yukon,
> > >
> > > Thank you for the inputs. I was able to look at the ~/store and kind of
> > > understand the storage structure. I also looked at the
> > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the
> > examples.
> > > Now I understand why you proposed a merge sort like approach for
> > > performing global ordering. Since the proposals are open, I am
> finalizing
> > > my draft and will have it up for review very soon.
> > >
> > > thanks,
> > > Sowmya
> > >
> > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org> wrote:
> > >
> > >> Hi Sowmya,
> > >>
> > >> ```
> > >> Also, it would be great if you can at a high level, help me understand
> > >> how the messages in the message queues are stored before the consumer
> > reads
> > >> them.
> > >> ```
> > >>
> > >>
> > >>
> > >> As shown in this figure, messages are sent to brokers by producer and
> > >> stored in commit log[1], then messages are dispatched to ConsumeQueue
> by
> > >> topic, the consumer pulls messages from the queue.
> > >>
> > >> I recommend you run a broker and send/consume some messages, then
> check
> > >> out the ~/store directory for details.
> > >>
> > >> Regards,
> > >> yukon
> > >>
> > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <so...@gmail.com>
> wrote:
> > >>
> > >>> Hello yukon,
> > >>>
> > >>> Currently FIFO can be achieved with a producer sending to one message
> > >>> queue, and when global ordering is required, multiple producers have
> to
> > >>> send to a single topic queue.
> > >>>
> > >>> We want to allow multiple producers to send messages on a topic to
> > >>> multiple message queues and still provide ordering guarantees to the
> > >>> consumer, so that all consumers see the same order of data and also
> the
> > >>> data is delivered in an ordered fashion.
> > >>>
> > >>> 1) Your idea of using a merge sort with the assumption that the first
> > >>> arriving message is treated as the first message to be delivered,
> > however,
> > >>> I want to propose an approach where when the producer sends a message
> > to a
> > >>> message queue, it must be done in a synchronous fashion and the
> > response
> > >>> will be that the message is accepted, which means that the message
> > follows
> > >>> the convention that all messages delivered prior by that producer
> have
> > been
> > >>> stored across groups and if not the producer will need to resend the
> > >>> message.
> > >>>
> > >>> We can use a variant of total causal ordering in the layer between
> the
> > >>> message queue and store.
> > >>>
> > >>> I have been busy with my class project so I couldn't make a lot of
> > >>> progress in detailing my approach. Also, it would be great if you can
> > at a
> > >>> high level, help me understand how the messages in the message queues
> > are
> > >>> stored before the consumer reads them.
> > >>>
> > >>> Does the consumer read directly from the message queue that the
> > producer
> > >>> sends data to? does the broker receive the queued producer messages,
> > store
> > >>> them and then pushes them to another queue for the consumer to read
> > from?
> > >>>
> > >>> For reference on total causal ordering:
> https://www.cl.cam.ac.uk/teach
> > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
> > >>>
> > >>> thanks,
> > >>> Sowmya
> > >>>
> > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org> wrote:
> > >>>
> > >>>> Hi Sowmya,
> > >>>>
> > >>>> Sorry for the late reply, do you have any update on this project?
> > >>>>
> > >>>> In RocketMQ, one message queue is a FIFO queue natively, so I
> proposed
> > >>>> a simple solution that performs merge sort on multiple queues to
> > improve
> > >>>> performance and scalability.
> > >>>>
> > >>>> While the order issue across producers is difficult, we could assume
> > >>>> that the message first arrives the broker should be consumed
> firstly.
> > >>>>
> > >>>> But it would be wonderful if you have a real design about the order
> > >>>> issue across producers based on the RocketMQ design and the storage
> > >>>> structure.
> > >>>>
> > >>>> Looking forward your design ~
> > >>>>
> > >>>> Regards,
> > >>>> yukon
> > >>>>
> > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <so...@gmail.com>
> > wrote:
> > >>>>
> > >>>>> Hello all,
> > >>>>>
> > >>>>> Adding a few more thoughts on the problem of establishing an order
> of
> > >>>>> messages across producers.
> > >>>>>
> > >>>>> Consider the Scenario
> > >>>>>
> > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1
> > >>>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2.
> > >>>>>
> > >>>>> If we assume that time(a1) < time(b1) < time(c1) and similarly
> > >>>>> time(a2) < time(b2) < time(c2)
> > >>>>>
> > >>>>> Are the following orders acceptable to the consumer?
> > >>>>>
> > >>>>> > a1, a2, b1, b2, c1, c2
> > >>>>> > a1, b1, c1, a2, b2, c2
> > >>>>> > a1 b1, b2, a2, a3, b3
> > >>>>>
> > >>>>> and an order displayed at one consumer is consistent across all
> > >>>>> consumers.
> > >>>>>
> > >>>>> This can be achieved using Total Causal Ordering at the Producer or
> > >>>>> Queue level, using Leslie Lamport's clock and synchronization
> > approach.
> > >>>>>
> > >>>>> For reference is the paper attached,
> http://lamport.azurewebsites.n
> > >>>>> et/pubs/time-clocks.pdf
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> Regards,
> > >>>>>
> > >>>>> Sowmya
> > >>>>>
> > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <so...@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Hello,
> > >>>>>>
> > >>>>>> I'm trying to work on the issue ROCKETMQ-122
> > >>>>>> <
> https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065>
> > as
> > >>>>>> a part of Google Summer of Code 2018. I've been spending some time
> > to
> > >>>>>> understand the system, architecture and the existing Messaging
> > Patterns.
> > >>>>>> I still have a few questions and would like to clarify my
> > assumptions.
> > >>>>>>
> > >>>>>> - Is the current FIFO order example limited to one message
> queue
> > >>>>>> per producer? Can the producer send the same message to
> multiple
> > queues?
> > >>>>>> Will the consumers of the queues be able to read messages in
> > Order?
> > >>>>>> - Can I assume that each producer will send messages to one
> > queue?
> > >>>>>> - Global Order is to be identified across all
> > >>>>>> GlobalOrderedProducer (a new producer that is to be used for
> > global order)
> > >>>>>> instances that are running.
> > >>>>>> - I think using a global clock can help establish the order
> > >>>>>> between 2 or more producers, however using some form of vector
> > >>>>>> clock might also help identify the global order of messages
> > between the
> > >>>>>> producers.
> > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how to read
> > >>>>>> globally ordered messages) can then compare messages across all
> > message
> > >>>>>> queues from the corresponding producers and extract the
> > messages. [ this is
> > >>>>>> the approach recommended by yukon on the issue page ]
> > >>>>>> - We can also potentially have another layer in the Message
> Queue
> > >>>>>> which accumulates all messages sent from producers and provides
> > one ordered
> > >>>>>> message queue for consumers to read from.
> > >>>>>>
> > >>>>>> Thank you for your patience and please let me know if my
> > >>>>>> understanding of the problem and the assumptions are right.
> > >>>>>>
> > >>>>>> Best Regards,
> > >>>>>> Sowmya
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> Regards,
> > >>>>>
> > >>>>> Sowmya
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>> Regards,
> > >>>
> > >>> Sowmya
> > >>>
> > >>>
> > >>
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Sowmya
> > >
> > >
> >
> >
> > --
> > Regards,
> >
> > Sowmya
> >
>
--
Regards,
Sowmya
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by yukon <yu...@apache.org>.
Hi,
Of course, we can work together to finetune your design draft.
Regards,
yukon
On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <so...@gmail.com> wrote:
> Hello, yukon and Von,
>
> I've shared my GSOC - 18 draft of the project. I'm looking forward to
> working with all of you to finetune the proposal. I will be allocating 20
> hours per week from now to the proposal acceptance phase to address
> questions and dive deep into any suggestions that you provide.
> I am really looking forward to work on this project.
>
> thanks,
> Sowmya
>
> On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <so...@gmail.com> wrote:
>
> > Hello yukon,
> >
> > Thank you for the inputs. I was able to look at the ~/store and kind of
> > understand the storage structure. I also looked at the
> > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the
> examples.
> > Now I understand why you proposed a merge sort like approach for
> > performing global ordering. Since the proposals are open, I am finalizing
> > my draft and will have it up for review very soon.
> >
> > thanks,
> > Sowmya
> >
> > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org> wrote:
> >
> >> Hi Sowmya,
> >>
> >> ```
> >> Also, it would be great if you can at a high level, help me understand
> >> how the messages in the message queues are stored before the consumer
> reads
> >> them.
> >> ```
> >>
> >>
> >>
> >> As shown in this figure, messages are sent to brokers by producer and
> >> stored in commit log[1], then messages are dispatched to ConsumeQueue by
> >> topic, the consumer pulls messages from the queue.
> >>
> >> I recommend you run a broker and send/consume some messages, then check
> >> out the ~/store directory for details.
> >>
> >> Regards,
> >> yukon
> >>
> >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <so...@gmail.com> wrote:
> >>
> >>> Hello yukon,
> >>>
> >>> Currently FIFO can be achieved with a producer sending to one message
> >>> queue, and when global ordering is required, multiple producers have to
> >>> send to a single topic queue.
> >>>
> >>> We want to allow multiple producers to send messages on a topic to
> >>> multiple message queues and still provide ordering guarantees to the
> >>> consumer, so that all consumers see the same order of data and also the
> >>> data is delivered in an ordered fashion.
> >>>
> >>> 1) Your idea of using a merge sort with the assumption that the first
> >>> arriving message is treated as the first message to be delivered,
> however,
> >>> I want to propose an approach where when the producer sends a message
> to a
> >>> message queue, it must be done in a synchronous fashion and the
> response
> >>> will be that the message is accepted, which means that the message
> follows
> >>> the convention that all messages delivered prior by that producer have
> been
> >>> stored across groups and if not the producer will need to resend the
> >>> message.
> >>>
> >>> We can use a variant of total causal ordering in the layer between the
> >>> message queue and store.
> >>>
> >>> I have been busy with my class project so I couldn't make a lot of
> >>> progress in detailing my approach. Also, it would be great if you can
> at a
> >>> high level, help me understand how the messages in the message queues
> are
> >>> stored before the consumer reads them.
> >>>
> >>> Does the consumer read directly from the message queue that the
> producer
> >>> sends data to? does the broker receive the queued producer messages,
> store
> >>> them and then pushes them to another queue for the consumer to read
> from?
> >>>
> >>> For reference on total causal ordering: https://www.cl.cam.ac.uk/teach
> >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
> >>>
> >>> thanks,
> >>> Sowmya
> >>>
> >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org> wrote:
> >>>
> >>>> Hi Sowmya,
> >>>>
> >>>> Sorry for the late reply, do you have any update on this project?
> >>>>
> >>>> In RocketMQ, one message queue is a FIFO queue natively, so I proposed
> >>>> a simple solution that performs merge sort on multiple queues to
> improve
> >>>> performance and scalability.
> >>>>
> >>>> While the order issue across producers is difficult, we could assume
> >>>> that the message first arrives the broker should be consumed firstly.
> >>>>
> >>>> But it would be wonderful if you have a real design about the order
> >>>> issue across producers based on the RocketMQ design and the storage
> >>>> structure.
> >>>>
> >>>> Looking forward your design ~
> >>>>
> >>>> Regards,
> >>>> yukon
> >>>>
> >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <so...@gmail.com>
> wrote:
> >>>>
> >>>>> Hello all,
> >>>>>
> >>>>> Adding a few more thoughts on the problem of establishing an order of
> >>>>> messages across producers.
> >>>>>
> >>>>> Consider the Scenario
> >>>>>
> >>>>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1
> >>>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2.
> >>>>>
> >>>>> If we assume that time(a1) < time(b1) < time(c1) and similarly
> >>>>> time(a2) < time(b2) < time(c2)
> >>>>>
> >>>>> Are the following orders acceptable to the consumer?
> >>>>>
> >>>>> > a1, a2, b1, b2, c1, c2
> >>>>> > a1, b1, c1, a2, b2, c2
> >>>>> > a1 b1, b2, a2, a3, b3
> >>>>>
> >>>>> and an order displayed at one consumer is consistent across all
> >>>>> consumers.
> >>>>>
> >>>>> This can be achieved using Total Causal Ordering at the Producer or
> >>>>> Queue level, using Leslie Lamport's clock and synchronization
> approach.
> >>>>>
> >>>>> For reference is the paper attached, http://lamport.azurewebsites.n
> >>>>> et/pubs/time-clocks.pdf
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Regards,
> >>>>>
> >>>>> Sowmya
> >>>>>
> >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <so...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hello,
> >>>>>>
> >>>>>> I'm trying to work on the issue ROCKETMQ-122
> >>>>>> <https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065>
> as
> >>>>>> a part of Google Summer of Code 2018. I've been spending some time
> to
> >>>>>> understand the system, architecture and the existing Messaging
> Patterns.
> >>>>>> I still have a few questions and would like to clarify my
> assumptions.
> >>>>>>
> >>>>>> - Is the current FIFO order example limited to one message queue
> >>>>>> per producer? Can the producer send the same message to multiple
> queues?
> >>>>>> Will the consumers of the queues be able to read messages in
> Order?
> >>>>>> - Can I assume that each producer will send messages to one
> queue?
> >>>>>> - Global Order is to be identified across all
> >>>>>> GlobalOrderedProducer (a new producer that is to be used for
> global order)
> >>>>>> instances that are running.
> >>>>>> - I think using a global clock can help establish the order
> >>>>>> between 2 or more producers, however using some form of vector
> >>>>>> clock might also help identify the global order of messages
> between the
> >>>>>> producers.
> >>>>>> - A GlobalOrderedConsumer ( consumer that knows how to read
> >>>>>> globally ordered messages) can then compare messages across all
> message
> >>>>>> queues from the corresponding producers and extract the
> messages. [ this is
> >>>>>> the approach recommended by yukon on the issue page ]
> >>>>>> - We can also potentially have another layer in the Message Queue
> >>>>>> which accumulates all messages sent from producers and provides
> one ordered
> >>>>>> message queue for consumers to read from.
> >>>>>>
> >>>>>> Thank you for your patience and please let me know if my
> >>>>>> understanding of the problem and the assumptions are right.
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Sowmya
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Regards,
> >>>>>
> >>>>> Sowmya
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Regards,
> >>>
> >>> Sowmya
> >>>
> >>>
> >>
> >
> >
> > --
> > Regards,
> >
> > Sowmya
> >
> >
>
>
> --
> Regards,
>
> Sowmya
>
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by sowmya s <so...@gmail.com>.
Hello, yukon and Von,
I've shared my GSOC - 18 draft of the project. I'm looking forward to
working with all of you to finetune the proposal. I will be allocating 20
hours per week from now to the proposal acceptance phase to address
questions and dive deep into any suggestions that you provide.
I am really looking forward to work on this project.
thanks,
Sowmya
On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <so...@gmail.com> wrote:
> Hello yukon,
>
> Thank you for the inputs. I was able to look at the ~/store and kind of
> understand the storage structure. I also looked at the
> DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the examples.
> Now I understand why you proposed a merge sort like approach for
> performing global ordering. Since the proposals are open, I am finalizing
> my draft and will have it up for review very soon.
>
> thanks,
> Sowmya
>
> On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org> wrote:
>
>> Hi Sowmya,
>>
>> ```
>> Also, it would be great if you can at a high level, help me understand
>> how the messages in the message queues are stored before the consumer reads
>> them.
>> ```
>>
>>
>>
>> As shown in this figure, messages are sent to brokers by producer and
>> stored in commit log[1], then messages are dispatched to ConsumeQueue by
>> topic, the consumer pulls messages from the queue.
>>
>> I recommend you run a broker and send/consume some messages, then check
>> out the ~/store directory for details.
>>
>> Regards,
>> yukon
>>
>> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <so...@gmail.com> wrote:
>>
>>> Hello yukon,
>>>
>>> Currently FIFO can be achieved with a producer sending to one message
>>> queue, and when global ordering is required, multiple producers have to
>>> send to a single topic queue.
>>>
>>> We want to allow multiple producers to send messages on a topic to
>>> multiple message queues and still provide ordering guarantees to the
>>> consumer, so that all consumers see the same order of data and also the
>>> data is delivered in an ordered fashion.
>>>
>>> 1) Your idea of using a merge sort with the assumption that the first
>>> arriving message is treated as the first message to be delivered, however,
>>> I want to propose an approach where when the producer sends a message to a
>>> message queue, it must be done in a synchronous fashion and the response
>>> will be that the message is accepted, which means that the message follows
>>> the convention that all messages delivered prior by that producer have been
>>> stored across groups and if not the producer will need to resend the
>>> message.
>>>
>>> We can use a variant of total causal ordering in the layer between the
>>> message queue and store.
>>>
>>> I have been busy with my class project so I couldn't make a lot of
>>> progress in detailing my approach. Also, it would be great if you can at a
>>> high level, help me understand how the messages in the message queues are
>>> stored before the consumer reads them.
>>>
>>> Does the consumer read directly from the message queue that the producer
>>> sends data to? does the broker receive the queued producer messages, store
>>> them and then pushes them to another queue for the consumer to read from?
>>>
>>> For reference on total causal ordering: https://www.cl.cam.ac.uk/teach
>>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
>>>
>>> thanks,
>>> Sowmya
>>>
>>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org> wrote:
>>>
>>>> Hi Sowmya,
>>>>
>>>> Sorry for the late reply, do you have any update on this project?
>>>>
>>>> In RocketMQ, one message queue is a FIFO queue natively, so I proposed
>>>> a simple solution that performs merge sort on multiple queues to improve
>>>> performance and scalability.
>>>>
>>>> While the order issue across producers is difficult, we could assume
>>>> that the message first arrives the broker should be consumed firstly.
>>>>
>>>> But it would be wonderful if you have a real design about the order
>>>> issue across producers based on the RocketMQ design and the storage
>>>> structure.
>>>>
>>>> Looking forward your design ~
>>>>
>>>> Regards,
>>>> yukon
>>>>
>>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <so...@gmail.com> wrote:
>>>>
>>>>> Hello all,
>>>>>
>>>>> Adding a few more thoughts on the problem of establishing an order of
>>>>> messages across producers.
>>>>>
>>>>> Consider the Scenario
>>>>>
>>>>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1
>>>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2.
>>>>>
>>>>> If we assume that time(a1) < time(b1) < time(c1) and similarly
>>>>> time(a2) < time(b2) < time(c2)
>>>>>
>>>>> Are the following orders acceptable to the consumer?
>>>>>
>>>>> > a1, a2, b1, b2, c1, c2
>>>>> > a1, b1, c1, a2, b2, c2
>>>>> > a1 b1, b2, a2, a3, b3
>>>>>
>>>>> and an order displayed at one consumer is consistent across all
>>>>> consumers.
>>>>>
>>>>> This can be achieved using Total Causal Ordering at the Producer or
>>>>> Queue level, using Leslie Lamport's clock and synchronization approach.
>>>>>
>>>>> For reference is the paper attached, http://lamport.azurewebsites.n
>>>>> et/pubs/time-clocks.pdf
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>>
>>>>> Sowmya
>>>>>
>>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <so...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I'm trying to work on the issue ROCKETMQ-122
>>>>>> <https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065> as
>>>>>> a part of Google Summer of Code 2018. I've been spending some time to
>>>>>> understand the system, architecture and the existing Messaging Patterns.
>>>>>> I still have a few questions and would like to clarify my assumptions.
>>>>>>
>>>>>> - Is the current FIFO order example limited to one message queue
>>>>>> per producer? Can the producer send the same message to multiple queues?
>>>>>> Will the consumers of the queues be able to read messages in Order?
>>>>>> - Can I assume that each producer will send messages to one queue?
>>>>>> - Global Order is to be identified across all
>>>>>> GlobalOrderedProducer (a new producer that is to be used for global order)
>>>>>> instances that are running.
>>>>>> - I think using a global clock can help establish the order
>>>>>> between 2 or more producers, however using some form of vector
>>>>>> clock might also help identify the global order of messages between the
>>>>>> producers.
>>>>>> - A GlobalOrderedConsumer ( consumer that knows how to read
>>>>>> globally ordered messages) can then compare messages across all message
>>>>>> queues from the corresponding producers and extract the messages. [ this is
>>>>>> the approach recommended by yukon on the issue page ]
>>>>>> - We can also potentially have another layer in the Message Queue
>>>>>> which accumulates all messages sent from producers and provides one ordered
>>>>>> message queue for consumers to read from.
>>>>>>
>>>>>> Thank you for your patience and please let me know if my
>>>>>> understanding of the problem and the assumptions are right.
>>>>>>
>>>>>> Best Regards,
>>>>>> Sowmya
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>>
>>>>> Sowmya
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>>
>>> Sowmya
>>>
>>>
>>
>
>
> --
> Regards,
>
> Sowmya
>
>
--
Regards,
Sowmya
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by sowmya s <so...@gmail.com>.
Hello yukon,
Thank you for the inputs. I was able to look at the ~/store and kind of
understand the storage structure. I also looked at the
DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the examples.
Now I understand why you proposed a merge sort like approach for performing
global ordering. Since the proposals are open, I am finalizing my draft and
will have it up for review very soon.
thanks,
Sowmya
On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org> wrote:
> Hi Sowmya,
>
> ```
> Also, it would be great if you can at a high level, help me understand how
> the messages in the message queues are stored before the consumer reads
> them.
> ```
>
>
>
> As shown in this figure, messages are sent to brokers by producer and
> stored in commit log[1], then messages are dispatched to ConsumeQueue by
> topic, the consumer pulls messages from the queue.
>
> I recommend you run a broker and send/consume some messages, then check
> out the ~/store directory for details.
>
> Regards,
> yukon
>
> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <so...@gmail.com> wrote:
>
>> Hello yukon,
>>
>> Currently FIFO can be achieved with a producer sending to one message
>> queue, and when global ordering is required, multiple producers have to
>> send to a single topic queue.
>>
>> We want to allow multiple producers to send messages on a topic to
>> multiple message queues and still provide ordering guarantees to the
>> consumer, so that all consumers see the same order of data and also the
>> data is delivered in an ordered fashion.
>>
>> 1) Your idea of using a merge sort with the assumption that the first
>> arriving message is treated as the first message to be delivered, however,
>> I want to propose an approach where when the producer sends a message to a
>> message queue, it must be done in a synchronous fashion and the response
>> will be that the message is accepted, which means that the message follows
>> the convention that all messages delivered prior by that producer have been
>> stored across groups and if not the producer will need to resend the
>> message.
>>
>> We can use a variant of total causal ordering in the layer between the
>> message queue and store.
>>
>> I have been busy with my class project so I couldn't make a lot of
>> progress in detailing my approach. Also, it would be great if you can at a
>> high level, help me understand how the messages in the message queues are
>> stored before the consumer reads them.
>>
>> Does the consumer read directly from the message queue that the producer
>> sends data to? does the broker receive the queued producer messages, store
>> them and then pushes them to another queue for the consumer to read from?
>>
>> For reference on total causal ordering: https://www.cl.cam.ac.uk/teach
>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
>>
>> thanks,
>> Sowmya
>>
>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org> wrote:
>>
>>> Hi Sowmya,
>>>
>>> Sorry for the late reply, do you have any update on this project?
>>>
>>> In RocketMQ, one message queue is a FIFO queue natively, so I proposed a
>>> simple solution that performs merge sort on multiple queues to improve
>>> performance and scalability.
>>>
>>> While the order issue across producers is difficult, we could assume
>>> that the message first arrives the broker should be consumed firstly.
>>>
>>> But it would be wonderful if you have a real design about the order
>>> issue across producers based on the RocketMQ design and the storage
>>> structure.
>>>
>>> Looking forward your design ~
>>>
>>> Regards,
>>> yukon
>>>
>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <so...@gmail.com> wrote:
>>>
>>>> Hello all,
>>>>
>>>> Adding a few more thoughts on the problem of establishing an order of
>>>> messages across producers.
>>>>
>>>> Consider the Scenario
>>>>
>>>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1
>>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2.
>>>>
>>>> If we assume that time(a1) < time(b1) < time(c1) and similarly time(a2)
>>>> < time(b2) < time(c2)
>>>>
>>>> Are the following orders acceptable to the consumer?
>>>>
>>>> > a1, a2, b1, b2, c1, c2
>>>> > a1, b1, c1, a2, b2, c2
>>>> > a1 b1, b2, a2, a3, b3
>>>>
>>>> and an order displayed at one consumer is consistent across all
>>>> consumers.
>>>>
>>>> This can be achieved using Total Causal Ordering at the Producer or
>>>> Queue level, using Leslie Lamport's clock and synchronization approach.
>>>>
>>>> For reference is the paper attached, http://lamport.azurewebsites.n
>>>> et/pubs/time-clocks.pdf
>>>>
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Sowmya
>>>>
>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <so...@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm trying to work on the issue ROCKETMQ-122
>>>>> <https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065> as
>>>>> a part of Google Summer of Code 2018. I've been spending some time to
>>>>> understand the system, architecture and the existing Messaging Patterns.
>>>>> I still have a few questions and would like to clarify my assumptions.
>>>>>
>>>>> - Is the current FIFO order example limited to one message queue
>>>>> per producer? Can the producer send the same message to multiple queues?
>>>>> Will the consumers of the queues be able to read messages in Order?
>>>>> - Can I assume that each producer will send messages to one queue?
>>>>> - Global Order is to be identified across all
>>>>> GlobalOrderedProducer (a new producer that is to be used for global order)
>>>>> instances that are running.
>>>>> - I think using a global clock can help establish the order
>>>>> between 2 or more producers, however using some form of vector
>>>>> clock might also help identify the global order of messages between the
>>>>> producers.
>>>>> - A GlobalOrderedConsumer ( consumer that knows how to read
>>>>> globally ordered messages) can then compare messages across all message
>>>>> queues from the corresponding producers and extract the messages. [ this is
>>>>> the approach recommended by yukon on the issue page ]
>>>>> - We can also potentially have another layer in the Message Queue
>>>>> which accumulates all messages sent from producers and provides one ordered
>>>>> message queue for consumers to read from.
>>>>>
>>>>> Thank you for your patience and please let me know if my understanding
>>>>> of the problem and the assumptions are right.
>>>>>
>>>>> Best Regards,
>>>>> Sowmya
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Sowmya
>>>>
>>>>
>>>
>>
>>
>> --
>> Regards,
>>
>> Sowmya
>>
>>
>
--
Regards,
Sowmya
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by yukon <yu...@apache.org>.
Hi Sowmya,
```
Also, it would be great if you can at a high level, help me understand how
the messages in the message queues are stored before the consumer reads
them.
```
As shown in this figure, messages are sent to brokers by producer and
stored in commit log[1], then messages are dispatched to ConsumeQueue by
topic, the consumer pulls messages from the queue.
I recommend you run a broker and send/consume some messages, then check out
the ~/store directory for details.
Regards,
yukon
On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <so...@gmail.com> wrote:
> Hello yukon,
>
> Currently FIFO can be achieved with a producer sending to one message
> queue, and when global ordering is required, multiple producers have to
> send to a single topic queue.
>
> We want to allow multiple producers to send messages on a topic to
> multiple message queues and still provide ordering guarantees to the
> consumer, so that all consumers see the same order of data and also the
> data is delivered in an ordered fashion.
>
> 1) Your idea of using a merge sort with the assumption that the first
> arriving message is treated as the first message to be delivered, however,
> I want to propose an approach where when the producer sends a message to a
> message queue, it must be done in a synchronous fashion and the response
> will be that the message is accepted, which means that the message follows
> the convention that all messages delivered prior by that producer have been
> stored across groups and if not the producer will need to resend the
> message.
>
> We can use a variant of total causal ordering in the layer between the
> message queue and store.
>
> I have been busy with my class project so I couldn't make a lot of
> progress in detailing my approach. Also, it would be great if you can at a
> high level, help me understand how the messages in the message queues are
> stored before the consumer reads them.
>
> Does the consumer read directly from the message queue that the producer
> sends data to? does the broker receive the queued producer messages, store
> them and then pushes them to another queue for the consumer to read from?
>
> For reference on total causal ordering: https://www.cl.cam.ac.uk/
> teaching/0910/ConcDistS/10b-ProcGp-order.pdf
>
> thanks,
> Sowmya
>
> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org> wrote:
>
>> Hi Sowmya,
>>
>> Sorry for the late reply, do you have any update on this project?
>>
>> In RocketMQ, one message queue is a FIFO queue natively, so I proposed a
>> simple solution that performs merge sort on multiple queues to improve
>> performance and scalability.
>>
>> While the order issue across producers is difficult, we could assume that
>> the message first arrives the broker should be consumed firstly.
>>
>> But it would be wonderful if you have a real design about the order
>> issue across producers based on the RocketMQ design and the storage
>> structure.
>>
>> Looking forward your design ~
>>
>> Regards,
>> yukon
>>
>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <so...@gmail.com> wrote:
>>
>>> Hello all,
>>>
>>> Adding a few more thoughts on the problem of establishing an order of
>>> messages across producers.
>>>
>>> Consider the Scenario
>>>
>>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1
>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2.
>>>
>>> If we assume that time(a1) < time(b1) < time(c1) and similarly time(a2)
>>> < time(b2) < time(c2)
>>>
>>> Are the following orders acceptable to the consumer?
>>>
>>> > a1, a2, b1, b2, c1, c2
>>> > a1, b1, c1, a2, b2, c2
>>> > a1 b1, b2, a2, a3, b3
>>>
>>> and an order displayed at one consumer is consistent across all
>>> consumers.
>>>
>>> This can be achieved using Total Causal Ordering at the Producer or
>>> Queue level, using Leslie Lamport's clock and synchronization approach.
>>>
>>> For reference is the paper attached, http://lamport.azurewebsites.n
>>> et/pubs/time-clocks.pdf
>>>
>>>
>>> --
>>> Regards,
>>>
>>> Sowmya
>>>
>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <so...@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I'm trying to work on the issue ROCKETMQ-122
>>>> <https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065> as
>>>> a part of Google Summer of Code 2018. I've been spending some time to
>>>> understand the system, architecture and the existing Messaging Patterns.
>>>> I still have a few questions and would like to clarify my assumptions.
>>>>
>>>> - Is the current FIFO order example limited to one message queue
>>>> per producer? Can the producer send the same message to multiple queues?
>>>> Will the consumers of the queues be able to read messages in Order?
>>>> - Can I assume that each producer will send messages to one queue?
>>>> - Global Order is to be identified across all GlobalOrderedProducer
>>>> (a new producer that is to be used for global order) instances that are
>>>> running.
>>>> - I think using a global clock can help establish the order between
>>>> 2 or more producers, however using some form of vector clock might
>>>> also help identify the global order of messages between the producers.
>>>> - A GlobalOrderedConsumer ( consumer that knows how to read
>>>> globally ordered messages) can then compare messages across all message
>>>> queues from the corresponding producers and extract the messages. [ this is
>>>> the approach recommended by yukon on the issue page ]
>>>> - We can also potentially have another layer in the Message Queue
>>>> which accumulates all messages sent from producers and provides one ordered
>>>> message queue for consumers to read from.
>>>>
>>>> Thank you for your patience and please let me know if my understanding
>>>> of the problem and the assumptions are right.
>>>>
>>>> Best Regards,
>>>> Sowmya
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>>
>>> Sowmya
>>>
>>>
>>
>
>
> --
> Regards,
>
> Sowmya
>
>
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by sowmya s <so...@gmail.com>.
Hello yukon,
Currently FIFO can be achieved with a producer sending to one message
queue, and when global ordering is required, multiple producers have to
send to a single topic queue.
We want to allow multiple producers to send messages on a topic to multiple
message queues and still provide ordering guarantees to the consumer, so
that all consumers see the same order of data and also the data is
delivered in an ordered fashion.
1) Your idea of using a merge sort with the assumption that the first
arriving message is treated as the first message to be delivered, however,
I want to propose an approach where when the producer sends a message to a
message queue, it must be done in a synchronous fashion and the response
will be that the message is accepted, which means that the message follows
the convention that all messages delivered prior by that producer have been
stored across groups and if not the producer will need to resend the
message.
We can use a variant of total causal ordering in the layer between the
message queue and store.
I have been busy with my class project so I couldn't make a lot of progress
in detailing my approach. Also, it would be great if you can at a high
level, help me understand how the messages in the message queues are stored
before the consumer reads them.
Does the consumer read directly from the message queue that the producer
sends data to? does the broker receive the queued producer messages, store
them and then pushes them to another queue for the consumer to read from?
For reference on total causal ordering:
https://www.cl.cam.ac.uk/teaching/0910/ConcDistS/10b-ProcGp-order.pdf
thanks,
Sowmya
On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org> wrote:
> Hi Sowmya,
>
> Sorry for the late reply, do you have any update on this project?
>
> In RocketMQ, one message queue is a FIFO queue natively, so I proposed a
> simple solution that performs merge sort on multiple queues to improve
> performance and scalability.
>
> While the order issue across producers is difficult, we could assume that
> the message first arrives the broker should be consumed firstly.
>
> But it would be wonderful if you have a real design about the order issue
> across producers based on the RocketMQ design and the storage structure.
>
> Looking forward your design ~
>
> Regards,
> yukon
>
> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <so...@gmail.com> wrote:
>
>> Hello all,
>>
>> Adding a few more thoughts on the problem of establishing an order of
>> messages across producers.
>>
>> Consider the Scenario
>>
>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1
>> Producer-2 produces messages a2, b2 and c2 into queue Queue2.
>>
>> If we assume that time(a1) < time(b1) < time(c1) and similarly time(a2) <
>> time(b2) < time(c2)
>>
>> Are the following orders acceptable to the consumer?
>>
>> > a1, a2, b1, b2, c1, c2
>> > a1, b1, c1, a2, b2, c2
>> > a1 b1, b2, a2, a3, b3
>>
>> and an order displayed at one consumer is consistent across all
>> consumers.
>>
>> This can be achieved using Total Causal Ordering at the Producer or Queue
>> level, using Leslie Lamport's clock and synchronization approach.
>>
>> For reference is the paper attached, http://lamport.azurewebsites.n
>> et/pubs/time-clocks.pdf
>>
>>
>> --
>> Regards,
>>
>> Sowmya
>>
>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <so...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I'm trying to work on the issue ROCKETMQ-122
>>> <https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065> as
>>> a part of Google Summer of Code 2018. I've been spending some time to
>>> understand the system, architecture and the existing Messaging Patterns.
>>> I still have a few questions and would like to clarify my assumptions.
>>>
>>> - Is the current FIFO order example limited to one message queue per
>>> producer? Can the producer send the same message to multiple queues? Will
>>> the consumers of the queues be able to read messages in Order?
>>> - Can I assume that each producer will send messages to one queue?
>>> - Global Order is to be identified across all GlobalOrderedProducer
>>> (a new producer that is to be used for global order) instances that are
>>> running.
>>> - I think using a global clock can help establish the order between
>>> 2 or more producers, however using some form of vector clock might
>>> also help identify the global order of messages between the producers.
>>> - A GlobalOrderedConsumer ( consumer that knows how to read globally
>>> ordered messages) can then compare messages across all message queues from
>>> the corresponding producers and extract the messages. [ this is the
>>> approach recommended by yukon on the issue page ]
>>> - We can also potentially have another layer in the Message Queue
>>> which accumulates all messages sent from producers and provides one ordered
>>> message queue for consumers to read from.
>>>
>>> Thank you for your patience and please let me know if my understanding
>>> of the problem and the assumptions are right.
>>>
>>> Best Regards,
>>> Sowmya
>>>
>>>
>>
>>
>> --
>> Regards,
>>
>> Sowmya
>>
>>
>
--
Regards,
Sowmya
Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging
Posted by yukon <yu...@apache.org>.
Hi Sowmya,
Sorry for the late reply, do you have any update on this project?
In RocketMQ, one message queue is a FIFO queue natively, so I proposed a
simple solution that performs merge sort on multiple queues to improve
performance and scalability.
While the order issue across producers is difficult, we could assume that
the message first arrives the broker should be consumed firstly.
But it would be wonderful if you have a real design about the order issue
across producers based on the RocketMQ design and the storage structure.
Looking forward your design ~
Regards,
yukon
On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <so...@gmail.com> wrote:
> Hello all,
>
> Adding a few more thoughts on the problem of establishing an order of
> messages across producers.
>
> Consider the Scenario
>
> Producer-1 produces messages a1, b1 and c1 into a queue Queue1
> Producer-2 produces messages a2, b2 and c2 into queue Queue2.
>
> If we assume that time(a1) < time(b1) < time(c1) and similarly time(a2) <
> time(b2) < time(c2)
>
> Are the following orders acceptable to the consumer?
>
> > a1, a2, b1, b2, c1, c2
> > a1, b1, c1, a2, b2, c2
> > a1 b1, b2, a2, a3, b3
>
> and an order displayed at one consumer is consistent across all consumers.
>
> This can be achieved using Total Causal Ordering at the Producer or Queue
> level, using Leslie Lamport's clock and synchronization approach.
>
> For reference is the paper attached, http://lamport.azurewebsites.
> net/pubs/time-clocks.pdf
>
>
> --
> Regards,
>
> Sowmya
>
> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <so...@gmail.com> wrote:
>
>> Hello,
>>
>> I'm trying to work on the issue ROCKETMQ-122
>> <https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065> as
>> a part of Google Summer of Code 2018. I've been spending some time to
>> understand the system, architecture and the existing Messaging Patterns.
>> I still have a few questions and would like to clarify my assumptions.
>>
>> - Is the current FIFO order example limited to one message queue per
>> producer? Can the producer send the same message to multiple queues? Will
>> the consumers of the queues be able to read messages in Order?
>> - Can I assume that each producer will send messages to one queue?
>> - Global Order is to be identified across all GlobalOrderedProducer
>> (a new producer that is to be used for global order) instances that are
>> running.
>> - I think using a global clock can help establish the order between 2
>> or more producers, however using some form of vector clock might also
>> help identify the global order of messages between the producers.
>> - A GlobalOrderedConsumer ( consumer that knows how to read globally
>> ordered messages) can then compare messages across all message queues from
>> the corresponding producers and extract the messages. [ this is the
>> approach recommended by yukon on the issue page ]
>> - We can also potentially have another layer in the Message Queue
>> which accumulates all messages sent from producers and provides one ordered
>> message queue for consumers to read from.
>>
>> Thank you for your patience and please let me know if my understanding of
>> the problem and the assumptions are right.
>>
>> Best Regards,
>> Sowmya
>>
>>
>
>
> --
> Regards,
>
> Sowmya
>
>