You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon" <ed...@apache.org> on 2014/05/12 03:01:19 UTC
[DISCUSS] Disk Queue and Spilling Queue
The old design of outgoing/incoming message queues is readable but it
has some problems, and the most performance and memory issues are
dependent upon this part.
1) To send a messages to destination Peer, we serialize, compress, and
bundle the messages. So, using disk or spilling queue for the outgoing
messages is pointless and cause of degradation. This issue SOLVED by
HAMA-853. We'll need to add disk-based bundle in the future.
2) Receive-side queue is also the same. Instead of unbundling (and
deserializing, decompressing) bundles into {memory, disk, or spilling}
queue, we should use bundles in efficient and asynchronous way.
If you agree with this, I'll start to refactor the whole queue system.
If you have any other ideas e.g., asynchronous message
synchronization, Pls let me know.
Thanks.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
Re: [DISCUSS] Disk Queue and Spilling Queue
Posted by "Edward J. Yoon" <ed...@apache.org>.
Say, that's an idea. However, I think the complex option is a recipe
for confusions.
IMO, we have to finish the basics clearly. Otherwise, everything can
be fizzled out, like Fault Tolerance and Superstep API.
On Wed, May 28, 2014 at 7:42 PM, Tommaso Teofili
<to...@gmail.com> wrote:
> 2014-05-28 12:21 GMT+02:00 Edward J. Yoon <ed...@apache.org>:
>
>> Even though we use the disk or spilling queue, we need to add whole
>> messages into the bundle object (memory). So, I said, they are
>> pointless. The bundle should be spillable. Receive-side queue is also
>> the same.
>>
>
> +1
>
>
>>
>> If there's no objections, I'll remove the spilling queue.
>>
>
> I'm not sure that's the right move, in the latest months we've been adding
> and removing things back and forth in this area.
> For example beforehand we didn't bundle everything together and so the
> spilling queue made sense (if I recall correctly), again I would prefer to
> have everything to be configurable, so by default for example we have no
> spilling queue and bundle message but we may configure things in order to
> use the spilling queue and to eventually send single messages rather than
> the whole bundle.
>
> Regards,
> Tommaso
>
>
>>
>>
>> On Tue, May 13, 2014 at 4:55 PM, Andronidis Anastasios
>> <an...@hotmail.com> wrote:
>> > I am +1. That plan is going to make things much more straight to other
>> developers too.
>> >
>> > Anastasis
>> >
>> > On 13 Μαϊ 2014, at 7:27 π.μ., Edward J. Yoon <ed...@apache.org>
>> wrote:
>> >
>> >> Just opened https://issues.apache.org/jira/browse/HAMA-903
>> >>
>> >> On Tue, May 13, 2014 at 10:51 AM, Edward J. Yoon <ed...@apache.org>
>> wrote:
>> >>> The AbstractMessageManager.loopBackMessages() method unbundle messages
>> >>> into localQueueForNextIteration. Finally, in clearOutgoingMessages()
>> >>> method, localQueue is switched by localQueueForNextIteration.
>> >>>
>> >>> My goal is simplify this procedure. This is huge overhead.
>> >>>
>> >>> Basically, BSPMessageBundle provides iterator access to the messages
>> >>> contained in the bundle. So, we can skip whole conversion from bundle
>> >>> to queue.
>> >>>
>> >>>> Is it going to use rpc?
>> >>>
>> >>> I won't touch RPC mechanism. Only queue implementations will be
>> changed.
>> >>>
>> >>> On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin <cl...@googlemail.com>
>> wrote:
>> >>>> Is it going to use rpc?
>> >>>>
>> >>>> Will it still use the interface, for instance, MessageManager.java?
>> >>>>
>> >>>> Just to check if any point for integrating with the current ongoing
>> >>>> refactoring process.
>> >>>>
>> >>>> If possible, perhaps decoupling io part and rpc from interface would
>> >>>> somehow simplify the integration progress.
>> >>>>
>> >>>>
>> >>>> On 12 May 2014 09:01, Edward J. Yoon <ed...@apache.org> wrote:
>> >>>>> The old design of outgoing/incoming message queues is readable but it
>> >>>>> has some problems, and the most performance and memory issues are
>> >>>>> dependent upon this part.
>> >>>>>
>> >>>>> 1) To send a messages to destination Peer, we serialize, compress,
>> and
>> >>>>> bundle the messages. So, using disk or spilling queue for the
>> outgoing
>> >>>>> messages is pointless and cause of degradation. This issue SOLVED by
>> >>>>> HAMA-853. We'll need to add disk-based bundle in the future.
>> >>>>>
>> >>>>> 2) Receive-side queue is also the same. Instead of unbundling (and
>> >>>>> deserializing, decompressing) bundles into {memory, disk, or
>> spilling}
>> >>>>> queue, we should use bundles in efficient and asynchronous way.
>> >>>>>
>> >>>>> If you agree with this, I'll start to refactor the whole queue
>> system.
>> >>>>>
>> >>>>> If you have any other ideas e.g., asynchronous message
>> >>>>> synchronization, Pls let me know.
>> >>>>>
>> >>>>> Thanks.
>> >>>>>
>> >>>>> --
>> >>>>> Best Regards, Edward J. Yoon
>> >>>>> CEO at DataSayer Co., Ltd.
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Best Regards, Edward J. Yoon
>> >>> CEO at DataSayer Co., Ltd.
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards, Edward J. Yoon
>> >> CEO at DataSayer Co., Ltd.
>> >
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> CEO at DataSayer Co., Ltd.
>>
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
Re: [DISCUSS] Disk Queue and Spilling Queue
Posted by Tommaso Teofili <to...@gmail.com>.
2014-05-28 12:21 GMT+02:00 Edward J. Yoon <ed...@apache.org>:
> Even though we use the disk or spilling queue, we need to add whole
> messages into the bundle object (memory). So, I said, they are
> pointless. The bundle should be spillable. Receive-side queue is also
> the same.
>
+1
>
> If there's no objections, I'll remove the spilling queue.
>
I'm not sure that's the right move, in the latest months we've been adding
and removing things back and forth in this area.
For example beforehand we didn't bundle everything together and so the
spilling queue made sense (if I recall correctly), again I would prefer to
have everything to be configurable, so by default for example we have no
spilling queue and bundle message but we may configure things in order to
use the spilling queue and to eventually send single messages rather than
the whole bundle.
Regards,
Tommaso
>
>
> On Tue, May 13, 2014 at 4:55 PM, Andronidis Anastasios
> <an...@hotmail.com> wrote:
> > I am +1. That plan is going to make things much more straight to other
> developers too.
> >
> > Anastasis
> >
> > On 13 Μαϊ 2014, at 7:27 π.μ., Edward J. Yoon <ed...@apache.org>
> wrote:
> >
> >> Just opened https://issues.apache.org/jira/browse/HAMA-903
> >>
> >> On Tue, May 13, 2014 at 10:51 AM, Edward J. Yoon <ed...@apache.org>
> wrote:
> >>> The AbstractMessageManager.loopBackMessages() method unbundle messages
> >>> into localQueueForNextIteration. Finally, in clearOutgoingMessages()
> >>> method, localQueue is switched by localQueueForNextIteration.
> >>>
> >>> My goal is simplify this procedure. This is huge overhead.
> >>>
> >>> Basically, BSPMessageBundle provides iterator access to the messages
> >>> contained in the bundle. So, we can skip whole conversion from bundle
> >>> to queue.
> >>>
> >>>> Is it going to use rpc?
> >>>
> >>> I won't touch RPC mechanism. Only queue implementations will be
> changed.
> >>>
> >>> On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin <cl...@googlemail.com>
> wrote:
> >>>> Is it going to use rpc?
> >>>>
> >>>> Will it still use the interface, for instance, MessageManager.java?
> >>>>
> >>>> Just to check if any point for integrating with the current ongoing
> >>>> refactoring process.
> >>>>
> >>>> If possible, perhaps decoupling io part and rpc from interface would
> >>>> somehow simplify the integration progress.
> >>>>
> >>>>
> >>>> On 12 May 2014 09:01, Edward J. Yoon <ed...@apache.org> wrote:
> >>>>> The old design of outgoing/incoming message queues is readable but it
> >>>>> has some problems, and the most performance and memory issues are
> >>>>> dependent upon this part.
> >>>>>
> >>>>> 1) To send a messages to destination Peer, we serialize, compress,
> and
> >>>>> bundle the messages. So, using disk or spilling queue for the
> outgoing
> >>>>> messages is pointless and cause of degradation. This issue SOLVED by
> >>>>> HAMA-853. We'll need to add disk-based bundle in the future.
> >>>>>
> >>>>> 2) Receive-side queue is also the same. Instead of unbundling (and
> >>>>> deserializing, decompressing) bundles into {memory, disk, or
> spilling}
> >>>>> queue, we should use bundles in efficient and asynchronous way.
> >>>>>
> >>>>> If you agree with this, I'll start to refactor the whole queue
> system.
> >>>>>
> >>>>> If you have any other ideas e.g., asynchronous message
> >>>>> synchronization, Pls let me know.
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>> --
> >>>>> Best Regards, Edward J. Yoon
> >>>>> CEO at DataSayer Co., Ltd.
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards, Edward J. Yoon
> >>> CEO at DataSayer Co., Ltd.
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> CEO at DataSayer Co., Ltd.
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> CEO at DataSayer Co., Ltd.
>
Re: [DISCUSS] Disk Queue and Spilling Queue
Posted by "Edward J. Yoon" <ed...@apache.org>.
Even though we use the disk or spilling queue, we need to add whole
messages into the bundle object (memory). So, I said, they are
pointless. The bundle should be spillable. Receive-side queue is also
the same.
If there's no objections, I'll remove the spilling queue.
On Tue, May 13, 2014 at 4:55 PM, Andronidis Anastasios
<an...@hotmail.com> wrote:
> I am +1. That plan is going to make things much more straight to other developers too.
>
> Anastasis
>
> On 13 Μαϊ 2014, at 7:27 π.μ., Edward J. Yoon <ed...@apache.org> wrote:
>
>> Just opened https://issues.apache.org/jira/browse/HAMA-903
>>
>> On Tue, May 13, 2014 at 10:51 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>> The AbstractMessageManager.loopBackMessages() method unbundle messages
>>> into localQueueForNextIteration. Finally, in clearOutgoingMessages()
>>> method, localQueue is switched by localQueueForNextIteration.
>>>
>>> My goal is simplify this procedure. This is huge overhead.
>>>
>>> Basically, BSPMessageBundle provides iterator access to the messages
>>> contained in the bundle. So, we can skip whole conversion from bundle
>>> to queue.
>>>
>>>> Is it going to use rpc?
>>>
>>> I won't touch RPC mechanism. Only queue implementations will be changed.
>>>
>>> On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin <cl...@googlemail.com> wrote:
>>>> Is it going to use rpc?
>>>>
>>>> Will it still use the interface, for instance, MessageManager.java?
>>>>
>>>> Just to check if any point for integrating with the current ongoing
>>>> refactoring process.
>>>>
>>>> If possible, perhaps decoupling io part and rpc from interface would
>>>> somehow simplify the integration progress.
>>>>
>>>>
>>>> On 12 May 2014 09:01, Edward J. Yoon <ed...@apache.org> wrote:
>>>>> The old design of outgoing/incoming message queues is readable but it
>>>>> has some problems, and the most performance and memory issues are
>>>>> dependent upon this part.
>>>>>
>>>>> 1) To send a messages to destination Peer, we serialize, compress, and
>>>>> bundle the messages. So, using disk or spilling queue for the outgoing
>>>>> messages is pointless and cause of degradation. This issue SOLVED by
>>>>> HAMA-853. We'll need to add disk-based bundle in the future.
>>>>>
>>>>> 2) Receive-side queue is also the same. Instead of unbundling (and
>>>>> deserializing, decompressing) bundles into {memory, disk, or spilling}
>>>>> queue, we should use bundles in efficient and asynchronous way.
>>>>>
>>>>> If you agree with this, I'll start to refactor the whole queue system.
>>>>>
>>>>> If you have any other ideas e.g., asynchronous message
>>>>> synchronization, Pls let me know.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> --
>>>>> Best Regards, Edward J. Yoon
>>>>> CEO at DataSayer Co., Ltd.
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> CEO at DataSayer Co., Ltd.
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> CEO at DataSayer Co., Ltd.
>
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
Re: [DISCUSS] Disk Queue and Spilling Queue
Posted by Andronidis Anastasios <an...@hotmail.com>.
I am +1. That plan is going to make things much more straight to other developers too.
Anastasis
On 13 Μαϊ 2014, at 7:27 π.μ., Edward J. Yoon <ed...@apache.org> wrote:
> Just opened https://issues.apache.org/jira/browse/HAMA-903
>
> On Tue, May 13, 2014 at 10:51 AM, Edward J. Yoon <ed...@apache.org> wrote:
>> The AbstractMessageManager.loopBackMessages() method unbundle messages
>> into localQueueForNextIteration. Finally, in clearOutgoingMessages()
>> method, localQueue is switched by localQueueForNextIteration.
>>
>> My goal is simplify this procedure. This is huge overhead.
>>
>> Basically, BSPMessageBundle provides iterator access to the messages
>> contained in the bundle. So, we can skip whole conversion from bundle
>> to queue.
>>
>>> Is it going to use rpc?
>>
>> I won't touch RPC mechanism. Only queue implementations will be changed.
>>
>> On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin <cl...@googlemail.com> wrote:
>>> Is it going to use rpc?
>>>
>>> Will it still use the interface, for instance, MessageManager.java?
>>>
>>> Just to check if any point for integrating with the current ongoing
>>> refactoring process.
>>>
>>> If possible, perhaps decoupling io part and rpc from interface would
>>> somehow simplify the integration progress.
>>>
>>>
>>> On 12 May 2014 09:01, Edward J. Yoon <ed...@apache.org> wrote:
>>>> The old design of outgoing/incoming message queues is readable but it
>>>> has some problems, and the most performance and memory issues are
>>>> dependent upon this part.
>>>>
>>>> 1) To send a messages to destination Peer, we serialize, compress, and
>>>> bundle the messages. So, using disk or spilling queue for the outgoing
>>>> messages is pointless and cause of degradation. This issue SOLVED by
>>>> HAMA-853. We'll need to add disk-based bundle in the future.
>>>>
>>>> 2) Receive-side queue is also the same. Instead of unbundling (and
>>>> deserializing, decompressing) bundles into {memory, disk, or spilling}
>>>> queue, we should use bundles in efficient and asynchronous way.
>>>>
>>>> If you agree with this, I'll start to refactor the whole queue system.
>>>>
>>>> If you have any other ideas e.g., asynchronous message
>>>> synchronization, Pls let me know.
>>>>
>>>> Thanks.
>>>>
>>>> --
>>>> Best Regards, Edward J. Yoon
>>>> CEO at DataSayer Co., Ltd.
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> CEO at DataSayer Co., Ltd.
>
>
>
> --
> Best Regards, Edward J. Yoon
> CEO at DataSayer Co., Ltd.
Re: [DISCUSS] Disk Queue and Spilling Queue
Posted by "Edward J. Yoon" <ed...@apache.org>.
Just opened https://issues.apache.org/jira/browse/HAMA-903
On Tue, May 13, 2014 at 10:51 AM, Edward J. Yoon <ed...@apache.org> wrote:
> The AbstractMessageManager.loopBackMessages() method unbundle messages
> into localQueueForNextIteration. Finally, in clearOutgoingMessages()
> method, localQueue is switched by localQueueForNextIteration.
>
> My goal is simplify this procedure. This is huge overhead.
>
> Basically, BSPMessageBundle provides iterator access to the messages
> contained in the bundle. So, we can skip whole conversion from bundle
> to queue.
>
>> Is it going to use rpc?
>
> I won't touch RPC mechanism. Only queue implementations will be changed.
>
> On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin <cl...@googlemail.com> wrote:
>> Is it going to use rpc?
>>
>> Will it still use the interface, for instance, MessageManager.java?
>>
>> Just to check if any point for integrating with the current ongoing
>> refactoring process.
>>
>> If possible, perhaps decoupling io part and rpc from interface would
>> somehow simplify the integration progress.
>>
>>
>> On 12 May 2014 09:01, Edward J. Yoon <ed...@apache.org> wrote:
>>> The old design of outgoing/incoming message queues is readable but it
>>> has some problems, and the most performance and memory issues are
>>> dependent upon this part.
>>>
>>> 1) To send a messages to destination Peer, we serialize, compress, and
>>> bundle the messages. So, using disk or spilling queue for the outgoing
>>> messages is pointless and cause of degradation. This issue SOLVED by
>>> HAMA-853. We'll need to add disk-based bundle in the future.
>>>
>>> 2) Receive-side queue is also the same. Instead of unbundling (and
>>> deserializing, decompressing) bundles into {memory, disk, or spilling}
>>> queue, we should use bundles in efficient and asynchronous way.
>>>
>>> If you agree with this, I'll start to refactor the whole queue system.
>>>
>>> If you have any other ideas e.g., asynchronous message
>>> synchronization, Pls let me know.
>>>
>>> Thanks.
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> CEO at DataSayer Co., Ltd.
>
>
>
> --
> Best Regards, Edward J. Yoon
> CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
Re: [DISCUSS] Disk Queue and Spilling Queue
Posted by "Edward J. Yoon" <ed...@apache.org>.
The AbstractMessageManager.loopBackMessages() method unbundle messages
into localQueueForNextIteration. Finally, in clearOutgoingMessages()
method, localQueue is switched by localQueueForNextIteration.
My goal is simplify this procedure. This is huge overhead.
Basically, BSPMessageBundle provides iterator access to the messages
contained in the bundle. So, we can skip whole conversion from bundle
to queue.
> Is it going to use rpc?
I won't touch RPC mechanism. Only queue implementations will be changed.
On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin <cl...@googlemail.com> wrote:
> Is it going to use rpc?
>
> Will it still use the interface, for instance, MessageManager.java?
>
> Just to check if any point for integrating with the current ongoing
> refactoring process.
>
> If possible, perhaps decoupling io part and rpc from interface would
> somehow simplify the integration progress.
>
>
> On 12 May 2014 09:01, Edward J. Yoon <ed...@apache.org> wrote:
>> The old design of outgoing/incoming message queues is readable but it
>> has some problems, and the most performance and memory issues are
>> dependent upon this part.
>>
>> 1) To send a messages to destination Peer, we serialize, compress, and
>> bundle the messages. So, using disk or spilling queue for the outgoing
>> messages is pointless and cause of degradation. This issue SOLVED by
>> HAMA-853. We'll need to add disk-based bundle in the future.
>>
>> 2) Receive-side queue is also the same. Instead of unbundling (and
>> deserializing, decompressing) bundles into {memory, disk, or spilling}
>> queue, we should use bundles in efficient and asynchronous way.
>>
>> If you agree with this, I'll start to refactor the whole queue system.
>>
>> If you have any other ideas e.g., asynchronous message
>> synchronization, Pls let me know.
>>
>> Thanks.
>>
>> --
>> Best Regards, Edward J. Yoon
>> CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
Re: [DISCUSS] Disk Queue and Spilling Queue
Posted by Chia-Hung Lin <cl...@googlemail.com>.
Is it going to use rpc?
Will it still use the interface, for instance, MessageManager.java?
Just to check if any point for integrating with the current ongoing
refactoring process.
If possible, perhaps decoupling io part and rpc from interface would
somehow simplify the integration progress.
On 12 May 2014 09:01, Edward J. Yoon <ed...@apache.org> wrote:
> The old design of outgoing/incoming message queues is readable but it
> has some problems, and the most performance and memory issues are
> dependent upon this part.
>
> 1) To send a messages to destination Peer, we serialize, compress, and
> bundle the messages. So, using disk or spilling queue for the outgoing
> messages is pointless and cause of degradation. This issue SOLVED by
> HAMA-853. We'll need to add disk-based bundle in the future.
>
> 2) Receive-side queue is also the same. Instead of unbundling (and
> deserializing, decompressing) bundles into {memory, disk, or spilling}
> queue, we should use bundles in efficient and asynchronous way.
>
> If you agree with this, I'll start to refactor the whole queue system.
>
> If you have any other ideas e.g., asynchronous message
> synchronization, Pls let me know.
>
> Thanks.
>
> --
> Best Regards, Edward J. Yoon
> CEO at DataSayer Co., Ltd.