You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon" <ed...@apache.org> on 2014/01/15 06:32:49 UTC
outgoingQueue and messages bundling
As I described here,
http://wiki.apache.org/hama/Architecture#Communication_and_Synchronization_Process
Currently, we puts the messages to be sent into outgoing queue. Then
we puts the messages of outgoing queue into BSPMessageBundle object
again.
This is huge overhead, and very inefficient in memory usage. Why
should we do like that?
Please let me know if i missing something important.
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: outgoingQueue and messages bundling
Posted by Suraj Menon <su...@apache.org>.
Nice! So you are reaching at some pain points of messaging code :). I
think, the code has been since I have seen it. It works for low number of
messages. For your concerns, there are message bundles not coming in the
path of the code yet in org.apache.hama.bsp.message.bundle . There is a
ByteBufferBSPMessageBundle in it. BSPMessageBundle represents unit piece of
data that could be serialized by sender and deserialized by receiver. It is
worthy for asynchronous communication. Your suggestion does make sense and
the whole thing requires code refactoring.
-Suraj
On Wed, Jan 15, 2014 at 3:44 AM, Edward J. Yoon <ed...@apache.org>wrote:
> If we use Spilling or Disk queues for outgoing queue, some memory
> consumption can be saved but we can't avoid deserializing and bundling
> messages overhead. According to my test, disk-based outgoing queue
> doesn't help much in memory usage.
>
> So, in my opinion, we should add outgoing messages in bundle object
> directly, and should consider about byte array bundle, or disk-based
> bundle. We can add something like OutgoingMessageManager.
>
> Does it make sense?
>
> On Wed, Jan 15, 2014 at 2:32 PM, Edward J. Yoon <ed...@apache.org>
> wrote:
> > As I described here,
> >
> http://wiki.apache.org/hama/Architecture#Communication_and_Synchronization_Process
> >
> > Currently, we puts the messages to be sent into outgoing queue. Then
> > we puts the messages of outgoing queue into BSPMessageBundle object
> > again.
> >
> > This is huge overhead, and very inefficient in memory usage. Why
> > should we do like that?
> >
> > Please let me know if i missing something important.
> >
> > --
> > Best Regards, Edward J. Yoon
> > @eddieyoon
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
Re: outgoingQueue and messages bundling
Posted by "Edward J. Yoon" <ed...@apache.org>.
If we use Spilling or Disk queues for outgoing queue, some memory
consumption can be saved but we can't avoid deserializing and bundling
messages overhead. According to my test, disk-based outgoing queue
doesn't help much in memory usage.
So, in my opinion, we should add outgoing messages in bundle object
directly, and should consider about byte array bundle, or disk-based
bundle. We can add something like OutgoingMessageManager.
Does it make sense?
On Wed, Jan 15, 2014 at 2:32 PM, Edward J. Yoon <ed...@apache.org> wrote:
> As I described here,
> http://wiki.apache.org/hama/Architecture#Communication_and_Synchronization_Process
>
> Currently, we puts the messages to be sent into outgoing queue. Then
> we puts the messages of outgoing queue into BSPMessageBundle object
> again.
>
> This is huge overhead, and very inefficient in memory usage. Why
> should we do like that?
>
> Please let me know if i missing something important.
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
--
Best Regards, Edward J. Yoon
@eddieyoon