You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon" <ed...@apache.org> on 2014/01/09 09:21:32 UTC

About HAMA-734 and queue behaviours

Hi,

I looked at HAMA-734 and HAMA-704.patch-v1 today, and it seems quite invalid.

HAMA-734's concept is creating a additional storage for persisting
messages. The problem is when there are huge amount of messages to be
persisted, like VertexIndexedMessageStorage in HAMA-704.patch-v1 case.
And in the worst case, the size of storage can be continuously
increased, until starting to read the messages from there. So, we have
to consider the memory-efficient storage again, like DiskQueue or
Spilling Queue. Moreover, every messages should be polled from
incoming queue, and should be determined whether to be stored in queue
or additional storage.

Simply, I like to add two queue behaviours: Regular and Persistent. If
the queue behaviour is regular, all of the unconsumed messages will be
lost (reset in every superstep). In persistent case, unconsumed
messages will be persisted in incoming queue.

If there are no feedbacks, I'll close the HAMA-734 as a invalid and
start to work on Queue implementations.

-- 
Best Regards, Edward J. Yoon
@eddieyoon