[jira] [Created] (GIRAPH-909) support succinct representation of messages in messagestores

["Pavan Kumar (JIRA)" <ji...@apache.org>]

Pavan Kumar created GIRAPH-909:
----------------------------------

             Summary: support succinct representation of messages in messagestores
                 Key: GIRAPH-909
                 URL: https://issues.apache.org/jira/browse/GIRAPH-909
             Project: Giraph
          Issue Type: Improvement
            Reporter: Pavan Kumar
            Assignee: Pavan Kumar


Currently we use ByteArrayVertexIdMessages data-structure to store vertex id & its messages. Even for ByteArrayOneToAllMessages data-structure, while storing in message-store we always convert it to ByteArrayVertexIdMessages, in this case if many vertices on a worker receive the same message, it is stored as many times. This uses up lot of memory, so if we have message stores that can avoid all this duplication in storing messages, we can bring down memory footprint by a large amount.

Note: however, here the pre-req is that your graph has to be partitioned such that a vertex sends messages to vertices only in a few other workers (not all of them - as happens with HashPartitioning)

This change depends heavily on GIRAPH-907 & GIRAPH-908
I already have patch for it, need to tidy up few things, will put it up by end of this week.



--
This message was sent by Atlassian JIRA
(v6.2#6252)