[jira] [Comment Edited] (REEF-1157) Improve memory usage and performance of the Java group communications code

["amil shanaka (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181500#comment-15181500 ] 

amil shanaka edited comment on REEF-1157 at 3/5/16 4:17 AM:
------------------------------------------------------------

Hi,
I'm a 3rd year under graduate in the department of Computer Science and Engineering at University of Moratuwa,Sri Lanka. In my experience, I've been worked with Java nio.ByteBuffer and also off-heap memory usage like java Unsafe,  nio.directByteBuffer and netty Direct Byte Buf for enhancing the performance with communicating meta data. So i would like to know more details regrading this project in detail that I could proceed .
Thank you,
amil


was (Author: amil101):
Hi,
I'm a 3rd year under graduate in the department of Computer Science and Engineering at University of Moratuwa,Sri Lanka. In my experience, I've been worked with Java nio.ByteBuffer and also off-heap memory usage like nio.directByteBuffer and netty Direct Byte Buf for enhancing the performance with communicating meta data. So i would like to know more details regrading this project in detail that I could proceed .
Thank you,
amil

> Improve memory usage and performance of the Java group communications code
> --------------------------------------------------------------------------
>
>                 Key: REEF-1157
>                 URL: https://issues.apache.org/jira/browse/REEF-1157
>             Project: REEF
>          Issue Type: Improvement
>          Components: GroupCommunications
>            Reporter: Markus Weimer
>            Priority: Minor
>              Labels: GSOC, gsoc2016, java
>
> The Group Communications library in .NET has seen two areas of improvement that we should follow up on in the Java code:
>   * Zero copies of the data being sent: the current code takes the user data, serializes it and then wraps it with the operator's envelope. That package is serialized and wrapped in the group communications envelope. Again, this is serialized and wrapped in the network service's envelope. That is what is eventually serialized and sent. All of these wrappings and serializations create copies of the (potentially large) user data. We should move to a zero-copy regime where the data is only serialized once.
>   * Pipelining: Many times, group communications is applied with vector types, e.g. when summing up {{double[]}}. Those operations can be done on a per-dimension basis, which reduces the memory load and improves throughput.
> This JIRA shall serve as an anchor for work done in this area.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)