[jira] [Commented] (REEF-179) Solve the large memory footprint problem due to multiple serializations during call to an operator

["Shravan Matthur Narayanamurthy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344163#comment-14344163 ] 

Shravan Matthur Narayanamurthy commented on REEF-179:
-----------------------------------------------------

I don't think I can work on this right now. So I have marked it unassigned.

My take on the codec situation:
User codec should not be called till it is finally needed. We need to hang on to references of these codecs. Finally when you have to write it to the stream, you need to send the output stream as a parameter into the sequence of codecs so that the user codec can finally write it directly into the socket and not to any temp space. A first effort on the java side was the StreamingCodec. Taking this to its logical conclusion, as far as I can remember, would have been a major effort.

> Solve the large memory footprint problem due to multiple serializations during call to an operator
> --------------------------------------------------------------------------------------------------
>
>                 Key: REEF-179
>                 URL: https://issues.apache.org/jira/browse/REEF-179
>             Project: REEF
>          Issue Type: Improvement
>          Components: REEF-IO, REEF.NET, Wake
>            Reporter: Dhruv Mahajan
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Right now  during each communication call, there are three serialization steps happening where actual data/message to pass is encoded again and again leading to larger memory footprints. We would like to optimize it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)