[GitHub] sanha opened a new pull request #92: [NEMO-176] Improve sequential read from disk

[GitBox <gi...@apache.org>]

sanha opened a new pull request #92: [NEMO-176] Improve sequential read from disk
URL: https://github.com/apache/incubator-nemo/pull/92
 
 
   JIRA: [NEMO-176: Improve sequential read from disk](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-176)
   
   **Major changes:**
   - Read whole data to read from file stream first and decode them later in `FileBlock`
   
   **Minor changes to note:**
   - Remove `limit` parameter in `InputStreamIterator` of `DataUtil` and make `DataUtil#deserializePartition` to limit the stream itself by the partition size.
   - Remove the assumption that "the source and destination parallelism of One-to-One communication is always 1" in `OutputWriter` and `InputReader`
   
   **Tests for the changes:**
   - Existing `DataTransferTest`, `BlockStoreTest` and other unit tests cover this change.
   - Existing integration tests also cover this change.
   
   **Other comments:**
   - N/A.
   
   resolves [NEMO-##](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-176)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services