[jira] [Commented] (REEF-1206) Update Deserializer API to allow deserialize from remote files directly

["Markus Weimer (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149483#comment-15149483 ] 

Markus Weimer commented on REEF-1206:
-------------------------------------

{quote}
For deserializer, it also needs to know the file paths are local or remote. For remote files, deserializer must use IFileSystem to access the files. For local files, normal .Net file system can be simply used. 
{quote}

Can't we use IFileSystem in both cases? We have local and remote implementations of that.

Also, the deserializer shouldn't have to know any of this, right? It could be based on {{Stream}} instances alone.

> Update Deserializer API to allow deserialize from remote files directly
> -----------------------------------------------------------------------
>
>                 Key: REEF-1206
>                 URL: https://issues.apache.org/jira/browse/REEF-1206
>             Project: REEF
>          Issue Type: Task
>            Reporter: Julia
>            Assignee: Julia
>
> Currently Deserialize() API accept a folder as input parameter. It only works for local files as only in local case, we can guarantee all the files are copied to the same folder and the folder only contains the files that need to be deserialized.  
> For remote file case, the file paths are passed from upper stream, the file folder may contain some other irrelevant files. In this case, passing a set of individual file oaths would be more more sense. 
> For deserializer, it also needs to know the file paths are local or remote. For remote files, deserializer must use IFileSystem to access the files. For local files, normal .Net file system can be simply used. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)