[jira] [Commented] (REEF-1479) Define interface for distributed dataset

["Joo Seong (Jason) Jeong (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368549#comment-15368549 ] 

Joo Seong (Jason) Jeong commented on REEF-1479:
-----------------------------------------------

I've added some code (mostly interfaces) based on what we discussed so far, [here|https://github.com/jsjason/incubator-reef/commit/6fac4f98f1242a17ca631628bf60fbb40129e76e].
[~markus.weimer] [~shravanmn] [~dkm2110] Would you like to take a look?

> Define interface for distributed dataset 
> -----------------------------------------
>
>                 Key: REEF-1479
>                 URL: https://issues.apache.org/jira/browse/REEF-1479
>             Project: REEF
>          Issue Type: Sub-task
>          Components: REEF.NET
>            Reporter: Joo Seong (Jason) Jeong
>
> As a first step of [REEF-1477|https://issues.apache.org/jira/browse/REEF-1477], we'd like to define an interface for the distributed dataset that we will work with. This dataset interface serves as an abstraction of many dataset partitions, one on each Evaluator. In some sense, the class {{IPartitionedInputDataSet}} is very similar to what we want, except that the new interface will contain action methods like {{RunIMRU}} or {{RunTransform}}.
> {code}
> interface IDataSet<T> {
>   // apply a transform to this dataset
>   // transformConf gets shipped to each partition
>   // partition-wise operation
>   IDataSet<T'> TransformPartitions(IConfiguration transformConf);
>   // general interface for applying operations
>   // aware of all partitions, compared to TransformPartitions()
>   IDataSet<T'> RunStage(IConfiguration stageConf);
>   // fetch the actual data to the local process
>   // may result in OutOfMemory exception if T is too large
>   T[] Collect();
> }
> {code}
> Writing the data on stable storage via a {{Store()}} method can be considered as a special case of {{RunStage()}}. On the other hand, {{Load()}} must be defined in a separate interface/class and may be dependent on the backing filesystem.
> Other interfaces that allow 'Stage' implementations and partition access from Tasks must also be newly defined.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)