[jira] [Updated] (REEF-1479) Define interface for distributed dataset

["Joo Seong (Jason) Jeong (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/REEF-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joo Seong (Jason) Jeong updated REEF-1479:
------------------------------------------
    Description: 
As a first step of [REEF-1477|https://issues.apache.org/jira/browse/REEF-1477], we'd like to define an interface for the distributed dataset that we will work with. This dataset interface serves as an abstraction of many dataset partitions, one on each Evaluator. In some sense, the class {{IPartitionedInputDataSet}} is very similar to what we want, except that the new interface will contain action methods like {{RunIMRU}} or {{RunTransform}}.

{code}
interface IDataSet<T> {
  // apply a transform to this dataset
  // transformConf gets shipped to each partition
  // partition-wise operation
  IDataSet<T'> TransformPartitions(IConfiguration transformConf);

  // general interface for running operations
  // 
  IDataSet<T'> RunStage(IConfiguration stageConf);

  // store this dataset to some destination
  void Store(URI uri);
}

interface IDataSetLoader<T> {
  // generate a dataset from some source
  IDataSet<T> Load(URI uri);
}
{code}

  was:
As a first step of [REEF-1477|https://issues.apache.org/jira/browse/REEF-1477], we'd like to define an interface for the distributed dataset that we will work with. This dataset interface serves as an abstraction of many dataset partitions, one on each Evaluator. In some sense, the class {{IPartitionedInputDataSet}} is very similar to what we want, except that the new interface will contain action methods like {{RunIMRU}} or {{RunTransform}}.

{code}
interface IDataSet<T> {
  // apply a transform to this dataset
  IDataSet<T'> RunTransform(Transform<T, T'> transform);

  // run an IMRU job on this dataset and get some results
  T' RunIMRU(IMRUConfiguration<T, T'> imruConfiguration);

  // store this dataset to some destination
  void Store(URI uri);
}

interface IDataSetLoader<T> {
  // generate a dataset from some source
  IDataSet<T> Load(URI uri);
}
{code}


> Define interface for distributed dataset 
> -----------------------------------------
>
>                 Key: REEF-1479
>                 URL: https://issues.apache.org/jira/browse/REEF-1479
>             Project: REEF
>          Issue Type: Sub-task
>          Components: REEF.NET
>            Reporter: Joo Seong (Jason) Jeong
>
> As a first step of [REEF-1477|https://issues.apache.org/jira/browse/REEF-1477], we'd like to define an interface for the distributed dataset that we will work with. This dataset interface serves as an abstraction of many dataset partitions, one on each Evaluator. In some sense, the class {{IPartitionedInputDataSet}} is very similar to what we want, except that the new interface will contain action methods like {{RunIMRU}} or {{RunTransform}}.
> {code}
> interface IDataSet<T> {
>   // apply a transform to this dataset
>   // transformConf gets shipped to each partition
>   // partition-wise operation
>   IDataSet<T'> TransformPartitions(IConfiguration transformConf);
>   // general interface for running operations
>   // 
>   IDataSet<T'> RunStage(IConfiguration stageConf);
>   // store this dataset to some destination
>   void Store(URI uri);
> }
> interface IDataSetLoader<T> {
>   // generate a dataset from some source
>   IDataSet<T> Load(URI uri);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)