[jira] [Commented] (HAMA-961) Parameter Server for large scale MLP

["Edward J. Yoon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HAMA-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612692#comment-14612692 ] 

Edward J. Yoon commented on HAMA-961:
-------------------------------------

Here's new neuron-centric programming interface I propose. I'll start the refactoring first.

{code}
public class ThreeLayerANNExample {

  /**
   * User-defined sigmoid actiavation function
   */
  public static class Sigmoid extends ActivationFunction {
    @Override
    public double apply(double input) {
      return 1.0 / (1 + Math.exp(-input));
    }

    @Override
    public double applyDerivative(double input) {
      return input * (1 - input);
    }
  }

  /**
   * User-defined cost function
   */
  public static class CrossEntropy extends CostFunction {
    @Override
    public double apply(double target, double actual) {
      double adjustedTarget = (target == 0 ? 0.000001 : target);
      adjustedTarget = (target == 1.0 ? 0.999999 : target);
      double adjustedActual = (actual == 0 ? 0.000001 : actual);
      adjustedActual = (actual == 1 ? 0.999999 : actual);
      return -adjustedTarget * Math.log(adjustedActual) - (1 - adjustedTarget)
          * Math.log(1 - adjustedActual);
    }

    @Override
    public double derivative(double target, double actual) {
      double adjustedTarget = target;
      double adjustedActual = actual;
      if (adjustedActual == 1) {
        adjustedActual = 0.999;
      } else if (actual == 0) {
        adjustedActual = 0.001;
      }
      if (adjustedTarget == 1) {
        adjustedTarget = 0.999;
      } else if (adjustedTarget == 0) {
        adjustedTarget = 0.001;
      }
      return -adjustedTarget / adjustedActual + (1 - adjustedTarget)
          / (1 - adjustedActual);
    }
  }

  public static void main(String[] args) throws Exception {
    ANNJob ann = new ANNJob();

    // set learning rate and momentum weight
    ann.setLearningRate(0.1);
    ann.setMomemtumWeight(0.1);

    // initialize the topology of the model and set the activation function and parallel degree.
    ann.addLayer(featureDimension, Sigmoid.class, numOfTasks);
    ann.addLayer(featureDimension, Sigmoid.class, numOfTasks);
    ann.addLayer(labelDimension, Sigmoid.class, numOfTasks);
  
    // set the cost function to evaluate the error
    ann.setCostFunction(CrossEntropy.class);

    ann.setMaxIteration(50);
    ann.setBatchSize(500);
    ann.setInputPath(path);
    ...
  
  }
}
{code}

> Parameter Server for large scale MLP
> ------------------------------------
>
>                 Key: HAMA-961
>                 URL: https://issues.apache.org/jira/browse/HAMA-961
>             Project: Hama
>          Issue Type: Improvement
>          Components: machine learning
>    Affects Versions: 0.7.0
>            Reporter: Edward J. Yoon
>            Assignee: Edward J. Yoon
>             Fix For: 0.8.0
>
>
> I've recently started to review the MLP source codes closely, and I'm thinking about some improvement and API refactoring e.g., APIs for user-defined neuron and synapse models, data structure, ..., etc.
> This issue is one of them, and related to train large models. I'm considering distributed parameter server (http://parameterserver.org) for managing parameters. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)