You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by bgreeven <gi...@git.apache.org> on 2014/07/03 06:03:58 UTC

[GitHub] spark pull request: [spark-2352] Implementation of an 1-hidden lay...

GitHub user bgreeven opened a pull request:

    https://github.com/apache/spark/pull/1290

    [spark-2352] Implementation of an 1-hidden layer Artificial Neural Network (ANN)

    The code contains a single layer ANN, with variable number of inputs, outputs and hidden nodes. It takes as input an RDD vector pairs, corresponding to the training set with inputs and outputs.
    
    A test program is also included, which also contains a graphical representation that can be switched on using the "graph" parameter. Without it, the summed squared error over the testing set is displayed.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/bgreeven/spark master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/1290.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1290
    
----
commit e60d8786d16e55264a61d8afc8d67bf06068aaaf
Author: Bert Greevenbosch <be...@huawei.com>
Date:   2014-07-03T03:33:22Z

    Create ParallelANN.scala
    
    This is the main ParallelANN class and associated Model

commit 52da23d54254b38eea7181872e4caa25981c028e
Author: Bert Greevenbosch <be...@huawei.com>
Date:   2014-07-03T03:34:41Z

    Create GeneralizedSteepestDescendAlgorithm
    
    This is the general steepest descend model, with as inputs Vectors and outputs Vectors or Doubles.

commit 152b8baf84742ecd0c622d41d3804eb74c0310a3
Author: Bert Greevenbosch <be...@huawei.com>
Date:   2014-07-03T03:36:31Z

    Create TestParallelANN.scala
    
    This is a test program for parallel ANNs.

commit c8af840149ac8d1903afe4ac826a626d030bd385
Author: Bert Greevenbosch <be...@huawei.com>
Date:   2014-07-03T03:42:29Z

    Create TestParallelANNgraphics.scala
    
    Visualisation tools; only used when "TestParallelANN" is given the "graph" parameter.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r19733442
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/ann/ArtificialNeuralNetwork.scala ---
    @@ -0,0 +1,528 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.mllib.ann
    +
    +import breeze.linalg.{DenseVector, Vector => BV, axpy => brzAxpy}
    +
    +import org.apache.spark.mllib.linalg.{Vector, Vectors}
    +import org.apache.spark.mllib.optimization._
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.random.XORShiftRandom
    +
    +/*
    + * Implements a Artificial Neural Network (ANN)
    + *
    + * The data consists of an input vector and an output vector, combined into a single vector
    + * as follows:
    + *
    + * [ ---input--- ---output--- ]
    + *
    + * NOTE: output values should be in the range [0,1]
    + *
    + * For a network of H hidden layers:
    + *
    + * hiddenLayersTopology(h) indicates the number of nodes in hidden layer h, excluding the bias
    + * node. h counts from 0 (first hidden layer, taking inputs from input layer) to H - 1 (last
    + * hidden layer, sending outputs to the output layer).
    + *
    + * hiddenLayersTopology is converted internally to topology, which adds the number of nodes
    + * in the input and output layers.
    + *
    + * noInput = topology(0), the number of input nodes
    + * noOutput = topology(L-1), the number of output nodes
    + *
    + * input = data( 0 to noInput-1 )
    + * output = data( noInput to noInput + noOutput - 1 )
    + *
    + * W_ijl is the weight from node i in layer l-1 to node j in layer l
    + * W_ijl goes to position ofsWeight(l) + j*(topology(l-1)+1) + i in the weights vector
    + *
    + * B_jl is the bias input of node j in layer l
    + * B_jl goes to position ofsWeight(l) + j*(topology(l-1)+1) + topology(l-1) in the weights vector
    + *
    + * error function: E( O, Y ) = sum( O_j - Y_j )
    + * (with O = (O_0, ..., O_(noOutput-1)) the output of the ANN,
    + * and (Y_0, ..., Y_(noOutput-1)) the input)
    + *
    + * node_jl is node j in layer l
    + * node_jl goes to position ofsNode(l) + j
    + *
    + * The weights gradient is defined as dE/dW_ijl and dE/dB_jl
    + * It has same mapping as W_ijl and B_jl
    + *
    + * For back propagation:
    + * delta_jl = dE/dS_jl, where S_jl the output of node_jl, but before applying the sigmoid
    + * delta_jl has the same mapping as node_jl
    + *
    + * Where E = ((estOutput-output),(estOutput-output)),
    + * the inner product of the difference between estimation and target output with itself.
    + *
    + */
    +
    +/**
    + * Artificial neural network (ANN) model
    + *
    + * @param weights the weights between the neurons in the ANN.
    + * @param topology array containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + */
    +class ArtificialNeuralNetworkModel private[mllib](val weights: Vector, val topology: Array[Int])
    +  extends Serializable with ANNHelper {
    +
    +  /**
    +   * Predicts values for a single data point using the trained model.
    +   *
    +   * @param testData represents a single data point.
    +   * @return prediction using the trained model.
    +   */
    +  def predict(testData: Vector): Vector = {
    +    Vectors.dense(computeValues(testData.toArray, weights.toArray))
    +  }
    +
    +  /**
    +   * Predict values for an RDD of data points using the trained model.
    +   *
    +   * @param testDataRDD RDD representing the input vectors.
    +   * @return RDD with predictions using the trained model as (input, output) pairs.
    +   */
    +  def predict(testDataRDD: RDD[Vector]): RDD[(Vector,Vector)] = {
    +    testDataRDD.map(T => (T, predict(T)) )
    +  }
    +
    +  private def computeValues(arrData: Array[Double], arrWeights: Array[Double]): Array[Double] = {
    +    val arrNodes = forwardRun(arrData, arrWeights)
    +    arrNodes.slice(arrNodes.size - topology(L), arrNodes.size)
    +  }
    +}
    +
    +/**
    + * Performs the training of an Artificial Neural Network (ANN)
    + *
    + * @param topology A vector containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + * @param maxNumIterations The maximum number of iterations for the training phase.
    + * @param convergenceTol Convergence tolerance for LBFGS. Smaller value for closer convergence.
    + */
    +class ArtificialNeuralNetwork private[mllib](
    +    topology: Array[Int],
    +    maxNumIterations: Int,
    +    convergenceTol: Double)
    +  extends Serializable {
    +
    +  private val gradient = new ANNLeastSquaresGradient(topology)
    +  private val updater = new ANNUpdater()
    +  private val optimizer = new LBFGS(gradient, updater).
    +    setConvergenceTol(convergenceTol).
    +    setMaxNumIterations(maxNumIterations)
    +
    + /**
    +   * Trains the ANN model.
    +   * Uses default convergence tolerance 1e-4 for LBFGS.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param initialWeights the initial weights of the ANN
    +   * @return ANN model.
    +   */
    +  private def run(trainingRDD: RDD[(Vector, Vector)], initialWeights: Vector):
    +      ArtificialNeuralNetworkModel = {
    +    val data = trainingRDD.map(v =>
    +      (0.0,
    +        Vectors.fromBreeze(DenseVector.vertcat(
    +          v._1.toBreeze.toDenseVector,
    +          v._2.toBreeze.toDenseVector))
    +        ))
    +    val weights = optimizer.optimize(data, initialWeights)
    +    new ArtificialNeuralNetworkModel(weights, topology)
    +  }
    +}
    +
    +/**
    + * Top level methods for training the artificial neural network (ANN)
    + */
    +object ArtificialNeuralNetwork {
    +
    +  private val defaultTolerance: Double = 1e-4
    +
    +  /**
    +   * Trains an ANN.
    +   * Uses default convergence tolerance 1e-4 for LBFGS.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param hiddenLayersTopology number of nodes per hidden layer, excluding the bias nodes.
    +   * @param maxNumIterations specifies maximum number of training iterations.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector, Vector)],
    +      hiddenLayersTopology: Array[Int],
    +      maxNumIterations: Int): ArtificialNeuralNetworkModel = {
    +    train(trainingRDD, hiddenLayersTopology, maxNumIterations, defaultTolerance)
    +  }
    +
    +  /**
    +   * Continues training of an ANN.
    +   * Uses default convergence tolerance 1e-4 for LBFGS.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param model model of an already partly trained ANN.
    +   * @param maxNumIterations maximum number of training iterations.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      model: ArtificialNeuralNetworkModel,
    +      maxNumIterations: Int): ArtificialNeuralNetworkModel = {
    +    train(trainingRDD, model, maxNumIterations, defaultTolerance)
    +  }
    +
    +  /**
    +   * Trains an ANN with given initial weights.
    +   * Uses default convergence tolerance 1e-4 for LBFGS.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param initialWeights initial weights vector.
    +   * @param maxNumIterations maximum number of training iterations.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      hiddenLayersTopology: Array[Int],
    +      initialWeights: Vector,
    +      maxNumIterations: Int): ArtificialNeuralNetworkModel = {
    +    train(trainingRDD, hiddenLayersTopology, initialWeights, maxNumIterations, defaultTolerance)
    +  }
    +
    +  /**
    +   * Trains an ANN using customized convergence tolerance.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param model model of an already partly trained ANN.
    +   * @param maxNumIterations maximum number of training iterations.
    +   * @param convergenceTol convergence tolerance for LBFGS. Smaller value for closer convergence.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      model: ArtificialNeuralNetworkModel,
    +      maxNumIterations: Int,
    +      convergenceTol: Double): ArtificialNeuralNetworkModel = {
    +    new ArtificialNeuralNetwork(model.topology, maxNumIterations, convergenceTol).
    +      run(trainingRDD, model.weights)
    +  }
    +
    +  /**
    +   * Continues training of an ANN using customized convergence tolerance.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param hiddenLayersTopology number of nodes per hidden layer, excluding the bias nodes.
    +   * @param maxNumIterations maximum number of training iterations.
    +   * @param convergenceTol convergence tolerance for LBFGS. Smaller value for closer convergence.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector, Vector)],
    +      hiddenLayersTopology: Array[Int],
    +      maxNumIterations: Int,
    +      convergenceTol: Double): ArtificialNeuralNetworkModel = {
    +    val topology = convertTopology(trainingRDD, hiddenLayersTopology)
    +    new ArtificialNeuralNetwork(topology, maxNumIterations, convergenceTol).
    +      run(trainingRDD, randomWeights(topology, false))
    +  }
    +
    +  /**
    +   * Trains an ANN with given initial weights.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param initialWeights initial weights vector.
    +   * @param maxNumIterations maximum number of training iterations.
    +   * @param convergenceTol convergence tolerance for LBFGS. Smaller value for closer convergence.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      hiddenLayersTopology: Array[Int],
    +      initialWeights: Vector,
    +      maxNumIterations: Int,
    +      convergenceTol: Double): ArtificialNeuralNetworkModel = {
    +    val topology = convertTopology(trainingRDD, hiddenLayersTopology)
    +    new ArtificialNeuralNetwork(topology, maxNumIterations, convergenceTol).
    +      run(trainingRDD, initialWeights)
    +  }
    +
    +  /**
    +   * Provides a random weights vector.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param hiddenLayersTopology number of nodes per hidden layer, excluding the bias nodes.
    +   * @return random weights vector.
    +   */
    +  def randomWeights(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      hiddenLayersTopology: Array[Int]): Vector = {
    +    val topology = convertTopology(trainingRDD, hiddenLayersTopology)
    +    return randomWeights(topology, false)
    +  }
    +
    +  /**
    +   * Provides a random weights vector, using given random seed.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for later training.
    +   * @param hiddenLayersTopology number of nodes per hidden layer, excluding the bias nodes.
    +   * @param seed random generator seed.
    +   * @return random weights vector.
    +   */
    +  def randomWeights(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      hiddenLayersTopology: Array[Int],
    +      seed: Int): Vector = {
    +    val topology = convertTopology(trainingRDD, hiddenLayersTopology)
    +    return randomWeights(topology, true, seed)
    +  }
    +
    +  /**
    +   * Provides a random weights vector, using given random seed.
    +   *
    +   * @param inputLayerSize size of input layer.
    +   * @param outputLayerSize size of output layer.
    +   * @param hiddenLayersTopology number of nodes per hidden layer, excluding the bias nodes.
    +   * @param seed random generator seed.
    +   * @return random weights vector.
    +   */
    +  def randomWeights(
    +                     inputLayerSize: Int,
    --- End diff --
    
    Fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-82779169
  
    Yup and once we have the gradients on edges of the graph, a simple block coordinate descent solver using adagrad should be good start..


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-69405594
  
    I've implemented basic Autoencoder based on this ANN: https://github.com/avulanov/spark/tree/autoencoder. I am thinking of moving it to this branch and will appreciate any comments. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56470085
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20683/consoleFull) for   PR 1290 at commit [`b3531d6`](https://github.com/apache/spark/commit/b3531d68dc36832115fd721a1a2efc0f99851661).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-72754968
  
    There are few PR related to different types of artificial neural networks. It makes sense to come up with common interfaces, reuse the code and make it easily extensible for new related PRs. Currently, we are refactoring the interface of this PR for this purpose. There is an https://issues.apache.org/jira/browse/SPARK-5575 to track ideas.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-54102428
  
    Now updated such that the code supports true back-propagation.
    
    Thanks to Alexander Ulanov (avulanov) for implementing true back-propagation in his repository first. This code borrows extensively from his code, and uses the same back-propagation algorithm (save for using arrays rather than matrices/vectors) and "layers" vector (here called "ontology").
    
    Looking forward to continue our collaboration!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-82510517
  
    @avulanov If we want an auto-encoder that's working at the scale of matrix factorization and LDA, we cannot assume that model fits into worker memory. Even for a decent 10M x 3M auto-encoder and 10K hidden units we need 10M x 10K + 3M x 10K model memory (most likely sparse)...If the model fits into worker memory, the solver fits into master memory and there is neither the need for distributed gradient calculation or coordinate descent based solver...Does @witgo 's branch also aggregates gradients in the same way ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56971412
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20858/consoleFull) for   PR 1290 at commit [`aaf3162`](https://github.com/apache/spark/commit/aaf31627110b379982d74c8882b6bd1491828b0e).
     * This patch **passes** unit tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OutputCanvas2D(wd: Int, ht: Int) extends Canvas `
      * `class OutputFrame2D( title: String ) extends Frame( title ) `
      * `class OutputCanvas3D(wd: Int, ht: Int, shadowFrac: Double) extends Canvas `
      * `class OutputFrame3D(title: String, shadowFrac: Double) extends Frame(title) `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-68169801
  
      [Test build #24841 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24841/consoleFull) for   PR 1290 at commit [`9fb76ba`](https://github.com/apache/spark/commit/9fb76badb0222fbfec6886152477bef76dc2eef8).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-69371908
  
    @jkbradley I totally agree that we need an extensible API for ANN, the problem is that it depends on MLlib API that is not that flexible yet. I wonder if we can bring Spark & MLlib developers to this discussion... Until then our only option is to implement workarounds. Some of my concerns:
       - Widen `Gradient` interface to pass (input, output) instead of `Vector, Double`. Currently we stack input and output into Vector and then unstack it in `Gradient` every time
       - Pluggable optimizers and updaters versus `AlgorithmWithXXX` pattern
       - Matrix-based batch processing of data
       - Asynchronous gradient update 
    
    Actually, my current main concern is scalability. If you want to use MLlib with small datasets such as mnist and shallow networks then you are OK, but you can do it also with plenty of other machine learning libraries. When it comes to bigger data, bigger clusters and even deep neural networks then certain MLlib interfaces become  the bottleneck.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-91413171
  
    @hhbyyh https://github.com/hhbyyh/NeuralNetwork ? let me look into it and I will provide feedback...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67711826
  
      [Test build #550 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/550/consoleFull) for   PR 1290 at commit [`2891e94`](https://github.com/apache/spark/commit/2891e9488330a7538eac0a6ec14a40c8b70b37dd).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OutputCanvas2D(wd: Int, ht: Int) extends Canvas `
      * `class OutputFrame2D( title: String ) extends Frame( title ) `
      * `class OutputCanvas3D(wd: Int, ht: Int, shadowFrac: Double) extends Canvas `
      * `class OutputFrame3D(title: String, shadowFrac: Double) extends Frame(title) `
      * `trait ANNClassifierHelper `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-54901132
  
    @bgreeven Thanks for contributing ANN! I made a quick scan over the code. Some high-level comments:
    
    1. The user guide is for normal users and it should focus on how to use ANN. If we want to leave some notes for developers, we can append a section at the end.
    2. We don't ask users to treat unit tests as demos or examples. Instead, we put a short code snippet in the user guide and put a complete example under `examples/`.
    3. `GeneralizedModel` and `GeneralizedAlgorithm` are definitely out of the scope of this PR and they should not live under `mllib.ann`. We can make a separate JIRA to discuss the APIs. Could you remove them in this PR?
    4. `predict` outputs the prediction for the first node. Would the first node be the only special node? How about having `predict(v)` output the full prediction and `predict(v, i)` output the prediction to the i-th node?
    5. Could you try to use LBFGS instead of GradientDescent?
    6. Please replace for loops by while loops. The latter is faster in Scala.
    7. Please follow the [Spark Code Style Guide](https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide) and update the style, e.g.,
      a. remove space after `(` and before `)`
      b. add ScalaDoc for all public classes and methods
      c. line width should be smaller than 100 chars (in both main and test)
      d. some verification code are left as comments, please find a way to move them to unit tests
      e. organize imports into groups and order them alphabetically within each group
      f. do not add `return` or `;` unless they are necessary
    8. Please use existing unit tests as templates. For example, please rename `TestParallelANN` to `ParallelANNSuite` and use `LocalSparkContext` and `FunSuite` for Spark setup and test assertions. Remove `main()` from unit tests.
    9. Is `Parallel` necessary in the name `ParallelANN`?
    10. What methods and classes are public? Could they be private or package private? Please generate the API doc and check the public ones.
    
    Given the size of the PR, I hope we can split it into multiple PRs. My recommendation would be
    
    1) Remove user guide, `GeneralizedAlgorithm`, and `GeneralizedModel` and focus on the implementation and unit tests.
    2) Add user guide and example code.
    3) Discuss interface for generalized algorithm and model.
    
    I suggest removing the user guide because we may need to go through multiple iterations for the code review of the implementation. It is not necessary to update the user guide with each update.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67919921
  
    Test failure seems unrelated (streaming).
    
    @bgreeven   I hope the Optimizer API can be cleaned up before too long, but I'd estimate it will take at least 1-2 months since I don't think anyone is actively working on it now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [spark-2352] Implementation of an 1-hidden lay...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-47962057
  
    Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-65264353
  
    Just in case, I've tested the ANN without the hidden layer and it seems to work as multinomial regression, though with (one-half) squared-error cost function and without softmax output. One needs to pass an empty array `Array[Int]()` as a hidden layer parameter.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r19719009
  
    --- Diff: docs/mllib-ann.md ---
    @@ -0,0 +1,223 @@
    +---
    +layout: global
    +title: Artificial Neural Networks - MLlib
    +displayTitle: <a href="mllib-guide.html">MLlib</a> - Artificial Neural Networks
    +---
    +
    +# Introduction
    +
    +This document describes the MLlib's Artificial Neural Network (ANN) implementation.
    +
    +The implementation currently consist of the following files:
    +
    +* 'ArtificialNeuralNetwork.scala': implements the ANN
    +* 'ANNSuite': implements automated tests for the ANN and its gradient
    +* 'ANNDemo': a demo that approximates three functions and shows a graphical representation of
    +the result
    +
    +# Summary of usage
    +
    +The "ArtificialNeuralNetwork" object is used as an interface to the neural network. It is
    +called as follows:
    +
    +```
    +val annModel = ArtificialNeuralNetwork.train(rdd, hiddenLayersTopology, maxNumIterations)
    --- End diff --
    
    @manishamde given the size of PR @mengxr suggested to split it into multiple PRs. There is an implementation of a classifier that is based on this artificial neural network https://github.com/avulanov/spark/tree/annclassifier. It employes `RDD[LabeledPoint]` and implements MLlib `Classifier`. Softmax output and cross-entropy error is usually used for better classification performance and they are not yet implemented. We've discussed this issue with @bgreeven and our thinking is to have interface in this PR that allows setting different error function and optimizer. Does it make sense?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67267487
  
    @avulanov @jkbradley 
    
    An advantage of the string is that you can pass it as an opaque from the ANNClassifier class to the ArtificialNeuralNetwork class, i.e. the ANNClassifier class will work with whatever optimiser is defined in the string, without actually knowing about it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-65880449
  
    @jkbradley @manishamde I did performance comparisons with multinomial regression and posted them here: https://github.com/apache/spark/pull/1379#issuecomment-65879536. Suggestions are very welcome.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-62197587
  
    I am considering 1 hidden layer ANN right now...For multiple layers I have not thought on it since to scale it (if you want the model to be distributed) I don't think ALS design fits...We need something through graphx APIs...have to think more...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-72568400
  
      [Test build #26555 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26555/consoleFull) for   PR 1290 at commit [`5de5bad`](https://github.com/apache/spark/commit/5de5badf32081f8bbd73166b705445b0cc37ebdd).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OutputCanvas2D(wd: Int, ht: Int) extends Canvas `
      * `class OutputFrame2D( title: String ) extends Frame( title ) `
      * `class OutputCanvas3D(wd: Int, ht: Int, shadowFrac: Double) extends Canvas `
      * `class OutputFrame3D(title: String, shadowFrac: Double) extends Frame(title) `
      * `trait ANNClassifierHelper `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-68171178
  
      [Test build #24841 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24841/consoleFull) for   PR 1290 at commit [`9fb76ba`](https://github.com/apache/spark/commit/9fb76badb0222fbfec6886152477bef76dc2eef8).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OutputCanvas2D(wd: Int, ht: Int) extends Canvas `
      * `class OutputFrame2D( title: String ) extends Frame( title ) `
      * `class OutputCanvas3D(wd: Int, ht: Int, shadowFrac: Double) extends Canvas `
      * `class OutputFrame3D(title: String, shadowFrac: Double) extends Frame(title) `
      * `trait ANNClassifierHelper `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-82663608
  
     @avulanov I like Jeff Dean's first paper...for that you need to distribute the neural net and do a block based solve...The another one that I saw recently which improves upon collaborative filtering using matrix factorization/plsa is https://www.cs.toronto.edu/~amnih/papers/rbmcf.pdf...Although this is RBM and I want to start with something simple like Jeff's paper (deep neural net using ReLu)...If there is a JIRA please let me know and we can discuss further on that...
    
    Also I agree with the distributed gradient part...we do that in all mllib algorithms...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r19717692
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/ann/ArtificialNeuralNetwork.scala ---
    @@ -0,0 +1,528 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.mllib.ann
    +
    +import breeze.linalg.{DenseVector, Vector => BV, axpy => brzAxpy}
    +
    +import org.apache.spark.mllib.linalg.{Vector, Vectors}
    +import org.apache.spark.mllib.optimization._
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.random.XORShiftRandom
    +
    +/*
    + * Implements a Artificial Neural Network (ANN)
    + *
    + * The data consists of an input vector and an output vector, combined into a single vector
    + * as follows:
    + *
    + * [ ---input--- ---output--- ]
    + *
    + * NOTE: output values should be in the range [0,1]
    + *
    + * For a network of H hidden layers:
    + *
    + * hiddenLayersTopology(h) indicates the number of nodes in hidden layer h, excluding the bias
    + * node. h counts from 0 (first hidden layer, taking inputs from input layer) to H - 1 (last
    + * hidden layer, sending outputs to the output layer).
    + *
    + * hiddenLayersTopology is converted internally to topology, which adds the number of nodes
    + * in the input and output layers.
    + *
    + * noInput = topology(0), the number of input nodes
    + * noOutput = topology(L-1), the number of output nodes
    + *
    + * input = data( 0 to noInput-1 )
    + * output = data( noInput to noInput + noOutput - 1 )
    + *
    + * W_ijl is the weight from node i in layer l-1 to node j in layer l
    + * W_ijl goes to position ofsWeight(l) + j*(topology(l-1)+1) + i in the weights vector
    + *
    + * B_jl is the bias input of node j in layer l
    + * B_jl goes to position ofsWeight(l) + j*(topology(l-1)+1) + topology(l-1) in the weights vector
    + *
    + * error function: E( O, Y ) = sum( O_j - Y_j )
    + * (with O = (O_0, ..., O_(noOutput-1)) the output of the ANN,
    + * and (Y_0, ..., Y_(noOutput-1)) the input)
    + *
    + * node_jl is node j in layer l
    + * node_jl goes to position ofsNode(l) + j
    + *
    + * The weights gradient is defined as dE/dW_ijl and dE/dB_jl
    + * It has same mapping as W_ijl and B_jl
    + *
    + * For back propagation:
    + * delta_jl = dE/dS_jl, where S_jl the output of node_jl, but before applying the sigmoid
    + * delta_jl has the same mapping as node_jl
    + *
    + * Where E = ((estOutput-output),(estOutput-output)),
    + * the inner product of the difference between estimation and target output with itself.
    + *
    + */
    +
    +/**
    + * Artificial neural network (ANN) model
    + *
    + * @param weights the weights between the neurons in the ANN.
    + * @param topology array containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + */
    +class ArtificialNeuralNetworkModel private[mllib](val weights: Vector, val topology: Array[Int])
    +  extends Serializable with ANNHelper {
    +
    +  /**
    +   * Predicts values for a single data point using the trained model.
    +   *
    +   * @param testData represents a single data point.
    +   * @return prediction using the trained model.
    +   */
    +  def predict(testData: Vector): Vector = {
    +    Vectors.dense(computeValues(testData.toArray, weights.toArray))
    +  }
    +
    +  /**
    +   * Predict values for an RDD of data points using the trained model.
    +   *
    +   * @param testDataRDD RDD representing the input vectors.
    +   * @return RDD with predictions using the trained model as (input, output) pairs.
    +   */
    +  def predict(testDataRDD: RDD[Vector]): RDD[(Vector,Vector)] = {
    +    testDataRDD.map(T => (T, predict(T)) )
    +  }
    +
    +  private def computeValues(arrData: Array[Double], arrWeights: Array[Double]): Array[Double] = {
    +    val arrNodes = forwardRun(arrData, arrWeights)
    +    arrNodes.slice(arrNodes.size - topology(L), arrNodes.size)
    +  }
    +}
    +
    +/**
    + * Performs the training of an Artificial Neural Network (ANN)
    + *
    + * @param topology A vector containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + * @param maxNumIterations The maximum number of iterations for the training phase.
    + * @param convergenceTol Convergence tolerance for LBFGS. Smaller value for closer convergence.
    + */
    +class ArtificialNeuralNetwork private[mllib](
    +    topology: Array[Int],
    +    maxNumIterations: Int,
    +    convergenceTol: Double)
    +  extends Serializable {
    +
    +  private val gradient = new ANNLeastSquaresGradient(topology)
    +  private val updater = new ANNUpdater()
    +  private val optimizer = new LBFGS(gradient, updater).
    +    setConvergenceTol(convergenceTol).
    +    setMaxNumIterations(maxNumIterations)
    +
    + /**
    +   * Trains the ANN model.
    +   * Uses default convergence tolerance 1e-4 for LBFGS.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param initialWeights the initial weights of the ANN
    +   * @return ANN model.
    +   */
    +  private def run(trainingRDD: RDD[(Vector, Vector)], initialWeights: Vector):
    +      ArtificialNeuralNetworkModel = {
    +    val data = trainingRDD.map(v =>
    +      (0.0,
    +        Vectors.fromBreeze(DenseVector.vertcat(
    +          v._1.toBreeze.toDenseVector,
    +          v._2.toBreeze.toDenseVector))
    +        ))
    +    val weights = optimizer.optimize(data, initialWeights)
    +    new ArtificialNeuralNetworkModel(weights, topology)
    +  }
    +}
    +
    +/**
    + * Top level methods for training the artificial neural network (ANN)
    + */
    +object ArtificialNeuralNetwork {
    +
    +  private val defaultTolerance: Double = 1e-4
    +
    +  /**
    +   * Trains an ANN.
    +   * Uses default convergence tolerance 1e-4 for LBFGS.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param hiddenLayersTopology number of nodes per hidden layer, excluding the bias nodes.
    +   * @param maxNumIterations specifies maximum number of training iterations.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector, Vector)],
    +      hiddenLayersTopology: Array[Int],
    +      maxNumIterations: Int): ArtificialNeuralNetworkModel = {
    +    train(trainingRDD, hiddenLayersTopology, maxNumIterations, defaultTolerance)
    +  }
    +
    +  /**
    +   * Continues training of an ANN.
    +   * Uses default convergence tolerance 1e-4 for LBFGS.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param model model of an already partly trained ANN.
    +   * @param maxNumIterations maximum number of training iterations.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      model: ArtificialNeuralNetworkModel,
    +      maxNumIterations: Int): ArtificialNeuralNetworkModel = {
    +    train(trainingRDD, model, maxNumIterations, defaultTolerance)
    +  }
    +
    +  /**
    +   * Trains an ANN with given initial weights.
    +   * Uses default convergence tolerance 1e-4 for LBFGS.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param initialWeights initial weights vector.
    +   * @param maxNumIterations maximum number of training iterations.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      hiddenLayersTopology: Array[Int],
    +      initialWeights: Vector,
    +      maxNumIterations: Int): ArtificialNeuralNetworkModel = {
    +    train(trainingRDD, hiddenLayersTopology, initialWeights, maxNumIterations, defaultTolerance)
    +  }
    +
    +  /**
    +   * Trains an ANN using customized convergence tolerance.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param model model of an already partly trained ANN.
    +   * @param maxNumIterations maximum number of training iterations.
    +   * @param convergenceTol convergence tolerance for LBFGS. Smaller value for closer convergence.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      model: ArtificialNeuralNetworkModel,
    +      maxNumIterations: Int,
    +      convergenceTol: Double): ArtificialNeuralNetworkModel = {
    +    new ArtificialNeuralNetwork(model.topology, maxNumIterations, convergenceTol).
    +      run(trainingRDD, model.weights)
    +  }
    +
    +  /**
    +   * Continues training of an ANN using customized convergence tolerance.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param hiddenLayersTopology number of nodes per hidden layer, excluding the bias nodes.
    +   * @param maxNumIterations maximum number of training iterations.
    +   * @param convergenceTol convergence tolerance for LBFGS. Smaller value for closer convergence.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector, Vector)],
    +      hiddenLayersTopology: Array[Int],
    +      maxNumIterations: Int,
    +      convergenceTol: Double): ArtificialNeuralNetworkModel = {
    +    val topology = convertTopology(trainingRDD, hiddenLayersTopology)
    +    new ArtificialNeuralNetwork(topology, maxNumIterations, convergenceTol).
    +      run(trainingRDD, randomWeights(topology, false))
    +  }
    +
    +  /**
    +   * Trains an ANN with given initial weights.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param initialWeights initial weights vector.
    +   * @param maxNumIterations maximum number of training iterations.
    +   * @param convergenceTol convergence tolerance for LBFGS. Smaller value for closer convergence.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      hiddenLayersTopology: Array[Int],
    +      initialWeights: Vector,
    +      maxNumIterations: Int,
    +      convergenceTol: Double): ArtificialNeuralNetworkModel = {
    +    val topology = convertTopology(trainingRDD, hiddenLayersTopology)
    +    new ArtificialNeuralNetwork(topology, maxNumIterations, convergenceTol).
    +      run(trainingRDD, initialWeights)
    +  }
    +
    +  /**
    +   * Provides a random weights vector.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param hiddenLayersTopology number of nodes per hidden layer, excluding the bias nodes.
    +   * @return random weights vector.
    +   */
    +  def randomWeights(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      hiddenLayersTopology: Array[Int]): Vector = {
    +    val topology = convertTopology(trainingRDD, hiddenLayersTopology)
    +    return randomWeights(topology, false)
    +  }
    +
    +  /**
    +   * Provides a random weights vector, using given random seed.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for later training.
    +   * @param hiddenLayersTopology number of nodes per hidden layer, excluding the bias nodes.
    +   * @param seed random generator seed.
    +   * @return random weights vector.
    +   */
    +  def randomWeights(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      hiddenLayersTopology: Array[Int],
    +      seed: Int): Vector = {
    +    val topology = convertTopology(trainingRDD, hiddenLayersTopology)
    +    return randomWeights(topology, true, seed)
    +  }
    +
    +  /**
    +   * Provides a random weights vector, using given random seed.
    +   *
    +   * @param inputLayerSize size of input layer.
    +   * @param outputLayerSize size of output layer.
    +   * @param hiddenLayersTopology number of nodes per hidden layer, excluding the bias nodes.
    +   * @param seed random generator seed.
    +   * @return random weights vector.
    +   */
    +  def randomWeights(
    +                     inputLayerSize: Int,
    +                     outputLayerSize: Int,
    +                     hiddenLayersTopology: Array[Int],
    +                     seed: Int): Vector = {
    +    val topology = inputLayerSize +: hiddenLayersTopology :+ outputLayerSize
    +    return randomWeights(topology, true, seed)
    +  }
    +
    +  private def convertTopology(
    +      input: RDD[(Vector,Vector)],
    +      hiddenLayersTopology: Array[Int] ): Array[Int] = {
    +    val firstElt = input.first
    +    firstElt._1.size +: hiddenLayersTopology :+ firstElt._2.size
    +  }
    +
    +  private def randomWeights(topology: Array[Int], useSeed: Boolean, seed: Int = 0): Vector = {
    +    val rand: XORShiftRandom =
    +      if( useSeed == false ) new XORShiftRandom() else new XORShiftRandom(seed)
    +    var i: Int = 0
    +    var l: Int = 0
    +    val noWeights = {
    +      var tmp = 0
    +      var i = 1
    +      while (i < topology.size) {
    +        tmp = tmp + topology(i) * (topology(i - 1) + 1)
    +        i += 1
    +      }
    +      tmp
    +    }
    +    val initialWeightsArr = new Array[Double](noWeights)
    +    var pos = 0
    +    l = 1
    +    while (l < topology.length) {
    +      i = 0
    +      while (i < (topology(l) * (topology(l - 1) + 1))) {
    +        initialWeightsArr(pos) = (rand.nextDouble * 4.8 - 2.4) / (topology(l - 1) + 1)
    +        pos += 1
    +        i += 1
    +      }
    +      l += 1
    +    }
    +    Vectors.dense(initialWeightsArr)
    +  }
    +}
    +
    +/**
    + * Helper methods for ANN
    + */
    +private[ann] trait ANNHelper {
    --- End diff --
    
    Should be a separate file.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by ZhangBanger <gi...@git.apache.org>.
Github user ZhangBanger commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r21571009
  
    --- Diff: docs/mllib-ann.md ---
    @@ -0,0 +1,239 @@
    +---
    +layout: global
    +title: Artificial Neural Networks - MLlib
    +displayTitle: <a href="mllib-guide.html">MLlib</a> - Artificial Neural Networks
    +---
    +
    +# Introduction
    +
    +This document describes the MLlib's Artificial Neural Network (ANN) implementation.
    +
    +The implementation currently consist of the following files:
    +
    +* 'ArtificialNeuralNetwork.scala': implements the ANN
    +* 'ANNSuite': implements automated tests for the ANN and its gradient
    +* 'ANNDemo': a demo that approximates three functions and shows a graphical representation of
    +the result
    +
    +# Summary of usage
    +
    +The "ArtificialNeuralNetwork" object is used as an interface to the neural network. It is
    +called as follows:
    +
    +```
    +val annModel = ArtificialNeuralNetwork.train(rdd, hiddenLayersTopology, maxNumIterations)
    +```
    +
    +where
    +
    +* `rdd` is an RDD of type (Vector,Vector), the first element containing the input vector and
    +the second the associated output vector.
    +* `hiddenLayersTopology` is an array of integers (Array[Int]), which contains the number of
    +nodes per hidden layer, starting with the layer that takes inputs from the input layer, and
    +finishing with the layer that outputs to the output layer. The bias nodes are not counted.
    +* `maxNumIterations` is an upper bound to the number of iterations to be performed.
    +* `ANNmodel` contains the trained ANN parameters, and can be used to calculated the ANNs
    +approximation to arbitrary input values.
    +
    +The approximations can be calculated as follows:
    +
    +```
    +val v_out = annModel.predict(v_in)
    +```
    +
    +where v_in is either a Vector or an RDD of Vectors, and v_out respectively a Vector or RDD of
    +(Vector,Vector) pairs, corresponding to input and output values.
    +
    +Further details and other calling options will be elaborated upon below.
    +
    +# Architecture and Notation
    +
    +The file ArtificialNeuralNetwork.scala implements the ANN. The following picture shows the
    +architecture of a 3-layer ANN:
    +
    +```
    + +-------+
    + |       |
    + | N_0,0 |
    + |       | 
    + +-------+        +-------+
    +                  |       |
    + +-------+        | N_0,1 |       +-------+
    + |       |        |       |       |       |
    + | N_1,0 |-       +-------+     ->| N_0,2 |
    + |       | \ Wij1              /  |       |
    + +-------+  --    +-------+  --   +-------+
    +               \  |       | / Wjk2
    +     :          ->| N_1,1 |-      +-------+
    +     :            |       |       |       |
    +     :            +-------+       | N_1,2 |
    +     :                            |       |
    +     :                :           +-------+
    +     :                :
    +     :                :                :
    +     :                : 
    +     :                :           +-------+
    +     :                :           |       |
    +     :                :           |N_K-1,2|
    +     :                            |       |
    +     :            +-------+       +-------+
    +     :            |       |
    +     :            |N_J-1,1|
    +                  |       |
    + +-------+        +-------+
    + |       | 
    + |N_I-1,0|  
    + |       |
    + +-------+
    +
    + +-------+        +--------+
    + |       |        |        |
    + |   -1  |        |   -1   |
    + |       |        |        |
    + +-------+        +--------+
    +
    +INPUT LAYER      HIDDEN LAYER    OUTPUT LAYER
    +```
    +
    +The i-th node in layer l is denoted by N_{i,l}, both i and l starting with 0. The weight
    +between node i in layer l-1 and node j in layer l is denoted by Wijl. Layer 0 is the input
    +layer, whereas layer L is the output layer.
    +
    +The ANN also implements bias units. These are nodes that always output the value -1. The bias
    +units are in all layers except the output layer. They act similar to other nodes, but do not
    +have input.
    +
    +The value of node N_{j,l} is calculated  as follows:
    +
    +`$N_{j,l} = g( \sum_{i=0}^{topology_l} W_{i,j,l)*N_{i,l-1} )$`
    +
    +Where g is the sigmoid function
    +
    +`$g(t) = \frac{e^{\beta t} }{1+e^{\beta t}}$`
    +
    +# LBFGS
    +
    +MLlib's ANN implementation uses the LBFGS optimisation algorithm for training. It minimises the
    +following error function:
    +
    +`$E = \sum_{k=0}^{K-1} (N_{k,L} - Y_k)^2$`
    +
    +where Y_k is the target output given inputs N_{0,0} ... N_{I-1,0}.
    +
    +# Implementation Details
    +
    +## The "ArtificialNeuralNetwork" class
    +
    +The "ArtificialNeuralNetwork" class has the following constructor:
    +
    +```
    +class ArtificialNeuralNetwork private(topology: Array[Int], maxNumIterations: Int,
    +convergenceTol: Double)
    +```
    +
    +* `topology` is an array of integers indicating then number of nodes per layer. For example, if
    +"topology" holds (3, 5, 1), it means that there are three input nodes, five nodes in a single
    +hidden layer and 1 output node.
    +* `maxNumIterations` indicates the number of iterations after which the LBFGS algorithm must
    +have stopped.
    +* `convergenceTol` indicates the acceptable error, and if reached the LBFGS algorithm will
    +stop. A lower value of "convergenceTol" will give a higher precision.
    +
    +## The "ArtificialNeuralNetwork" object
    +
    +The object "ArtificialNeuralNetwork" is the interface to the "ArtificialNeuralNetwork" class.
    +The object contains the training function. There are six different instances of the training
    +function, each for use with different parameters. All take as the first parameter the RDD
    +"input", which contains pairs of input and output vectors.
    +
    +In addition, there are three functions for generating random weights. Two take a fixed seed,
    +which is useful for testing if one wants to start with the same weights in every test.
    +
    +* `def train(trainingRDD: RDD[(Vector, Vector)], hiddenLayersTopology: Array[Int], maxNumIterations:
    +Int): ArtificialNeuralNetworkModel`: starts training with random initial weights, and a default
    +convergenceTol=1e-4.
    +* `def train(trainingRDD: RDD[(Vector, Vector)], model: ArtificialNeuralNetworkModel,
    +maxNumIterations: Int): ArtificialNeuralNetworkModel`: resumes training given an earlier
    +calculated model, and a default convergenceTol=1e-4.
    +* `def train(trainingRDD: RDD[(Vector,Vector)], hiddenLayersTopology: Array[Int],
    +initialWeights: Vector, maxNumIterations: Int): ArtificialNeuralNetworkModel`: Trains an ANN
    +with given initial weights, and a default convergenceTol=1e-4.
    +* `def train(trainingRDD: RDD[(Vector, Vector)], hiddenLayersTopology: Array[Int], maxNumIterations:
    +Int, convergenceTol: Double): ArtificialNeuralNetworkModel`: starts training with random
    +initial weights. Allows setting a customised "convergenceTol".
    +* `def train(trainingRDD: RDD[(Vector, Vector)], model: ArtificialNeuralNetworkModel,
    +maxNumIterations: Int, convergenceTol: Double): ArtificialNeuralNetworkModel`: resumes training
    +given an earlier calculated model. Allows setting a customised "convergenceTol".
    +* `def train(trainingRDD: RDD[(Vector,Vector)], hiddenLayersTopology: Array[Int],
    +initialWeights: Vector, maxNumIterations: Int, convergenceTol: Double): 
    +ArtificialNeuralNetworkModel`: Trains an ANN with given initial weights. Allows setting a
    +customised "convergenceTol".
    +* `def randomWeights(trainingRDD: RDD[(Vector,Vector)], hiddenLayersTopology: Array[Int]):
    +Vector`: Generates a random weights vector.
    +*`def randomWeights(trainingRDD: RDD[(Vector,Vector)], hiddenLayersTopology: Array[Int],
    +seed: Int): Vector`: Generates a random weights vector with given seed.
    +*`def randomWeights(inputLayerSize: Int, outputLayerSize: Int, hiddenLayersTopology: Array[Int],
    +seed: Int): Vector`: Generates a random weights vector, using given random seed, input layer
    +size, hidden layers topology and output layer size.
    +
    +Notice that the "hiddenLayersTopology" differs from the "topology" array. The
    +"hiddenLayersTopology" does not include the number of nodes in the input and output layers. The
    +number of nodes in input and output layers is calculated from the first element of the training
    +RDD. For example, the "topology" array (3, 5, 7, 1) would have a "hiddenLayersTopology" (5, 7),
    +the values 3 and 1 are deduced from the training data. The rationale for having these different
    +arrays is that future methods may have a different mapping between input values and input nodes
    +or output values and output nodes.
    +
    +## The "ArtificialNeuralNetworkModel" class
    +
    +All training functions return the trained ANN using the class "ArtificialNeuralNetworkModel".
    +This class has the following function:
    +
    +* `predict(testData: Vector): Vector` calculates the output vector given input vector
    +"testData".
    +* `predict(testData: RDD[Vector]): RDD[(Vector,Vector)]` returns (input, output) vector pairs,
    +using input vector pairs in "testData".
    +
    +The weights used by "predict" come from the model.
    +
    +## Training
    +
    +We have chosen to implement the ANN with LBFGS as optimiser function. We compared it with
    +Statistical Gradient Descent. LBGFS was much faster, but in accordance is also earlier with
    --- End diff --
    
    Is it 'Stochastic' or 'Statistical' Gradient Descent?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-54980593
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20043/consoleFull) for   PR 1290 at commit [`62e915d`](https://github.com/apache/spark/commit/62e915d1442d5f67574d06ea6ca2cb662d60313c).
     * This patch **fails** unit tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-53034921
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19078/consoleFull) for   PR 1290 at commit [`a1b6a7f`](https://github.com/apache/spark/commit/a1b6a7fd56fe4c687d237f560991e228e448a072).
     * This patch **passes** unit tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `The 'ParallelANN' class is the main class of the ANN. This class uses a trait 'ANN', which includes functions for calculating the hidden layer ('computeHidden') and calculation of the output ('computeValues'). The output of 'computeHidden' includes the bias node in the hidden layer, such that it does not need to handle the hidden bias node differently.`
      * `The input of the training function is an RDD with (input/output) training pairs, each input and output being stored as a 'Vector'. The training function returns a variable of from class 'ParallelANNModel', as described below.`
      * `The 'ParallelANN' class implements a Artificial Neural Network (ANN), using the stochastic gradient descent method. It takes as input an RDD of input/output values of type 'Vector', and returns an object of type 'ParallelANNModel' containing the parameters of the trained ANN. The 'ParallelANNModel' object can also be used to calculate results after training.`
      * `abstract class GeneralizedSteepestDescentModel(val weights: Vector )`
      * `trait ANN `
      * `class LeastSquaresGradientANN(`
      * `class ANNUpdater extends Updater `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-70021272
  
    @jkbradley thank you for summarizing the ideas. I've added a link to another PR and written my proposals for API.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by loachli <gi...@git.apache.org>.
Github user loachli commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-70057455
  
    In my opnion, we might do the thing like LogisticRegression. i.e, add new class like ANNClassifierWithLBFGS in ANNClassifier.scala . The goal is to let optimizer be extendable and keep the core algorithm stable.So when other guys want to add one optimizer, the only thing then need to do is and another class, such as ANNClassifierWithXXXX. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-54999561
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20044/consoleFull) for   PR 1290 at commit [`98b427a`](https://github.com/apache/spark/commit/98b427af32359db3d8d04ab40a62dd43c7f1791d).
     * This patch **fails** unit tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by loachli <gi...@git.apache.org>.
Github user loachli commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-69278665
  
    Hi jkbradley:
      Could you tell the jira number  related  to  ”new spark.ml package and its design doc”
    
    
    发件人: jkbradley [mailto:notifications@github.com]
    发送时间: 2015年1月9日 3:51
    收件人: apache/spark
    抄送: Lizhengbing (bing, BIPA)
    主题: Re: [spark] [MLLIB] [spark-2352] Implementation of an Artificial Neural Network (ANN) (#1290)
    
    
    @bgreeven<https://github.com/bgreeven> I’m not too surprised that the majority vote (a.k.a. one vs. all) did not do very well; it does not scale well with the number of classes. A tree (or better yet, error-corrected output codes) generally work better, in my experience.
    
    @avulanov<https://github.com/avulanov> True, we try for consistency with APIs, except where we’re changing the norm. There is not a clear write-up about the “norm,” although the new spark.ml package andHc (in the JIRA) give an overview of some parts. Basically, we’re aiming to make things more pluggable and extensible, while minimizing API change. If that requires short-term API changes (such as switching away from ANNWithX method names), that can be acceptable.
    
    @bgreeven<https://github.com/bgreeven> @avulanov<https://github.com/avulanov> The test results look pretty good, though I’m not sure what to expect for accuracy. I think the main item remaining is figuring out the public API. It’s tough since neural networks / deep learning are a rapidly evolving field, and there are a lot of model & algorithm variants out there. Ideally, we could put together a design doc (to be linked from the JIRA) for this big feature which would:
    
      *   Design a public API for neural networks and deep learning
         *   Comparison of other major libraries’ APIs
         *   Minimum viable product API for an initial PR
         *   Path for the future:
            *   What extensions might we need to do, and can we keep the public API stable for these?
            *   What extensions might users want to do? Is the API easily extensible and/or pluggable, or can we make it so in the future without changing the existing public API?
      *   Briefly discuss the algorithm
         *   Alg sketch, limitations, etc.
         *   Alternative algorithms, and a path for making the optimization algorithm pluggable in the future (as we’ve discussed a bit in the PR conversation)
    
    I realize it takes quite a while to get a big new feature ready. If you’d like to encourage early adoption, you could also post this for now as a package for Spark, while the PR is made fully ready.
    
    CC: @mengxr<https://github.com/mengxr>
    
    —
    Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/1290#issuecomment-69237765>.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-50572425
  
    QA results for PR 1290:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>abstract class GeneralizedSteepestDescendModel(val weights: Vector )<br>trait ANN {<br>class LeastSquaresGradientANN(<br>class ANNUpdater extends Updater {<br>class ParallelANN (<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17412/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67380904
  
    I meant to use the actual Optimizer class, as @Lewuathe suggested.  However, I prefer to use setter methods:
    ```
    val ann = new ArtificialNeuralNetwork()
    ann.setOptimizer(new GradientDescent())
    ann.optimizer.setParam(...)
    ```
    It's easier to maintain binary compatibility by using setters (whenever we add new parameters to the algorithm).  When parameters are passed to train(), every new parameter we add in the future will require a new train() method (since Java & default parameters don't play well together).
    
    If you need to check whether it's an acceptable optimizer for ANN, then I was thinking of a match-case like @bgreeven wrote above.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-53030948
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19078/consoleFull) for   PR 1290 at commit [`a1b6a7f`](https://github.com/apache/spark/commit/a1b6a7fd56fe4c687d237f560991e228e448a072).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-62089562
  
    We can not use existing Gradient classes,Let the whole iterative process is completed in the form of matrix calculation.Moreover We can use the als algorithm design, cut the matrix into the appropriate pieces.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-72568405
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26555/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-57076438
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20930/consoleFull) for   PR 1290 at commit [`804c07a`](https://github.com/apache/spark/commit/804c07a3abd6a0e81d0f04b4a08f88df29cad357).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56790721
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20795/consoleFull) for   PR 1290 at commit [`0db8951`](https://github.com/apache/spark/commit/0db89511aa95fcdf980f9e93b7a14c823eed620f).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-53032802
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19081/consoleFull) for   PR 1290 at commit [`9bb9766`](https://github.com/apache/spark/commit/9bb9766cbfb7128bf2ccaa16b9927916d1680e7b).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-53029642
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19076/consoleFull) for   PR 1290 at commit [`3d79aa6`](https://github.com/apache/spark/commit/3d79aa6a0b5c409dab8e743aa489d5f7418f848f).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-60749010
  
      [Test build #22359 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22359/consoleFull) for   PR 1290 at commit [`6d167c5`](https://github.com/apache/spark/commit/6d167c52184582d58557fd16a462416391d29d82).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-50591157
  
    QA tests have started for PR 1290. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17440/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-53031845
  
    Added documentation


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-54757556
  
    @elyast I am playing with various settings right now, use higher learning rate or more iterations
    I also thought that breeze will be faster but according to our tests gradient computation in breeze is in par with bgreeven implementation, and there is additional overhead of rolling/unrolling matrices to/from weight vector. Moreover, using native BLAS/LAPACK through breeze doesn't speed anything up. Interestingly, this corresponds to graphs in https://github.com/fommil/netlib-java. I will be happy if you prove the opposite. That said, code with breeze is easier to read. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67089450
  
      [Test build #545 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/545/consoleFull) for   PR 1290 at commit [`5e86c5e`](https://github.com/apache/spark/commit/5e86c5edab4c58fee55ddae841f29105f62ceec4).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56957114
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20857/consoleFull) for   PR 1290 at commit [`19d2faa`](https://github.com/apache/spark/commit/19d2faac9047bfe382646bb8c4cd1e5ea2faf2f7).
     * This patch **fails** unit tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OutputCanvas2D(wd: Int, ht: Int) extends Canvas `
      * `class OutputFrame2D( title: String ) extends Frame( title ) `
      * `class OutputCanvas3D(wd: Int, ht: Int, shadowFrac: Double) extends Canvas `
      * `class OutputFrame3D(title: String, shadowFrac: Double) extends Frame(title) `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56473769
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20684/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56473764
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20684/consoleFull) for   PR 1290 at commit [`a28aa4a`](https://github.com/apache/spark/commit/a28aa4a7c91b402c95f81aaad254661cdf06607d).
     * This patch **passes** unit tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OutputCanvas2D(wd: Int, ht: Int) extends Canvas `
      * `class OutputFrame2D( title: String ) extends Frame( title ) `
      * `class OutputCanvas3D(wd: Int, ht: Int, shadowFrac: Double) extends Canvas `
      * `class OutputFrame3D(title: String, shadowFrac: Double) extends Frame(title) `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67265933
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24516/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-61499579
  
    **[Test build #22818 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22818/consoleFull)**     for PR 1290 at commit [`73759d5`](https://github.com/apache/spark/commit/73759d5bc587216c2484e9ebbfaac1fe0ab78bfb)     after a configured wait of `120m`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-62623318
  
    @bgreeven @avulanov About supporting various optimizer, updater, gradient, and error function options, I vote for keeping them as parameters, rather than having different versions of the class with different names.  (The LogisticRegressionWithX pattern is really awkward and not sustainable if we add more optimizers.)  I vote for separating these into (a) optimizer, (b) regularization type (translating to updater internally), and (c) error function (where the gradient should be paired with the error function, not a separate parameter).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-72559891
  
      [Test build #26555 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26555/consoleFull) for   PR 1290 at commit [`5de5bad`](https://github.com/apache/spark/commit/5de5badf32081f8bbd73166b705445b0cc37ebdd).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [spark-2352] Implementation of an 1-hidden lay...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-47962624
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-62247730
  
    Our sparse data very often has ~10M users and 1M features...for 1 hidden layer net with 10K nodes, we need 10M x 10K + 1M x 10K doubles...if the model is not distributed (like ALS design) it's not possible to put such complicated model into memory of one node...having said that it makes sense to have a baseline implementation so that it can be used as a reference for further enhancements...also as a auto encoder, neural nets should generate better results than sparse coding (which we can do in mllib through the PR) due to the non-linearity of hidden units...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r20620263
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/ann/ArtificialNeuralNetwork.scala ---
    @@ -0,0 +1,528 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.mllib.ann
    +
    +import breeze.linalg.{DenseVector, Vector => BV, axpy => brzAxpy}
    +
    +import org.apache.spark.mllib.linalg.{Vector, Vectors}
    +import org.apache.spark.mllib.optimization._
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.random.XORShiftRandom
    +
    +/*
    + * Implements a Artificial Neural Network (ANN)
    + *
    + * The data consists of an input vector and an output vector, combined into a single vector
    + * as follows:
    + *
    + * [ ---input--- ---output--- ]
    + *
    + * NOTE: output values should be in the range [0,1]
    + *
    + * For a network of H hidden layers:
    + *
    + * hiddenLayersTopology(h) indicates the number of nodes in hidden layer h, excluding the bias
    + * node. h counts from 0 (first hidden layer, taking inputs from input layer) to H - 1 (last
    + * hidden layer, sending outputs to the output layer).
    + *
    + * hiddenLayersTopology is converted internally to topology, which adds the number of nodes
    + * in the input and output layers.
    + *
    + * noInput = topology(0), the number of input nodes
    + * noOutput = topology(L-1), the number of output nodes
    + *
    + * input = data( 0 to noInput-1 )
    + * output = data( noInput to noInput + noOutput - 1 )
    + *
    + * W_ijl is the weight from node i in layer l-1 to node j in layer l
    + * W_ijl goes to position ofsWeight(l) + j*(topology(l-1)+1) + i in the weights vector
    + *
    + * B_jl is the bias input of node j in layer l
    + * B_jl goes to position ofsWeight(l) + j*(topology(l-1)+1) + topology(l-1) in the weights vector
    + *
    + * error function: E( O, Y ) = sum( O_j - Y_j )
    + * (with O = (O_0, ..., O_(noOutput-1)) the output of the ANN,
    + * and (Y_0, ..., Y_(noOutput-1)) the input)
    + *
    + * node_jl is node j in layer l
    + * node_jl goes to position ofsNode(l) + j
    + *
    + * The weights gradient is defined as dE/dW_ijl and dE/dB_jl
    + * It has same mapping as W_ijl and B_jl
    + *
    + * For back propagation:
    + * delta_jl = dE/dS_jl, where S_jl the output of node_jl, but before applying the sigmoid
    + * delta_jl has the same mapping as node_jl
    + *
    + * Where E = ((estOutput-output),(estOutput-output)),
    + * the inner product of the difference between estimation and target output with itself.
    + *
    + */
    +
    +/**
    + * Artificial neural network (ANN) model
    + *
    + * @param weights the weights between the neurons in the ANN.
    + * @param topology array containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + */
    +class ArtificialNeuralNetworkModel private[mllib](val weights: Vector, val topology: Array[Int])
    +  extends Serializable with ANNHelper {
    +
    +  /**
    +   * Predicts values for a single data point using the trained model.
    +   *
    +   * @param testData represents a single data point.
    +   * @return prediction using the trained model.
    +   */
    +  def predict(testData: Vector): Vector = {
    +    Vectors.dense(computeValues(testData.toArray, weights.toArray))
    +  }
    +
    +  /**
    +   * Predict values for an RDD of data points using the trained model.
    +   *
    +   * @param testDataRDD RDD representing the input vectors.
    +   * @return RDD with predictions using the trained model as (input, output) pairs.
    +   */
    +  def predict(testDataRDD: RDD[Vector]): RDD[(Vector,Vector)] = {
    +    testDataRDD.map(T => (T, predict(T)) )
    +  }
    +
    +  private def computeValues(arrData: Array[Double], arrWeights: Array[Double]): Array[Double] = {
    +    val arrNodes = forwardRun(arrData, arrWeights)
    +    arrNodes.slice(arrNodes.size - topology(L), arrNodes.size)
    +  }
    +}
    +
    +/**
    + * Performs the training of an Artificial Neural Network (ANN)
    + *
    + * @param topology A vector containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + * @param maxNumIterations The maximum number of iterations for the training phase.
    + * @param convergenceTol Convergence tolerance for LBFGS. Smaller value for closer convergence.
    + */
    +class ArtificialNeuralNetwork private[mllib](
    +    topology: Array[Int],
    +    maxNumIterations: Int,
    +    convergenceTol: Double)
    +  extends Serializable {
    +
    +  private val gradient = new ANNLeastSquaresGradient(topology)
    +  private val updater = new ANNUpdater()
    +  private val optimizer = new LBFGS(gradient, updater).
    --- End diff --
    
    @witgo @jkbradley: We started out using GradientDescent, and replaced it with LBFGS later. Indeed LBFGS is much faster, however, as a result it also shows earlier overfitting.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-62195511
  
    I think I am missing something from your description. In ANN we need to compute K+1 weight matrices: W1(NxH1), W2(H1xH2), ... WK(HKxM), where N is the number of inputs (size of feature space), M is the number of outputs, K is the number of hidden layers, Hi is the size of i-th hidden layer. (Also there are bias vectors but we can forget about them for the sake of simplicity). Could you suggest why do you have only two matrices in your description?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-55349677
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20186/consoleFull) for   PR 1290 at commit [`770ea2a`](https://github.com/apache/spark/commit/770ea2a078575ae8203f11068cc50526d31c1792).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-50851118
  
    QA tests have started for PR 1290. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17649/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-71290964
  
      [Test build #26036 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26036/consoleFull) for   PR 1290 at commit [`d18e9b5`](https://github.com/apache/spark/commit/d18e9b5460019970d5bcbb5a0e816aff5a05bf39).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OutputCanvas2D(wd: Int, ht: Int) extends Canvas `
      * `class OutputFrame2D( title: String ) extends Frame( title ) `
      * `class OutputCanvas3D(wd: Int, ht: Int, shadowFrac: Double) extends Canvas `
      * `class OutputFrame3D(title: String, shadowFrac: Double) extends Frame(title) `
      * `trait ANNClassifierHelper `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-70313952
  
    Below are results for experiments with batch sizes.
    
    First, I tested what kind of improvement the batch makes per backpropagation iteration. I used a single machine with Core i5 2.5GhZ, 8GB, Win64 and tried different batch sizes and versions of matrix multiplication (no native BLAS, reference-win64 and system-win64). The dataset was mnist, 780 features in each vector. The batch matrix size is num_features times batch_size. Graph is log-log plotted.
    
    ![image](https://cloud.githubusercontent.com/assets/939524/5783296/76089c90-9d76-11e4-8719-bb8c589d13a4.png)
    
    
    It turns out that the optimal size is around 100 for any version of multiplication routine. The size of batch matrix for batch size of 100 is 78000 in this experiment. Perhaps, one need to pick batch that forms a matrix of thousands of elements, i.e. ~ 10K / (num_features) or 100K / (num_features). I confirmed this observation by experiment with my other data. Also, if we look at graph for matrix multiplication (dgemm) from here: https://github.com/fommil/netlib-java, we can find that the optimal matrix size is around 10^3 (the end of horizontal line with constant time). It is more or less in line with my results, although Backpropagation is heavier than bgemm test. 
    
    Second, I tested how batch performs on a Linux cluster with reasonable sized data: cluster of 6 machines (Xeon 3.3GHz 4 cores, 16GB RAM) with 12 workers total, mnist8m dataset. I trained ANNClassifier for 40 iterations with no hidden layer, which is 784x10 topology. The batch size and the use of native-system-blas were parameters of my experiment. Resulting error on mnist test was around 9% in all experiments.
    
    ![image](https://cloud.githubusercontent.com/assets/939524/5783415/7c6cb764-9d77-11e4-9972-42c888999671.png)
    
    This graph also suggest that the size of the batch should be 100, or proportional to 100K / (num_features). 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67588076
  
    I think we still need to have two optimizers: SGD and LBFGS, because 1)it will be easier to compare to other methods that use them 2)There should be a choice when one optimizer fails. Also, updater should be a parameter. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67258283
  
    We are discussing with @bgreeven how we can add other optimizers to the ANN. Right now, we have only `ArtificialNeuralNetwork.run` that uses `LBFGS` by default, but we can have `.runSGD` or `.runLBFGS` instead. Then, the question is how to implement the train functions of the Algorithm object. We can use `trainWithSGD` or `trainSGD` etc. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-61481496
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22817/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-61584664
  
      [Test build #22851 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22851/consoleFull) for   PR 1290 at commit [`73759d5`](https://github.com/apache/spark/commit/73759d5bc587216c2484e9ebbfaac1fe0ab78bfb).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-82484515
  
    @debasish83 Sounds interesting!
    1. Gradient calculation is distributed by means of data, i.e. each worker calculates delta gradient on its part of the data. Although, it would be interesting to distribute the model for the cases when it does not fit into the workers memory.
    2. If you have a different solver you can try using it for artificial neural networks by replacing the current one (which is LBFGS)
    3. If you wish to select features, do you plan to use L1 on the bottom layer? In this case you might need to add it in the code by yourself.
    
    This branch is relatively stable, I've done a lot of testing (see earlier posts) so you can experiment with it. There is a separate branch with stacked autoencoder (based on the code from this branch), which I've also tested with a lot:  https://github.com/avulanov/spark/tree/autoencoder
    
    The main goal of the PR you mentioned is DBN and RBM. Our plan with @witgo and @mengxr is to develop a more generic and scalable interface for artificial neural networks and the port existing code. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-64283686
  
    @manishamde Thanks for the useful references! It seems that model parallelization for ANN is a challenging problem. I asked this question to few presenters on the recent AMP CAMP and they confirm this point given that present MLlib interfaces are not very well suited for this task. Moreover, there will be a huge communication overhead during the update step for big models that can still fit into memory. I took a look at the other algorithms rather than back propagation listed in this paper: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=393138&tag=1. A number of models needs to be evaluated in genetic algorithm which even hardens the task. Simulated annealing which is a global optimization routine seems to be more promising. However, with the model distributed across several nodes one needs to copy data points to all nodes that store the model. I suggest to stick with the current implementation until one finds a clear and better approach. Does it make sense?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67712918
  
    I've just updated this branch with matrix form of back-propagation adopted from my old code https://github.com/avulanov/spark/tree/neuralnetwork. It was slower than the code with loops and I thought the problem was with slow parameter roll/unroll. Apparently, the problem was that I did not take advantage of the stride from breeze matrices. It was fixed and now the backpropagation itself is slightly faster (~25% depending on the size of matrices). The complimentary benefit is that the algorithm's code is more readable.
    
    @bgreeven Could you write a brief description to the ANN test called "Gradient of ANN" to let the reader understand more clearly what we are testing?
    
    @bgreeven @jkbradley @Lewuathe I was thinking more about the parameters for the ANN. The main problem for me with the current approach is that one needs to duplicate optimizer's parameters instead of setting them directly and thus one has to implement a Cartesian product of ANN and Optimizer parameters. It would be great to separate ANN parameters from optimization parameters, at least. This does not fit into the current `.train` method pattern. For example, there might be a public constructor for ANN (say any ML model) that accepts a topology and error function. After initialization, one can run `trainWithXXX` method. Or, one can ask the instance to instantiate an optimizer `getOptimizer`, set its parameters directly and then simply run `train()`. 
    
    @mengxr Could you also join our discussion? We will really appreciate your opinion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-64285419
  
    @Lewuathe Ganglia is used to monitor the cluster performance and it shows that cores are busy all the  way and memory underutilized. No resource manager is used.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [spark-2352] Implementation of an 1-hidden lay...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-47962042
  
    @bgreeven Please add `[MLLIB]` to your PR, following https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark . It makes easier for people who want to search MLlib's PRs. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-50572128
  
    @bgreeven Jenkins will be automatically triggered for future updates.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-61480481
  
      [Test build #22818 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22818/consoleFull) for   PR 1290 at commit [`73759d5`](https://github.com/apache/spark/commit/73759d5bc587216c2484e9ebbfaac1fe0ab78bfb).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r19720701
  
    --- Diff: docs/mllib-ann.md ---
    @@ -0,0 +1,223 @@
    +---
    +layout: global
    +title: Artificial Neural Networks - MLlib
    +displayTitle: <a href="mllib-guide.html">MLlib</a> - Artificial Neural Networks
    +---
    +
    +# Introduction
    +
    +This document describes the MLlib's Artificial Neural Network (ANN) implementation.
    +
    +The implementation currently consist of the following files:
    +
    +* 'ArtificialNeuralNetwork.scala': implements the ANN
    +* 'ANNSuite': implements automated tests for the ANN and its gradient
    +* 'ANNDemo': a demo that approximates three functions and shows a graphical representation of
    +the result
    +
    +# Summary of usage
    +
    +The "ArtificialNeuralNetwork" object is used as an interface to the neural network. It is
    +called as follows:
    +
    +```
    +val annModel = ArtificialNeuralNetwork.train(rdd, hiddenLayersTopology, maxNumIterations)
    --- End diff --
    
    @avulanov Thanks for the clarification. Sounds good to me. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-52889153
  
    I have updated the code. Indeed the LeastSquaresGradientANN.compute function was the culprit.
    
    I removed the Breeze instructions, and replaced them by simple Array[Double] instructions. I think especially the removal of taking a Breeze subvector helps.
    
    In addition, there are some values that could be re-used, and don't need re-calculation for each loop in the LeastSquaresGradientANN.compute function. So I have changed the loop order and moved some computations up the loop hierarchy.
    
    There is considerable speed-up. It now works well with a test set size of 256 input, 128 hidden and 26 output nodes (letter classifier).
    
    
    From: Alexander Ulanov [mailto:notifications@github.com]
    Sent: 18 August 2014 16:41
    To: apache/spark
    Cc: Bert Greevenbosch
    Subject: Re: [spark] [MLLIB] [spark-2352] Implementation of an 1-hidden layer Artificial Neural Network (ANN) (#1290)
    
    
    @bgreeven<https://github.com/bgreeven> I've looked at your code and the algorithm seems to be implemented correctly to the best of my knowledge. Probably, copying of the array of weights harms the performance. I played with single threaded implementation of perceptron in Scala and it works fine for my size of data (i.e. around few minutes).
    
    —
    Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/1290#issuecomment-52465553>.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-55900054
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20466/consoleFull) for   PR 1290 at commit [`f1af6cf`](https://github.com/apache/spark/commit/f1af6cfc9c50470c94ce178d8acf80228d496071).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56790021
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20791/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [spark-2352] Implementation of an 1-hidden lay...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-47962626
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16306/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-66454787
  
      [Test build #24310 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24310/consoleFull) for   PR 1290 at commit [`5e86c5e`](https://github.com/apache/spark/commit/5e86c5edab4c58fee55ddae841f29105f62ceec4).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67259839
  
    I'd recommend specifying the optimization algorithm as a parameter to the ANN, rather than having separate train() methods.  That should make it easier for users to swap out optimizers (e.g., if they pass in the optimizer as a string to their script).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56473120
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20683/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56470743
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20684/consoleFull) for   PR 1290 at commit [`a28aa4a`](https://github.com/apache/spark/commit/a28aa4a7c91b402c95f81aaad254661cdf06607d).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-53035766
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19079/consoleFull) for   PR 1290 at commit [`78dde56`](https://github.com/apache/spark/commit/78dde5638c5615ad2d97adfa0df9504481ece51c).
     * This patch **passes** unit tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `The 'ParallelANN' class is the main class of the ANN. This class uses a trait 'ANN', which includes functions for calculating the hidden layer ('computeHidden') and calculation of the output ('computeValues'). The output of 'computeHidden' includes the bias node in the hidden layer, such that it does not need to handle the hidden bias node differently.`
      * `The input of the training function is an RDD with (input/output) training pairs, each input and output being stored as a 'Vector'. The training function returns a variable of from class 'ParallelANNModel', as described below.`
      * `The 'ParallelANN' class implements a Artificial Neural Network (ANN), using the stochastic gradient descent method. It takes as input an RDD of input/output values of type 'Vector', and returns an object of type 'ParallelANNModel' containing the parameters of the trained ANN. The 'ParallelANNModel' object can also be used to calculate results after training.`
      * `abstract class GeneralizedSteepestDescentModel(val weights: Vector )`
      * `trait ANN `
      * `class LeastSquaresGradientANN(`
      * `class ANNUpdater extends Updater `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-83326047
  
    @avulanov thanks for the reference...I am more interested in fully connected layers first...Let's discuss it in few weeks...I will ping...
    I am understanding LDA's graphx implementation...LDA has 2 partitions (document and words) but I am considering 2 partitions which both have features and we add 10K nodes more in this graph (hidden layer)...now the issue is that every data point needs to be passed over this graph but assuming the data is sparse (which is often the case for these models) can we do something ? If the data is not sparse and has 3M columns then I don't think anything will work...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-60853279
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22385/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67087128
  
    @witgo @ankurdave Thank you for suggestion! I merged #3677 into the ANN code and then I run 100 iterations on mnist (60K instances). The running time improved only 1%. Is it expected to be so small or it should be bigger in different settings? Could you elaborate on how the mentioned PR might improve performance or this PR? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67254937
  
    @jkbradley I've rebased the branch, could you run the test again?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by hunggpham <gi...@git.apache.org>.
Github user hunggpham commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-52107969
  
    I finally see the backprop codes in the 2 for loops inside
    LeastSquaresGradientANN
    that calculates the gradient which is then used to update the weights by
    ANNUpdater.
    
    Thanks, Bert.
    
    
    On Tue, Aug 12, 2014 at 8:54 PM, Bert Greevenbosch <notifications@github.com
    > wrote:
    
    > The ANN uses the existent GradientDescent from mllib.optimization for back
    > propagation. It uses the gradient from the new LeastSquaresGradientANN
    > class, and updates using the new ANNUpdater class.
    >
    > This line in ANNUpdater.compute is the backbone of the back propagation:
    >
    > brzAxpy(-thisIterStepSize, gradient.toBreeze, brzWeights)
    >
    > —
    > Reply to this email directly or view it on GitHub
    > <https://github.com/apache/spark/pull/1290#issuecomment-51997639>.
    >


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-50851021
  
    Thanks a lot! I have added the extension now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-82750982
  
    Parameter server + matrix calculations is usually more common.
    The performance of matrix calculations is better.   


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-54101907
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19572/consoleFull) for   PR 1290 at commit [`8d60379`](https://github.com/apache/spark/commit/8d60379153077701dd414762488ead15abe79b9d).
     * This patch **fails** unit tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `abstract class GeneralizedModel(val weights: Vector )`
      * `trait ANN `
      * `class LeastSquaresGradientANN(`
      * `class ANNUpdater extends Updater `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-63401446
  
    @avulanov I'll look around for papers which might allow for comparisons; I'm not sure offhand.
    
    For experiments, I agree with @manishamde may getting at---dividing into 2 types of tests:
    (1) accuracy tests: Here, comparing with single-threaded implementations on small datasets sounds fine.
    (2) scaling tests: By self-speedup tests, I was referring to the scaling tests which @manishamde is mentioning above.  Comparing this ANN implementation with itself to see how it scales in various ways: increasing # examples, # nodes in the model, # machines, etc.  That might let us spot bottlenecks or inefficiencies, even if there aren't good alternate implementations available for comparison.
    
    If I find papers on distributed implementations referencing code available, I'll be sure to post here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-61757505
  
    I think that these 3 parameters should be somehow bound otherwise one can plug a gradient with vector length that does not correspond to the ANN size. We could provide a fabric of correct gradients, or, what is better, to create a `trait ANNGradient` that must be used for any ANN gradient. It should have few functions that allow setting error function for example. However, some of ML algorithms with specific optimization are separate classes in MLlib, such as `SVMWithSGD`. If we follow this route we can create abstract `trait ANN` with vals of optimizer, gradient and updater that have to be initialized somehow in the descendants. We'll have one descendant - `ANNWithLBFGS` - the current implementation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56798296
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20797/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-50572101
  
    Jenkins, add to whitelist.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-82057716
  
    @avulanov could you please point me to a stable branch that I can experiment with..I am focused on collaborative filtering and implemented various matrix factorization formulations (quadratic and nonlinear forms):
    https://issues.apache.org/jira/browse/SPARK-2426
    https://issues.apache.org/jira/browse/SPARK-6323
    Sparse Coding and PLSA are very useful for feature extraction but there are paper where neural nets (autoencoder) have beaten both of them.
    
    I want to start experimenting with the autoencoder variants of neural net. Specifically I will be focused on these 3 aspects:
    
    1. Distributing the neural net gradient calculation (I think you have already distributed the gradient calculation which is exactly what I want). If not we should do it on graphx similar to LDA architecture.
    
    2. A block coordinate descent solver running on Master with BFGS/OWLQN as the inner solver.
    
    3. Use L1 regularization in place of drop out heuristic to automatically select interesting features to make the model sparse


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67912528
  
      [Test build #24718 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24718/consoleFull) for   PR 1290 at commit [`a59f170`](https://github.com/apache/spark/commit/a59f170ac6c671718f9e27ee0db8bbed16f6447a).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OutputCanvas2D(wd: Int, ht: Int) extends Canvas `
      * `class OutputFrame2D( title: String ) extends Frame( title ) `
      * `class OutputCanvas3D(wd: Int, ht: Int, shadowFrac: Double) extends Canvas `
      * `class OutputFrame3D(title: String, shadowFrac: Double) extends Frame(title) `
      * `trait ANNClassifierHelper `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67264237
  
      [Test build #24517 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24517/consoleFull) for   PR 1290 at commit [`2891e94`](https://github.com/apache/spark/commit/2891e9488330a7538eac0a6ec14a40c8b70b37dd).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OutputCanvas2D(wd: Int, ht: Int) extends Canvas `
      * `class OutputFrame2D( title: String ) extends Frame( title ) `
      * `class OutputCanvas3D(wd: Int, ht: Int, shadowFrac: Double) extends Canvas `
      * `class OutputFrame3D(title: String, shadowFrac: Double) extends Frame(title) `
      * `trait ANNClassifierHelper `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-62187026
  
    @witgo Thank you for your suggestion! Could you elaborate how als algorithm design could be used?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r19723239
  
    --- Diff: docs/mllib-ann.md ---
    @@ -0,0 +1,223 @@
    +---
    +layout: global
    +title: Artificial Neural Networks - MLlib
    +displayTitle: <a href="mllib-guide.html">MLlib</a> - Artificial Neural Networks
    +---
    +
    +# Introduction
    +
    +This document describes the MLlib's Artificial Neural Network (ANN) implementation.
    +
    +The implementation currently consist of the following files:
    +
    +* 'ArtificialNeuralNetwork.scala': implements the ANN
    +* 'ANNSuite': implements automated tests for the ANN and its gradient
    +* 'ANNDemo': a demo that approximates three functions and shows a graphical representation of
    +the result
    +
    +# Summary of usage
    +
    +The "ArtificialNeuralNetwork" object is used as an interface to the neural network. It is
    +called as follows:
    +
    +```
    +val annModel = ArtificialNeuralNetwork.train(rdd, hiddenLayersTopology, maxNumIterations)
    --- End diff --
    
    @manishamde: In most cases, one hidden layer is enough. For some special functions two hidden layers are needed. This is a nice text about the choice of number of layers and number of nodes per layer:
    http://www.heatonresearch.com/node/707
    Especially the number of nodes per layer depends heavily on the particular problem.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56789789
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20793/consoleFull) for   PR 1290 at commit [`5b91bba`](https://github.com/apache/spark/commit/5b91bba9cde8f4257d37725fc167dee140e22be9).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56344396
  
    I also needed to change the demo, as the fast convergence doesn't give an interesting converging graph anymore. I moved the demo to the examples directory, but we can consider whether we want to keep it at all.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by tolgap <gi...@git.apache.org>.
Github user tolgap commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-70637578
  
    I gave a shot at trying out the ANN with the MNIST dataset. I find it difficult to get the correct error rate from the output of the predictor.
    
    Given an output `RDD[(Vector, Vector)]`, with both `Vector` being 10 dimensional (0..9), how do I get the error rate out of it? I've tried:
    
    ```Scala
      val errRate  = output.map {
        T =>
          val p = T._2.toArray
          val l = T._1.toArray
    
          (p(0) - l(0)) * (p(0) - l(0)) +
            (p(1) - l(1)) * (p(1) - l(1)) +
            (p(2) - l(2)) * (p(2) - l(2)) +
            (p(3) - l(3)) * (p(3) - l(3)) +
            (p(4) - l(4)) * (p(4) - l(4)) +
            (p(5) - l(5)) * (p(5) - l(5)) +
            (p(6) - l(6)) * (p(6) - l(6)) +
            (p(7) - l(7)) * (p(7) - l(7)) +
            (p(8) - l(8)) * (p(8) - l(8)) +
            (p(9) - l(9)) * (p(9) - l(9))
      }.reduce((u,v) => u + v)
    ```
    
    But this gives me an error rate > `1`, namely `1708.46666263`. I am expecting an error rate between 0 and 1. This same error rate calculation (Squared error) is also used in the `ANNSuite` example.
    
    p.s: I am still researching Machine Learning/Neural Network, so I am not very knowledgable. I would very much appreciate the help.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67275869
  
    The optimizer parameter doesn't have to be a string.  It could be an optimizer type as in GeneralizedLinearAlgorithm.  The optimizer type can store whatever parameters it needs.  This is a more modular setup than separate train() methods or passing strings; it won't require new code in ANN if a new optimizer is added.
    
    If you want to rule out certain optimizers, then doing a match-case on the optimizer type would be reasonable (in a setOptimizer() method in ANN).
    
    How does that sound?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-50853746
  
    QA results for PR 1290:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>abstract class GeneralizedSteepestDescendModel(val weights: Vector )<br>trait ANN {<br>class LeastSquaresGradientANN(<br>class ANNUpdater extends Updater {<br>class ParallelANN (<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17649/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by loachli <gi...@git.apache.org>.
Github user loachli commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67812105
  
    @avulanov: *Could you write a brief description to the ANN test called "Gradient of ANN" to let the reader understand more clearly what we are testing?*
    
    The test ensures that the ANNLeastSquaresGradient is correctly implemented.
    
    It does so by comparing the computation of the gradient in the ANNLeastSquaresGradient class with an approximation of the gradient. If we denote the squared error of the neural network with E(w, x), where w=(w_ijl) is the weights vector, ANNLeastSquaresGradient.compute calculates the gradient dE(w_ijl, x)/dw_ijl. This gradient is subsequently compared with the approximation
    
    (E(w_ijl + eps, x) - E(w_ijl)) / eps
    
    for all weights w_ijl. For eps we chose the value 1e-6.
    
    If the difference is not too big (less than "accept=1e-7"), the ANNLeastSquaresGradient.compute function is supposed to be correctly implemented.
    
    Maybe we should add some comments in the code about this.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-54102197
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19573/consoleFull) for   PR 1290 at commit [`f12165a`](https://github.com/apache/spark/commit/f12165a031b387bf640bb9ab05c106699c88ee88).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-55353216
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20186/consoleFull) for   PR 1290 at commit [`770ea2a`](https://github.com/apache/spark/commit/770ea2a078575ae8203f11068cc50526d31c1792).
     * This patch **passes** unit tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r19717985
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/ann/ArtificialNeuralNetwork.scala ---
    @@ -0,0 +1,528 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.mllib.ann
    +
    +import breeze.linalg.{DenseVector, Vector => BV, axpy => brzAxpy}
    +
    +import org.apache.spark.mllib.linalg.{Vector, Vectors}
    +import org.apache.spark.mllib.optimization._
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.random.XORShiftRandom
    +
    +/*
    + * Implements a Artificial Neural Network (ANN)
    + *
    + * The data consists of an input vector and an output vector, combined into a single vector
    + * as follows:
    + *
    + * [ ---input--- ---output--- ]
    + *
    + * NOTE: output values should be in the range [0,1]
    + *
    + * For a network of H hidden layers:
    + *
    + * hiddenLayersTopology(h) indicates the number of nodes in hidden layer h, excluding the bias
    + * node. h counts from 0 (first hidden layer, taking inputs from input layer) to H - 1 (last
    + * hidden layer, sending outputs to the output layer).
    + *
    + * hiddenLayersTopology is converted internally to topology, which adds the number of nodes
    + * in the input and output layers.
    + *
    + * noInput = topology(0), the number of input nodes
    + * noOutput = topology(L-1), the number of output nodes
    + *
    + * input = data( 0 to noInput-1 )
    + * output = data( noInput to noInput + noOutput - 1 )
    + *
    + * W_ijl is the weight from node i in layer l-1 to node j in layer l
    + * W_ijl goes to position ofsWeight(l) + j*(topology(l-1)+1) + i in the weights vector
    + *
    + * B_jl is the bias input of node j in layer l
    + * B_jl goes to position ofsWeight(l) + j*(topology(l-1)+1) + topology(l-1) in the weights vector
    + *
    + * error function: E( O, Y ) = sum( O_j - Y_j )
    + * (with O = (O_0, ..., O_(noOutput-1)) the output of the ANN,
    + * and (Y_0, ..., Y_(noOutput-1)) the input)
    + *
    + * node_jl is node j in layer l
    + * node_jl goes to position ofsNode(l) + j
    + *
    + * The weights gradient is defined as dE/dW_ijl and dE/dB_jl
    + * It has same mapping as W_ijl and B_jl
    + *
    + * For back propagation:
    + * delta_jl = dE/dS_jl, where S_jl the output of node_jl, but before applying the sigmoid
    + * delta_jl has the same mapping as node_jl
    + *
    + * Where E = ((estOutput-output),(estOutput-output)),
    + * the inner product of the difference between estimation and target output with itself.
    + *
    + */
    +
    +/**
    + * Artificial neural network (ANN) model
    + *
    + * @param weights the weights between the neurons in the ANN.
    + * @param topology array containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + */
    +class ArtificialNeuralNetworkModel private[mllib](val weights: Vector, val topology: Array[Int])
    +  extends Serializable with ANNHelper {
    +
    +  /**
    +   * Predicts values for a single data point using the trained model.
    +   *
    +   * @param testData represents a single data point.
    +   * @return prediction using the trained model.
    +   */
    +  def predict(testData: Vector): Vector = {
    +    Vectors.dense(computeValues(testData.toArray, weights.toArray))
    +  }
    +
    +  /**
    +   * Predict values for an RDD of data points using the trained model.
    +   *
    +   * @param testDataRDD RDD representing the input vectors.
    +   * @return RDD with predictions using the trained model as (input, output) pairs.
    +   */
    +  def predict(testDataRDD: RDD[Vector]): RDD[(Vector,Vector)] = {
    +    testDataRDD.map(T => (T, predict(T)) )
    +  }
    +
    +  private def computeValues(arrData: Array[Double], arrWeights: Array[Double]): Array[Double] = {
    +    val arrNodes = forwardRun(arrData, arrWeights)
    +    arrNodes.slice(arrNodes.size - topology(L), arrNodes.size)
    +  }
    +}
    +
    +/**
    + * Performs the training of an Artificial Neural Network (ANN)
    + *
    + * @param topology A vector containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + * @param maxNumIterations The maximum number of iterations for the training phase.
    + * @param convergenceTol Convergence tolerance for LBFGS. Smaller value for closer convergence.
    + */
    +class ArtificialNeuralNetwork private[mllib](
    +    topology: Array[Int],
    +    maxNumIterations: Int,
    +    convergenceTol: Double)
    +  extends Serializable {
    +
    +  private val gradient = new ANNLeastSquaresGradient(topology)
    +  private val updater = new ANNUpdater()
    +  private val optimizer = new LBFGS(gradient, updater).
    +    setConvergenceTol(convergenceTol).
    +    setMaxNumIterations(maxNumIterations)
    +
    + /**
    +   * Trains the ANN model.
    +   * Uses default convergence tolerance 1e-4 for LBFGS.
    --- End diff --
    
    I think the convergence is parameter in the constructor and you are setting it for the optimizer ```setConvergenceTol(convergenceTol)```. Is this comment still valid?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-82734110
  
    @witgo do you want to use parameter server for gradient calculation as well or you would like to do using graphx (follow LDA architecture ?) 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-63375342
  
    @avulanov Thanks for conducting the experiments. Could you plot graphs for the experiments that you conducted with changing number of features and number of machines. It will be good to understand weak scaling (scaling #machines with the size of the dataset) and strong scaling (fixed size dataset with additional machines machine added for speedup) performance. You could look at [strong scaling](https://github.com/apache/spark/pull/79) experiments that @etrain performed for the first decision tree PR for reference. 
    
    Also, could you compare the accuracy with similar implementation in Python or R?
    
    Finally, Decision trees and random forests support multiclass classification.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-68171181
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24841/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-55911175
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20466/consoleFull) for   PR 1290 at commit [`f1af6cf`](https://github.com/apache/spark/commit/f1af6cfc9c50470c94ce178d8acf80228d496071).
     * This patch **passes** unit tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-50572111
  
    Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-69849072
  
    Added batch processing to forward and back-propagation, i.e. data points are stacked into matrix and then processed in matrix form which can be hardware-accelerated with Blas. This feature should speed-up back-propagation if batch size is chosen properly even if native-Blas is not plugged. I did few experiments and it turns out that for better performance batch should result in 
    
       - Native-system-Blas and Native-reference-blas: matrices of tens thousands elements. E.g. if each data point is 780 features and batch size is 100 then batch matrix will contain 78000 elements
       - No natives, just f2jblas: thousands of elements, e.g. 780*10(batchSize)=7800 elements
    
    These suggestions correlate with graphs from netlib-java: https://github.com/fommil/netlib-java. I will post graphs and performance comparisons on larger scale soon.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by mburke13 <gi...@git.apache.org>.
Github user mburke13 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-49531526
  
    @bgreeven Are you continuing work on this pull request so that it passes all unit tests?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-83776903
  
    yeah if it can be parallelized by data it's best to do that and not do any graphx joins because for graphx the painful thing is to balance the graph and most of the time that step will need more work than rest of the stuff :-(


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-49123127
  
    QA results for PR 1290:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>abstract class GeneralizedSteepestDescendModel(val weights: Vector )<br>trait ANN {<br>class LeastSquaresGradientANN( noInp: Integer, noHid: Integer, noOut: Integer, b: Double ) extends Gradient with ANN {<br>class ANNUpdater extends Updater {<br>class ParallelANN (<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16708/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-52905520
  
    @bgreeven In the meantime I did another implementation of neural networks classifier with arbitrary number of hidden layers. It uses `GradientDescent` to train on `RDD[LabeledPoint]` as other classifiers in mllib, has unit tests and works on my example 2000x500x18 with ~2000 instances. It doesn't have bias units though.  Do you wish that we join our efforts? I can make a pull request into your branch. My implementation is below: https://github.com/avulanov/spark/blob/neuralnetwork/mllib/src/main/scala/org/apache/spark/mllib/classification/NeuralNetwork.scala (branch https://github.com/avulanov/spark/tree/neuralnetwork)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-53033539
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19076/consoleFull) for   PR 1290 at commit [`3d79aa6`](https://github.com/apache/spark/commit/3d79aa6a0b5c409dab8e743aa489d5f7418f848f).
     * This patch **passes** unit tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `The 'ParallelANN' class is the main class of the ANN. This class uses a trait 'ANN', which includes functions for calculating the hidden layer ('computeHidden') and calculation of the output ('computeValues'). The output of 'computeHidden' includes the bias node in the hidden layer, such that it does not need to handle the hidden bias node differently.`
      * `The input of the training function is an RDD with (input/output) training pairs, each input and output being stored as a 'Vector'. The training function returns a variable of from class 'ParallelANNModel', as described below.`
      * `The 'ParallelANN' class implements a Artificial Neural Network (ANN), using the stochastic gradient descent method. It takes as input an RDD of input/output values of type 'Vector', and returns an object of type 'ParallelANNModel' containing the parameters of the trained ANN. The 'ParallelANNModel' object can also be used to calculate results after training.`
      * `abstract class GeneralizedSteepestDescentModel(val weights: Vector )`
      * `trait ANN `
      * `class LeastSquaresGradientANN(`
      * `class ANNUpdater extends Updater `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67429185
  
    You're right that optimization APIs are not great currently.  The changes you suggest (removing private[mllib] and changing the Optimizer APIs) should be done at some point, but they would require a fair amount of design and coding.  A good plan might be:
    * For now, do not expose an optimizer parameter.  Only allow one (LBFGS?).
    * A later PR can improve the Optimizer APIs.
    * A later PR can add an optimizer parameter to ANN.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-84708094
  
    @witgo there are lot of useful building blocks in your RBM PR...are you planning to consolidate them in this PR ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-50436968
  
    Jenkins, retest this please.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67703503
  
      [Test build #550 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/550/consoleFull) for   PR 1290 at commit [`2891e94`](https://github.com/apache/spark/commit/2891e9488330a7538eac0a6ec14a40c8b70b37dd).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-82528305
  
    @debasish83 The current implementation is intended for more typical use of artificial neural network, although it would be interesting task to implement distributed model in general, not only for the sparse case. Are you going to build recommender system with artificial neural networks? Could you please elaborate on this or point me to the relevant papers. 
    
    If model fits into memory there is still a need for distributed gradient if your data is big. You can process each part of your data on a separate worker simultaneously and then do an update. This will not give linear speed-up but still worth doing. That's how it is implemented in MLlib's optimizers and also both in this branch and GuoQiang's.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-54101740
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19572/consoleFull) for   PR 1290 at commit [`8d60379`](https://github.com/apache/spark/commit/8d60379153077701dd414762488ead15abe79b9d).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-54980577
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20043/consoleFull) for   PR 1290 at commit [`62e915d`](https://github.com/apache/spark/commit/62e915d1442d5f67574d06ea6ca2cb662d60313c).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r19717451
  
    --- Diff: docs/mllib-ann.md ---
    @@ -0,0 +1,223 @@
    +---
    +layout: global
    +title: Artificial Neural Networks - MLlib
    +displayTitle: <a href="mllib-guide.html">MLlib</a> - Artificial Neural Networks
    +---
    +
    +# Introduction
    +
    +This document describes the MLlib's Artificial Neural Network (ANN) implementation.
    +
    +The implementation currently consist of the following files:
    +
    +* 'ArtificialNeuralNetwork.scala': implements the ANN
    +* 'ANNSuite': implements automated tests for the ANN and its gradient
    +* 'ANNDemo': a demo that approximates three functions and shows a graphical representation of
    +the result
    +
    +# Summary of usage
    +
    +The "ArtificialNeuralNetwork" object is used as an interface to the neural network. It is
    +called as follows:
    +
    +```
    +val annModel = ArtificialNeuralNetwork.train(rdd, hiddenLayersTopology, maxNumIterations)
    --- End diff --
    
    How common is it to have different number of nodes in the hidden layer? I am wondering whether there should be support for a simpler method ```def train(rdd, numNodesHiddenLayers, maxNumIterations)``` and perhaps even a simpler ```def train(rdd)``` with good default settings to help users get started.
    
    @mengxr @jkbradley Would the upcoming MLlib API feature make this suggestion moot with support for default parameters?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-52282101
  
    @bgreeven I've tried to train `ParallelANNWithSGD` with 3 layers 1000x500x18, numiterations 1000, stepSize 1. My dataset has ~2000 instances, 1000 features, 18 classes. After 17 hours it didn't finish and I killed the Spark process. I think there are some performance issues. I'll try to look at your code but without comments it would be challenging :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-50421747
  
    Hi Matthew,
    
    Sure, I can. I was on holiday during the last two weeks, but now back in office. I'll update the code this week.
    
    Best regards,
    Bert
    
    ________________________________
    Bert Greevenbosch
    Huawei Technologies Co., Ltd.
    
    bert.greevenbosch@huawei.com
    
    Huawei Industrial Base F1-8
    Bantian, Longgang District
    Shenzhen 518129
    P.R. China
    
    http://www.huawei.com
    ________________________________
    本邮件及其附件含有华为公司的保密信息,仅限于发送给上面地址中列出的个人或群组。禁
    止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、或散发)本邮件中
    的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本邮件!
    This e-mail and its attachments contain confidential information from HUAWEI, which
    is intended only for the person or entity whose address is listed above. Any use of the
    information contained herein in any way (including, but not limited to, total or partial
    disclosure, reproduction, or dissemination) by persons other than the intended
    recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by
    phone or email immediately and delete it!
    
    From: Matthew Burke [mailto:notifications@github.com]
    Sent: 20 July 2014 06:46
    To: apache/spark
    Cc: Bert Greevenbosch
    Subject: Re: [spark] [MLLIB] [spark-2352] Implementation of an 1-hidden layer Artificial Neural Network (ANN) (#1290)
    
    
    @bgreeven<https://github.com/bgreeven> Are you continuing work on this pull request so that it passes all unit tests?
    
    —
    Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/1290#issuecomment-49531526>.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67254970
  
      [Test build #24516 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24516/consoleFull) for   PR 1290 at commit [`467885b`](https://github.com/apache/spark/commit/467885bd4dc7b235c508457c7f1c14ef16493578).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by Zehao <gi...@git.apache.org>.
Github user Zehao commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-81390078
  
    Hi, are you still working on this PR? I have a few questions about the latest code from https://github.com/bgreeven/spark .
    
    1. I ran the code and found that the parameter  `gradient` in `ANNUpdater.compute` were all zeros during each iteration , and so it didn't change the weights. Does it matter ? I haven't well understand the difference between `ANNLeastSquaresGradient.compute` and `ANNUpdater.compute` yet ,  but `ANNLeastSquaresGradient.compute`  do change the weights.
    2. In the model , every neuron has a bias node , is it the standard ANN model? 
    
    
    
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-49123100
  
    QA tests have started for PR 1290. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16708/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-70715282
  
    @lovmoy sigmoid


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-71284930
  
      [Test build #26036 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26036/consoleFull) for   PR 1290 at commit [`d18e9b5`](https://github.com/apache/spark/commit/d18e9b5460019970d5bcbb5a0e816aff5a05bf39).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-65731474
  
    I've submitted a feature request to Spark Jira regarding the ANN-based classifier: https://issues.apache.org/jira/browse/SPARK-4752. It is implemented already and I would like to share it with community. However ANN PR is not in the main branch, so I cannot perform a valid PR for the classifier code. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-53037323
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19081/consoleFull) for   PR 1290 at commit [`9bb9766`](https://github.com/apache/spark/commit/9bb9766cbfb7128bf2ccaa16b9927916d1680e7b).
     * This patch **passes** unit tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `The 'ParallelANN' class is the main class of the ANN. This class uses a trait 'ANN', which includes functions for calculating the hidden layer ('computeHidden') and calculation of the output ('computeValues'). The output of 'computeHidden' includes the bias node in the hidden layer, such that it does not need to handle the hidden bias node differently.`
      * `The input of the training function is an RDD with (input/output) training pairs, each input and output being stored as a 'Vector'. The training function returns a variable of from class 'ParallelANNModel', as described below.`
      * `The 'ParallelANN' class implements a Artificial Neural Network (ANN), using the stochastic gradient descent method. It takes as input an RDD of input/output values of type 'Vector', and returns an object of type 'ParallelANNModel' containing the parameters of the trained ANN. The 'ParallelANNModel' object can also be used to calculate results after training.`
      * `abstract class GeneralizedSteepestDescentModel(val weights: Vector )`
      * `trait ANN `
      * `class LeastSquaresGradientANN(`
      * `class ANNUpdater extends Updater `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-62085010
  
    We should use matrix to calculate the forward propagation ,back propagation 
    see http://deeplearning.stanford.edu/wiki/index.php/Neural_Network_Vectorization


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-66876521
  
    @avulanov I will submit a new PR about `AdaDelta` and `AdaGrad` in the next week
    It should be able to use in this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-61436212
  
    @bgreeven Another general suggestion: consider adding logging to the code. It goes a long way in debugging errors and get statuses on long running job. Check DecisionTree/RandomForest/GradientBoosting for example. We need to add more logging to LogisticRegression (object GradientDescent extends Logging but there are no logs there) as well but I guess that's a separate PR.
    
    cc: @mengxr @jkbradley


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by hunggpham <gi...@git.apache.org>.
Github user hunggpham commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-51898209
  
    Yes i forked your repository and can see the codes now.    One question:
    I don't see  backprop codes.    Will that be added later?   Thanks.
    
    Sent from cell phone.   Please excuse typo & brevity
    On Aug 12, 2014 1:34 AM, "Bert Greevenbosch" <no...@github.com>
    wrote:
    
    > SteepestDescend -> SteepestDescent can be changed. Thanks for noticing.
    >
    > Hung Pham, did it work out for you now?
    >
    > —
    > Reply to this email directly or view it on GitHub
    > <https://github.com/apache/spark/pull/1290#issuecomment-51875281>.
    >


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-61749598
  
    Let's discuss a bit more about making the optimiser, updater, gradient, and error function customizable.
    
    Notice that for the current LBFGS algorithm, the error function is used both for the gradient (as the error function is minimized), which is used in the updater and optimizer. Hence for a pluggable error function, the gradient needs to be pluggable.
    
    I think there would be value in making the "updater" and "optimizer" pluggable too. For the optimizers we have already seen the candidates LBFGS and SGD, both with their pres and contras. Also, there may be other optimizers that use something else than the gradient. Since the updater currently depends on the gradient, I suggest to make it pluggable too. (I played around a bit with a genetic optimizer - doesn't work very well but is an example of an optimizer that doesn't use the gradient.)
    
    Maybe we can start with making the "optimizer", "gradient" and "updater" in the ArtificialNeuralNetwork class vars instead of vals. Then we can create a different ANN object for each "optimizer", "gradient" and "updater" combination, e.g. "ArtificialNeuralNetworkWithLBFGS". We also need to remove the convergenceTol from the ArtificialNeuralNetwork constructor, since that is LBFGS specific.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-70024716
  
    @avulanov  Sorry, but I object!  I don't think the name should be changed, even if only 1 optimization algorithm is supported.  I hope support for other optimizers can be added later to the same class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by yb33 <gi...@git.apache.org>.
Github user yb33 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-65342521
  
    Hi guys, I exchanged a couple of emails offline with Alexander. Per his request I will start running some additional benchmark tests on the other data sets I got (i.e. advertising data, etc). My co-worker Girish will also join.
    
    Meanwhile, I have a question: how should the MLLib NN behave when it encounters some missing values in the input data (which is a very typical situation for industry/commercial data, including some of my data sets)?  There are several possibilities, including but not limited to: 1. Quit as soon as any value is missing  2. Ignore the row that has any missing values, but continue to the next rows.  3. Replace missing values with user-specified (or hard-coded, such as zero, for example) defaults and continue.  4. More complex possibilities (to somehow use all data that is not missing, even partial row data).  I suggest option (2) for now, what do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-62783988
  
    @debasish83 I agree, such setup will not fit into the memory of a singe node. Are you talking about recommender system? As far as I know, they are usually addressed by different models and algorithms rather than ANN and back-propagation. Could you suggest other problems with such a big model size that are usually solved with ANN? Probably, the data should have millions of features, such as weather forecasting. Just wondering...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r19717699
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/ann/ArtificialNeuralNetwork.scala ---
    @@ -0,0 +1,528 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.mllib.ann
    +
    +import breeze.linalg.{DenseVector, Vector => BV, axpy => brzAxpy}
    +
    +import org.apache.spark.mllib.linalg.{Vector, Vectors}
    +import org.apache.spark.mllib.optimization._
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.random.XORShiftRandom
    +
    +/*
    + * Implements a Artificial Neural Network (ANN)
    + *
    + * The data consists of an input vector and an output vector, combined into a single vector
    + * as follows:
    + *
    + * [ ---input--- ---output--- ]
    + *
    + * NOTE: output values should be in the range [0,1]
    + *
    + * For a network of H hidden layers:
    + *
    + * hiddenLayersTopology(h) indicates the number of nodes in hidden layer h, excluding the bias
    + * node. h counts from 0 (first hidden layer, taking inputs from input layer) to H - 1 (last
    + * hidden layer, sending outputs to the output layer).
    + *
    + * hiddenLayersTopology is converted internally to topology, which adds the number of nodes
    + * in the input and output layers.
    + *
    + * noInput = topology(0), the number of input nodes
    + * noOutput = topology(L-1), the number of output nodes
    + *
    + * input = data( 0 to noInput-1 )
    + * output = data( noInput to noInput + noOutput - 1 )
    + *
    + * W_ijl is the weight from node i in layer l-1 to node j in layer l
    + * W_ijl goes to position ofsWeight(l) + j*(topology(l-1)+1) + i in the weights vector
    + *
    + * B_jl is the bias input of node j in layer l
    + * B_jl goes to position ofsWeight(l) + j*(topology(l-1)+1) + topology(l-1) in the weights vector
    + *
    + * error function: E( O, Y ) = sum( O_j - Y_j )
    + * (with O = (O_0, ..., O_(noOutput-1)) the output of the ANN,
    + * and (Y_0, ..., Y_(noOutput-1)) the input)
    + *
    + * node_jl is node j in layer l
    + * node_jl goes to position ofsNode(l) + j
    + *
    + * The weights gradient is defined as dE/dW_ijl and dE/dB_jl
    + * It has same mapping as W_ijl and B_jl
    + *
    + * For back propagation:
    + * delta_jl = dE/dS_jl, where S_jl the output of node_jl, but before applying the sigmoid
    + * delta_jl has the same mapping as node_jl
    + *
    + * Where E = ((estOutput-output),(estOutput-output)),
    + * the inner product of the difference between estimation and target output with itself.
    + *
    + */
    +
    +/**
    + * Artificial neural network (ANN) model
    + *
    + * @param weights the weights between the neurons in the ANN.
    + * @param topology array containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + */
    +class ArtificialNeuralNetworkModel private[mllib](val weights: Vector, val topology: Array[Int])
    +  extends Serializable with ANNHelper {
    +
    +  /**
    +   * Predicts values for a single data point using the trained model.
    +   *
    +   * @param testData represents a single data point.
    +   * @return prediction using the trained model.
    +   */
    +  def predict(testData: Vector): Vector = {
    +    Vectors.dense(computeValues(testData.toArray, weights.toArray))
    +  }
    +
    +  /**
    +   * Predict values for an RDD of data points using the trained model.
    +   *
    +   * @param testDataRDD RDD representing the input vectors.
    +   * @return RDD with predictions using the trained model as (input, output) pairs.
    +   */
    +  def predict(testDataRDD: RDD[Vector]): RDD[(Vector,Vector)] = {
    +    testDataRDD.map(T => (T, predict(T)) )
    +  }
    +
    +  private def computeValues(arrData: Array[Double], arrWeights: Array[Double]): Array[Double] = {
    +    val arrNodes = forwardRun(arrData, arrWeights)
    +    arrNodes.slice(arrNodes.size - topology(L), arrNodes.size)
    +  }
    +}
    +
    +/**
    + * Performs the training of an Artificial Neural Network (ANN)
    + *
    + * @param topology A vector containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + * @param maxNumIterations The maximum number of iterations for the training phase.
    + * @param convergenceTol Convergence tolerance for LBFGS. Smaller value for closer convergence.
    + */
    +class ArtificialNeuralNetwork private[mllib](
    +    topology: Array[Int],
    +    maxNumIterations: Int,
    +    convergenceTol: Double)
    +  extends Serializable {
    +
    +  private val gradient = new ANNLeastSquaresGradient(topology)
    +  private val updater = new ANNUpdater()
    +  private val optimizer = new LBFGS(gradient, updater).
    +    setConvergenceTol(convergenceTol).
    +    setMaxNumIterations(maxNumIterations)
    +
    + /**
    +   * Trains the ANN model.
    +   * Uses default convergence tolerance 1e-4 for LBFGS.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param initialWeights the initial weights of the ANN
    +   * @return ANN model.
    +   */
    +  private def run(trainingRDD: RDD[(Vector, Vector)], initialWeights: Vector):
    +      ArtificialNeuralNetworkModel = {
    +    val data = trainingRDD.map(v =>
    +      (0.0,
    +        Vectors.fromBreeze(DenseVector.vertcat(
    +          v._1.toBreeze.toDenseVector,
    +          v._2.toBreeze.toDenseVector))
    +        ))
    +    val weights = optimizer.optimize(data, initialWeights)
    +    new ArtificialNeuralNetworkModel(weights, topology)
    +  }
    +}
    +
    +/**
    + * Top level methods for training the artificial neural network (ANN)
    + */
    +object ArtificialNeuralNetwork {
    +
    +  private val defaultTolerance: Double = 1e-4
    +
    +  /**
    +   * Trains an ANN.
    +   * Uses default convergence tolerance 1e-4 for LBFGS.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param hiddenLayersTopology number of nodes per hidden layer, excluding the bias nodes.
    +   * @param maxNumIterations specifies maximum number of training iterations.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector, Vector)],
    +      hiddenLayersTopology: Array[Int],
    +      maxNumIterations: Int): ArtificialNeuralNetworkModel = {
    +    train(trainingRDD, hiddenLayersTopology, maxNumIterations, defaultTolerance)
    +  }
    +
    +  /**
    +   * Continues training of an ANN.
    +   * Uses default convergence tolerance 1e-4 for LBFGS.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param model model of an already partly trained ANN.
    +   * @param maxNumIterations maximum number of training iterations.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      model: ArtificialNeuralNetworkModel,
    +      maxNumIterations: Int): ArtificialNeuralNetworkModel = {
    +    train(trainingRDD, model, maxNumIterations, defaultTolerance)
    +  }
    +
    +  /**
    +   * Trains an ANN with given initial weights.
    +   * Uses default convergence tolerance 1e-4 for LBFGS.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param initialWeights initial weights vector.
    +   * @param maxNumIterations maximum number of training iterations.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      hiddenLayersTopology: Array[Int],
    +      initialWeights: Vector,
    +      maxNumIterations: Int): ArtificialNeuralNetworkModel = {
    +    train(trainingRDD, hiddenLayersTopology, initialWeights, maxNumIterations, defaultTolerance)
    +  }
    +
    +  /**
    +   * Trains an ANN using customized convergence tolerance.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param model model of an already partly trained ANN.
    +   * @param maxNumIterations maximum number of training iterations.
    +   * @param convergenceTol convergence tolerance for LBFGS. Smaller value for closer convergence.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      model: ArtificialNeuralNetworkModel,
    +      maxNumIterations: Int,
    +      convergenceTol: Double): ArtificialNeuralNetworkModel = {
    +    new ArtificialNeuralNetwork(model.topology, maxNumIterations, convergenceTol).
    +      run(trainingRDD, model.weights)
    +  }
    +
    +  /**
    +   * Continues training of an ANN using customized convergence tolerance.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param hiddenLayersTopology number of nodes per hidden layer, excluding the bias nodes.
    +   * @param maxNumIterations maximum number of training iterations.
    +   * @param convergenceTol convergence tolerance for LBFGS. Smaller value for closer convergence.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector, Vector)],
    +      hiddenLayersTopology: Array[Int],
    +      maxNumIterations: Int,
    +      convergenceTol: Double): ArtificialNeuralNetworkModel = {
    +    val topology = convertTopology(trainingRDD, hiddenLayersTopology)
    +    new ArtificialNeuralNetwork(topology, maxNumIterations, convergenceTol).
    +      run(trainingRDD, randomWeights(topology, false))
    +  }
    +
    +  /**
    +   * Trains an ANN with given initial weights.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param initialWeights initial weights vector.
    +   * @param maxNumIterations maximum number of training iterations.
    +   * @param convergenceTol convergence tolerance for LBFGS. Smaller value for closer convergence.
    +   * @return ANN model.
    +   */
    +  def train(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      hiddenLayersTopology: Array[Int],
    +      initialWeights: Vector,
    +      maxNumIterations: Int,
    +      convergenceTol: Double): ArtificialNeuralNetworkModel = {
    +    val topology = convertTopology(trainingRDD, hiddenLayersTopology)
    +    new ArtificialNeuralNetwork(topology, maxNumIterations, convergenceTol).
    +      run(trainingRDD, initialWeights)
    +  }
    +
    +  /**
    +   * Provides a random weights vector.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for training.
    +   * @param hiddenLayersTopology number of nodes per hidden layer, excluding the bias nodes.
    +   * @return random weights vector.
    +   */
    +  def randomWeights(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      hiddenLayersTopology: Array[Int]): Vector = {
    +    val topology = convertTopology(trainingRDD, hiddenLayersTopology)
    +    return randomWeights(topology, false)
    +  }
    +
    +  /**
    +   * Provides a random weights vector, using given random seed.
    +   *
    +   * @param trainingRDD RDD containing (input, output) pairs for later training.
    +   * @param hiddenLayersTopology number of nodes per hidden layer, excluding the bias nodes.
    +   * @param seed random generator seed.
    +   * @return random weights vector.
    +   */
    +  def randomWeights(
    +      trainingRDD: RDD[(Vector,Vector)],
    +      hiddenLayersTopology: Array[Int],
    +      seed: Int): Vector = {
    +    val topology = convertTopology(trainingRDD, hiddenLayersTopology)
    +    return randomWeights(topology, true, seed)
    +  }
    +
    +  /**
    +   * Provides a random weights vector, using given random seed.
    +   *
    +   * @param inputLayerSize size of input layer.
    +   * @param outputLayerSize size of output layer.
    +   * @param hiddenLayersTopology number of nodes per hidden layer, excluding the bias nodes.
    +   * @param seed random generator seed.
    +   * @return random weights vector.
    +   */
    +  def randomWeights(
    +                     inputLayerSize: Int,
    +                     outputLayerSize: Int,
    +                     hiddenLayersTopology: Array[Int],
    +                     seed: Int): Vector = {
    +    val topology = inputLayerSize +: hiddenLayersTopology :+ outputLayerSize
    +    return randomWeights(topology, true, seed)
    +  }
    +
    +  private def convertTopology(
    +      input: RDD[(Vector,Vector)],
    +      hiddenLayersTopology: Array[Int] ): Array[Int] = {
    +    val firstElt = input.first
    +    firstElt._1.size +: hiddenLayersTopology :+ firstElt._2.size
    +  }
    +
    +  private def randomWeights(topology: Array[Int], useSeed: Boolean, seed: Int = 0): Vector = {
    +    val rand: XORShiftRandom =
    +      if( useSeed == false ) new XORShiftRandom() else new XORShiftRandom(seed)
    +    var i: Int = 0
    +    var l: Int = 0
    +    val noWeights = {
    +      var tmp = 0
    +      var i = 1
    +      while (i < topology.size) {
    +        tmp = tmp + topology(i) * (topology(i - 1) + 1)
    +        i += 1
    +      }
    +      tmp
    +    }
    +    val initialWeightsArr = new Array[Double](noWeights)
    +    var pos = 0
    +    l = 1
    +    while (l < topology.length) {
    +      i = 0
    +      while (i < (topology(l) * (topology(l - 1) + 1))) {
    +        initialWeightsArr(pos) = (rand.nextDouble * 4.8 - 2.4) / (topology(l - 1) + 1)
    +        pos += 1
    +        i += 1
    +      }
    +      l += 1
    +    }
    +    Vectors.dense(initialWeightsArr)
    +  }
    +}
    +
    +/**
    + * Helper methods for ANN
    + */
    +private[ann] trait ANNHelper {
    +  protected val topology: Array[Int]
    +  protected def g(x: Double) = 1.0 / (1.0 + math.exp(-x))
    +  protected val L = topology.length - 1
    +  protected val noWeights = {
    +    var tmp = 0
    +    var l = 1
    +    while (l <= L) {
    +      tmp = tmp + topology(l) * (topology(l - 1) + 1)
    +      l += 1
    +    }
    +    tmp
    +  }
    +  protected val ofsWeight: Array[Int] = {
    +    val tmp = new Array[Int](L + 1)
    +    var curPos = 0
    +    tmp(0) = 0
    +    var l = 1
    +    while (l <= L) {
    +      tmp(l) = curPos
    +      curPos = curPos + (topology(l - 1) + 1) * topology(l)
    +      l += 1
    +    }
    +    tmp
    +  }
    +  protected val noNodes: Int = {
    +    var tmp: Integer = 0
    +    var l = 0
    +    while (l < topology.size) {
    +      tmp = tmp + topology(l)
    +      l += 1
    +    }
    +    tmp
    +  }
    +  protected val ofsNode: Array[Int] = {
    +    val tmp = new Array[Int](L + 1)
    +    tmp(0) = 0
    +    var l = 1
    +    while (l <= L) {
    +      tmp(l) = tmp(l - 1) + topology(l - 1)
    +      l += 1
    +    }
    +    tmp
    +  }
    +
    +  protected def forwardRun(arrData: Array[Double], arrWeights: Array[Double]): Array[Double] = {
    +    val arrNodes = new Array[Double](noNodes)
    +    var i: Int = 0
    +    var j: Int = 0
    +    var l: Int = 0
    +    i = 0
    +    while (i < topology(0)) {
    +      arrNodes(i) = arrData(i)
    +      i += 1
    +    }
    +    l = 1
    +    while (l <= L) {
    +      j = 0
    +      while (j < topology(l)) {
    +        var cum: Double = 0.0
    +        i = 0
    +        while (i < topology(l - 1)) {
    +          cum = cum +
    +            arrWeights(ofsWeight(l) + (topology(l - 1) + 1) * j + i) *
    +              arrNodes(ofsNode(l - 1) + i)
    +          i += 1
    +        }
    +        cum = cum + arrWeights(ofsWeight(l) + (topology(l - 1) + 1) * j + topology(l - 1))
    +        arrNodes(ofsNode(l) + j) = g(cum)
    +        j += 1
    +      }
    +      l += 1
    +    }
    +    arrNodes
    +  }
    +}
    +
    +private class ANNLeastSquaresGradient(val topology: Array[Int]) extends Gradient with ANNHelper {
    --- End diff --
    
    Best to add to a separate file.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-50863593
  
    QA results for PR 1290:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>abstract class GeneralizedSteepestDescendModel(val weights: Vector )<br>trait ANN {<br>class LeastSquaresGradientANN(<br>class ANNUpdater extends Updater {<br>class ParallelANN (<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17665/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-54990826
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20044/consoleFull) for   PR 1290 at commit [`98b427a`](https://github.com/apache/spark/commit/98b427af32359db3d8d04ab40a62dd43c7f1791d).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67721910
  
      [Test build #24660 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24660/consoleFull) for   PR 1290 at commit [`1d0ba3e`](https://github.com/apache/spark/commit/1d0ba3e3a0a43c8aa664657ac325d9a8a5d81227).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OutputCanvas2D(wd: Int, ht: Int) extends Canvas `
      * `class OutputFrame2D( title: String ) extends Frame( title ) `
      * `class OutputCanvas3D(wd: Int, ht: Int, shadowFrac: Double) extends Canvas `
      * `class OutputFrame3D(title: String, shadowFrac: Double) extends Frame(title) `
      * `trait ANNClassifierHelper `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-53036266
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19080/consoleFull) for   PR 1290 at commit [`95a88c6`](https://github.com/apache/spark/commit/95a88c660e7c67894b8b86c11017a1da480a9182).
     * This patch **passes** unit tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `The 'ParallelANN' class is the main class of the ANN. This class uses a trait 'ANN', which includes functions for calculating the hidden layer ('computeHidden') and calculation of the output ('computeValues'). The output of 'computeHidden' includes the bias node in the hidden layer, such that it does not need to handle the hidden bias node differently.`
      * `The input of the training function is an RDD with (input/output) training pairs, each input and output being stored as a 'Vector'. The training function returns a variable of from class 'ParallelANNModel', as described below.`
      * `The 'ParallelANN' class implements a Artificial Neural Network (ANN), using the stochastic gradient descent method. It takes as input an RDD of input/output values of type 'Vector', and returns an object of type 'ParallelANNModel' containing the parameters of the trained ANN. The 'ParallelANNModel' object can also be used to calculate results after training.`
      * `In multiclass classification, all `$2^`
      * `public final class JavaDecisionTree `
      * `abstract class GeneralizedSteepestDescentModel(val weights: Vector )`
      * `trait ANN `
      * `class LeastSquaresGradientANN(`
      * `class ANNUpdater extends Updater `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-82070543
  
    Also how is this https://github.com/apache/spark/pull/3222 different ? I am confused for autoencoder which one is a better start...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-71290975
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26036/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56952211
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20857/consoleFull) for   PR 1290 at commit [`19d2faa`](https://github.com/apache/spark/commit/19d2faac9047bfe382646bb8c4cd1e5ea2faf2f7).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-64285217
  
    @jkbradley Thank you for good explanation! As for (1), single threaded ANN implemented in C++ shows similar accuracy. (2) seems like some additional coding needs to be done. I'll plan to work on it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-66461896
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24310/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-55147473
  
    test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-70717896
  
    @tolgap Mean squared error from `ANNSuite` is not an error rate. In mnist case, I guess you are looking for classification error, or (1 - accuracy). You need to convert the output of the network to labels/classes and compare them to the actual labels/classes. Number of misses divided by the total number of samples will be the classification error. Alternatively, you might want to use `ANNClassifier` and `MulticlassMetrics`, for example: https://github.com/apache/spark/pull/1155#issuecomment-70714725 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-85244508
  
    @witgo Thanks for the links! I hope soon we'll be able to merge it with the basic interface/implementation from https://github.com/avulanov/spark/tree/ann-interface (ann-interface-gemm, ann-dropout). I am finishing performance testing right now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-55195691
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/37/consoleFull) for   PR 1290 at commit [`770ea2a`](https://github.com/apache/spark/commit/770ea2a078575ae8203f11068cc50526d31c1792).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56796798
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20793/consoleFull) for   PR 1290 at commit [`5b91bba`](https://github.com/apache/spark/commit/5b91bba9cde8f4257d37725fc167dee140e22be9).
     * This patch **passes** unit tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OutputCanvas2D(wd: Int, ht: Int) extends Canvas `
      * `class OutputFrame2D( title: String ) extends Frame( title ) `
      * `class OutputCanvas3D(wd: Int, ht: Int, shadowFrac: Double) extends Canvas `
      * `class OutputFrame3D(title: String, shadowFrac: Double) extends Frame(title) `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r20591105
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/ann/ArtificialNeuralNetwork.scala ---
    @@ -0,0 +1,528 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.mllib.ann
    +
    +import breeze.linalg.{DenseVector, Vector => BV, axpy => brzAxpy}
    +
    +import org.apache.spark.mllib.linalg.{Vector, Vectors}
    +import org.apache.spark.mllib.optimization._
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.random.XORShiftRandom
    +
    +/*
    + * Implements a Artificial Neural Network (ANN)
    + *
    + * The data consists of an input vector and an output vector, combined into a single vector
    + * as follows:
    + *
    + * [ ---input--- ---output--- ]
    + *
    + * NOTE: output values should be in the range [0,1]
    + *
    + * For a network of H hidden layers:
    + *
    + * hiddenLayersTopology(h) indicates the number of nodes in hidden layer h, excluding the bias
    + * node. h counts from 0 (first hidden layer, taking inputs from input layer) to H - 1 (last
    + * hidden layer, sending outputs to the output layer).
    + *
    + * hiddenLayersTopology is converted internally to topology, which adds the number of nodes
    + * in the input and output layers.
    + *
    + * noInput = topology(0), the number of input nodes
    + * noOutput = topology(L-1), the number of output nodes
    + *
    + * input = data( 0 to noInput-1 )
    + * output = data( noInput to noInput + noOutput - 1 )
    + *
    + * W_ijl is the weight from node i in layer l-1 to node j in layer l
    + * W_ijl goes to position ofsWeight(l) + j*(topology(l-1)+1) + i in the weights vector
    + *
    + * B_jl is the bias input of node j in layer l
    + * B_jl goes to position ofsWeight(l) + j*(topology(l-1)+1) + topology(l-1) in the weights vector
    + *
    + * error function: E( O, Y ) = sum( O_j - Y_j )
    + * (with O = (O_0, ..., O_(noOutput-1)) the output of the ANN,
    + * and (Y_0, ..., Y_(noOutput-1)) the input)
    + *
    + * node_jl is node j in layer l
    + * node_jl goes to position ofsNode(l) + j
    + *
    + * The weights gradient is defined as dE/dW_ijl and dE/dB_jl
    + * It has same mapping as W_ijl and B_jl
    + *
    + * For back propagation:
    + * delta_jl = dE/dS_jl, where S_jl the output of node_jl, but before applying the sigmoid
    + * delta_jl has the same mapping as node_jl
    + *
    + * Where E = ((estOutput-output),(estOutput-output)),
    + * the inner product of the difference between estimation and target output with itself.
    + *
    + */
    +
    +/**
    + * Artificial neural network (ANN) model
    + *
    + * @param weights the weights between the neurons in the ANN.
    + * @param topology array containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + */
    +class ArtificialNeuralNetworkModel private[mllib](val weights: Vector, val topology: Array[Int])
    +  extends Serializable with ANNHelper {
    +
    +  /**
    +   * Predicts values for a single data point using the trained model.
    +   *
    +   * @param testData represents a single data point.
    +   * @return prediction using the trained model.
    +   */
    +  def predict(testData: Vector): Vector = {
    +    Vectors.dense(computeValues(testData.toArray, weights.toArray))
    +  }
    +
    +  /**
    +   * Predict values for an RDD of data points using the trained model.
    +   *
    +   * @param testDataRDD RDD representing the input vectors.
    +   * @return RDD with predictions using the trained model as (input, output) pairs.
    +   */
    +  def predict(testDataRDD: RDD[Vector]): RDD[(Vector,Vector)] = {
    +    testDataRDD.map(T => (T, predict(T)) )
    +  }
    +
    +  private def computeValues(arrData: Array[Double], arrWeights: Array[Double]): Array[Double] = {
    +    val arrNodes = forwardRun(arrData, arrWeights)
    +    arrNodes.slice(arrNodes.size - topology(L), arrNodes.size)
    +  }
    +}
    +
    +/**
    + * Performs the training of an Artificial Neural Network (ANN)
    + *
    + * @param topology A vector containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + * @param maxNumIterations The maximum number of iterations for the training phase.
    + * @param convergenceTol Convergence tolerance for LBFGS. Smaller value for closer convergence.
    + */
    +class ArtificialNeuralNetwork private[mllib](
    +    topology: Array[Int],
    +    maxNumIterations: Int,
    +    convergenceTol: Double)
    +  extends Serializable {
    +
    +  private val gradient = new ANNLeastSquaresGradient(topology)
    +  private val updater = new ANNUpdater()
    +  private val optimizer = new LBFGS(gradient, updater).
    --- End diff --
    
    I suspect LBFGS will converge faster.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by elyast <gi...@git.apache.org>.
Github user elyast commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-54734035
  
    Hi @avulanov I run your NeuralNetworkSuite (from your fork / neuralnetwork brach), however it fails randomly, are you sure you have implemented in correctly?
    
    By the way guys did you thought about regularization? 
    
    @bgreeven I checked your though and I see a lots of iterations I would recommend to replace it with breeze vector operations since it will be much faster (its true also for matlab or R code).
    
    Let me know what do you think and if I can help with anything.
    
    Last thing to consider is to factor out activation function, so instead sigmoid one can use softmax, tan or else.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67265931
  
    **[Test build #24516 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24516/consoleFull)**     for PR 1290 at commit [`467885b`](https://github.com/apache/spark/commit/467885bd4dc7b235c508457c7f1c14ef16493578)     after a configured wait of `120m`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-83760684
  
    @debasish83 Sounds interesting! I think this might also work with non-fully connected networks, though they can be parallilized by data as you mentioned. You might also want to consider using BIDMat since it has optimizations for computations on sparse data.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-52465553
  
    @bgreeven I've looked at your code and the algorithm seems to be implemented correctly to the best of my knowledge. Probably, copying of the array of weights harms the performance. I played with single threaded implementation of perceptron in Scala and it works fine for my size of data (i.e. around few minutes).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-63375573
  
    I found this reference recently about Netflix's distributed implementation of neural nets that could be relevant for MLlib. http://techblog.netflix.com/2014/02/distributed-neural-networks-with-gpus.html


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-70303487
  
    Eventually, we will have more optimizers in MLlib, and it does not make sense to have (# models) x (# optimizers) classes.  It would be *a lot* of duplicated code.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-69848450
  
      [Test build #25491 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25491/consoleFull) for   PR 1290 at commit [`4b8d297`](https://github.com/apache/spark/commit/4b8d2978667b7b1d0dad37f0dbf36c7ac2935af9).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by lovmoy <gi...@git.apache.org>.
Github user lovmoy commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-70615067
  
    Hi, @bgreeven  What is the active function of the hidden layer?  Thank you.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-62190595
  
    For matrix factorization we have user x product sparse matrix...You can think of this sparse matrix as the feature matrix for ANN...Now consider two matrices H1 and H2 of size feature x rank...where rank is the number of hidden layers...With this the problem is minimize || X - f(H1'X)H2 || + lambdaL1(H1) + lambdaL2(H2)
    
    The major difference is can H1'X breaks the way matrix factorization breaks ? If it can then we should be able to use ALS design...or an extension of ALS design...
    
    But say the hidden layer grows from 1 to 10 (Latest Google paper mentioned 22 layers)...then I don't think this idea works...we have to formulate the problem on graphx where the model is distributed over workers and not built on Master  
    
    @witgo you think we can break f(H1'X) in ALS way? I have not thought more on it !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-50848722
  
    @bgreeven The filename `mllib/src/main/scala/org/apache/spark/mllib/ann/GeneralizedSteepestDescendAlgorithm` doesn't have `.scala` extension.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-61433892
  
    @bgreeven I haven't studied the implementation details yet but I had a question about the API. I realize that RDD[(Vector, Vector)] is a more general structure for training data but it might be a good idea in order to provide support for RDD[LabeledPoint] (with conversion to internal format of RDD[(Vector, Vector)] as well so that ANN can be "plugged" in place of other algorithms in the machine learning pipeline for single label classification and regression uses cases.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56791275
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20797/consoleFull) for   PR 1290 at commit [`d4a692c`](https://github.com/apache/spark/commit/d4a692c2decb044115424561b5618a112f54c2f8).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-84750038
  
    The latest code in the branch [Sent2vec (WIP)](https://github.com/witgo/spark/tree/Sent2vec).
    This branch includes two new classes
     [Sentence2vec](https://github.com/witgo/spark/blob/Sent2vec/mllib/src/main/scala/org/apache/spark/mllib/feature/Sentence2vec.scala) A sentence classification with convolutional-pooling structure
    [SentenceClassifier](https://github.com/witgo/spark/blob/Sent2vec/mllib/src/main/scala/org/apache/spark/mllib/feature/SentenceClassifier.scala) A latent semantic model with convolutional-pooling structure



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67087246
  
    @jkbradley Thank you!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67264242
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24517/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67721911
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24660/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67716558
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24658/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [spark-2352] Implementation of an 1-hidden lay...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-47962391
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [spark-2352] Implementation of an 1-hidden lay...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-47864843
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-62824356
  
    @avulanov I meant first sparse auto encoder with 1 layer and followed by a deep net...these huge nets are mostly for automated feature extraction...of course we can do sparse coding using ALS design but I am not sure if  X - W'H formulation in ALS is as effective as X - f(XW)H even if we pass W'H from ALS through some non-linear function...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67710620
  
      [Test build #24658 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24658/consoleFull) for   PR 1290 at commit [`ed6a7c6`](https://github.com/apache/spark/commit/ed6a7c67ad5393a28e45ef3cb3141f21ff53cd93).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by Lewuathe <gi...@git.apache.org>.
Github user Lewuathe commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67315423
  
    @bgreeven I think @jkbradley means that it might be better to pass `Optimizer` itself to train method or set method. For example.
    
    ```scala
    def train(..., optimizer: Optimizer): ArtificalNeuralNetworkModel = {
        optimizer match {
          case GradientDescent(...) =>
             // Perform SGD
          case LBFGS(...) =>
             // Perform LBFGS
        }
    } 
    ```
    
    Users can select any optimizers and configure appropriate parameters.
    
    @jkbradley I'm sorry if it is not what you mean. But this is my proposal anyway. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r20589805
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/ann/ArtificialNeuralNetwork.scala ---
    @@ -0,0 +1,528 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.mllib.ann
    +
    +import breeze.linalg.{DenseVector, Vector => BV, axpy => brzAxpy}
    +
    +import org.apache.spark.mllib.linalg.{Vector, Vectors}
    +import org.apache.spark.mllib.optimization._
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.random.XORShiftRandom
    +
    +/*
    + * Implements a Artificial Neural Network (ANN)
    + *
    + * The data consists of an input vector and an output vector, combined into a single vector
    + * as follows:
    + *
    + * [ ---input--- ---output--- ]
    + *
    + * NOTE: output values should be in the range [0,1]
    + *
    + * For a network of H hidden layers:
    + *
    + * hiddenLayersTopology(h) indicates the number of nodes in hidden layer h, excluding the bias
    + * node. h counts from 0 (first hidden layer, taking inputs from input layer) to H - 1 (last
    + * hidden layer, sending outputs to the output layer).
    + *
    + * hiddenLayersTopology is converted internally to topology, which adds the number of nodes
    + * in the input and output layers.
    + *
    + * noInput = topology(0), the number of input nodes
    + * noOutput = topology(L-1), the number of output nodes
    + *
    + * input = data( 0 to noInput-1 )
    + * output = data( noInput to noInput + noOutput - 1 )
    + *
    + * W_ijl is the weight from node i in layer l-1 to node j in layer l
    + * W_ijl goes to position ofsWeight(l) + j*(topology(l-1)+1) + i in the weights vector
    + *
    + * B_jl is the bias input of node j in layer l
    + * B_jl goes to position ofsWeight(l) + j*(topology(l-1)+1) + topology(l-1) in the weights vector
    + *
    + * error function: E( O, Y ) = sum( O_j - Y_j )
    + * (with O = (O_0, ..., O_(noOutput-1)) the output of the ANN,
    + * and (Y_0, ..., Y_(noOutput-1)) the input)
    + *
    + * node_jl is node j in layer l
    + * node_jl goes to position ofsNode(l) + j
    + *
    + * The weights gradient is defined as dE/dW_ijl and dE/dB_jl
    + * It has same mapping as W_ijl and B_jl
    + *
    + * For back propagation:
    + * delta_jl = dE/dS_jl, where S_jl the output of node_jl, but before applying the sigmoid
    + * delta_jl has the same mapping as node_jl
    + *
    + * Where E = ((estOutput-output),(estOutput-output)),
    + * the inner product of the difference between estimation and target output with itself.
    + *
    + */
    +
    +/**
    + * Artificial neural network (ANN) model
    + *
    + * @param weights the weights between the neurons in the ANN.
    + * @param topology array containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + */
    +class ArtificialNeuralNetworkModel private[mllib](val weights: Vector, val topology: Array[Int])
    +  extends Serializable with ANNHelper {
    +
    +  /**
    +   * Predicts values for a single data point using the trained model.
    +   *
    +   * @param testData represents a single data point.
    +   * @return prediction using the trained model.
    +   */
    +  def predict(testData: Vector): Vector = {
    +    Vectors.dense(computeValues(testData.toArray, weights.toArray))
    +  }
    +
    +  /**
    +   * Predict values for an RDD of data points using the trained model.
    +   *
    +   * @param testDataRDD RDD representing the input vectors.
    +   * @return RDD with predictions using the trained model as (input, output) pairs.
    +   */
    +  def predict(testDataRDD: RDD[Vector]): RDD[(Vector,Vector)] = {
    +    testDataRDD.map(T => (T, predict(T)) )
    +  }
    +
    +  private def computeValues(arrData: Array[Double], arrWeights: Array[Double]): Array[Double] = {
    +    val arrNodes = forwardRun(arrData, arrWeights)
    +    arrNodes.slice(arrNodes.size - topology(L), arrNodes.size)
    +  }
    +}
    +
    +/**
    + * Performs the training of an Artificial Neural Network (ANN)
    + *
    + * @param topology A vector containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + * @param maxNumIterations The maximum number of iterations for the training phase.
    + * @param convergenceTol Convergence tolerance for LBFGS. Smaller value for closer convergence.
    + */
    +class ArtificialNeuralNetwork private[mllib](
    +    topology: Array[Int],
    +    maxNumIterations: Int,
    +    convergenceTol: Double)
    +  extends Serializable {
    +
    +  private val gradient = new ANNLeastSquaresGradient(topology)
    +  private val updater = new ANNUpdater()
    +  private val optimizer = new LBFGS(gradient, updater).
    --- End diff --
    
    We should use `GradientDescent`? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-69237765
  
    @bgreeven  I’m not too surprised that the majority vote (a.k.a. one vs. all) did not do very well; it does not scale well with the number of classes.  A tree (or better yet, error-corrected output codes) generally work better, in my experience.
    
    @avulanov  True, we try for consistency with APIs, except where we’re changing the norm.  There is not a clear write-up about the “norm,” although the new spark.ml package and its design doc (in the JIRA) give an overview of some parts.  Basically, we’re aiming to make things more pluggable and extensible, while minimizing API change.  If that requires short-term API changes (such as switching away from ANNWithX method names), that can be acceptable.
    
    @bgreeven @avulanov The test results look pretty good, though I’m not sure what to expect for accuracy.  I think the main item remaining is figuring out the public API.  It’s tough since neural networks / deep learning are a rapidly evolving field, and there are a lot of model & algorithm variants out there.  Ideally, we could put together a design doc (to be linked from the JIRA) for this big feature which would:
    
    * Design a public API for neural networks and deep learning
     * Comparison of other major libraries’ APIs
     * Minimum viable product API for an initial PR
     * Path for the future:
        * What extensions might we need to do, and can we keep the public API stable for these?
        * What extensions might users want to do?  Is the API easily extensible and/or pluggable, or can we make it so in the future without changing the existing public API?
    * Briefly discuss the algorithm
     * Alg sketch, limitations, etc.
     * Alternative algorithms, and a path for making the optimization algorithm pluggable in the future (as we’ve discussed a bit in the PR conversation)
    
    I realize it takes quite a while to get a big new feature ready.  If you’d like to encourage early adoption, you could also post this for now as a package for Spark, while the PR is made fully ready.
    
    CC: @mengxr 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by Lewuathe <gi...@git.apache.org>.
Github user Lewuathe commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67475009
  
    I agree with @jkbradley. 
    > For now, do not expose an optimizer parameter. Only allow one (LBFGS?).
    
    Changing scope of each API should be done considerably. In this case it is the tradeoff between publicity of optimizers and usability of ANN. ANN currently seems to require LBFGS therefore only making it public is the reasonable way. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-55548415
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20315/consoleFull) for   PR 1290 at commit [`3a72fae`](https://github.com/apache/spark/commit/3a72fae62d674d8bc85646ab5a8ab3acaa7d98e7).
     * This patch **passes** unit tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67083539
  
    The test logs have expired...rerunning


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56348790
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20650/consoleFull) for   PR 1290 at commit [`8acd799`](https://github.com/apache/spark/commit/8acd799abc20a054a14ec99780b7609c87f3255f).
     * This patch **fails** unit tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OutputCanvas2D(wd: Int, ht: Int) extends Canvas `
      * `class OutputFrame2D( title: String ) extends Frame( title ) `
      * `class OutputCanvas3D(wd: Int, ht: Int, shadowFrac: Double) extends Canvas `
      * `class OutputFrame3D(title: String, shadowFrac: Double) extends Frame(title) `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-55154710
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20109/consoleFull) for   PR 1290 at commit [`770ea2a`](https://github.com/apache/spark/commit/770ea2a078575ae8203f11068cc50526d31c1792).
     * This patch **fails** unit tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-82756880
  
    I am not sure how to write Figure 1 using matrix but using graphx the gradient calculation is just a forward backward pass on the distributed graph...Also I think it is easier to add more layers using graphx...making the graph balanced is a issue but the matrix alternative is not very clear to me...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-62199347
  
    I think before doing this we need to consider the benefits versus the current implementation with loops. I can name one - the readability of code will be better with matrices. Could you suggest others? The  scalability is not an issue until all weight matrices (unrolled into vector for 'Gradient`) fit into memory of one node.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-51997639
  
    The ANN uses the existent GradientDescent from mllib.optimization for back propagation. It uses the gradient from the new LeastSquaresGradientANN class, and updates using the new ANNUpdater class.
    
    This line in ANNUpdater.compute is the backbone of the back propagation:
    
    brzAxpy(-thisIterStepSize, gradient.toBreeze, brzWeights)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by Lewuathe <gi...@git.apache.org>.
Github user Lewuathe commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-64176809
  
    @avulanov What type of resource manager did you use for checking performance?
    I'm trying to look into the scalability how performance will increase depending of the size of cluster. So as a reference, I want to know what type of resource manger(YARN, Mesos or standalone) was used. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56957127
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20857/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-50591442
  
    QA results for PR 1290:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>abstract class GeneralizedSteepestDescendModel(val weights: Vector )<br>trait ANN {<br>class LeastSquaresGradientANN(<br>class ANNUpdater extends Updater {<br>class ParallelANN (<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17440/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-55202042
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/37/consoleFull) for   PR 1290 at commit [`770ea2a`](https://github.com/apache/spark/commit/770ea2a078575ae8203f11068cc50526d31c1792).
     * This patch **passes** unit tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-53037400
  
    Joining efforts / cooperation is always good of course. :-)
    
    Let me have a closer look at your code first, and see how it differs from mine. I'll try it with my data and see its outcome, usability and speed.
    
    Since the optimisation, my code also works with 1024 input nodes, 512 hidden nodes and 26 output nodes. I still need to play a bit more to find the optimal parameters for this particular problem though.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-50569408
  
    I updated the two sources to comply with "sbt/sbt scalastyle". Maybe retry the unit tests with the new modifications?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by tolgap <gi...@git.apache.org>.
Github user tolgap commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67461517
  
    @bgreeven I have cloned your branch and am trying to run the MNIST dataset. I can't quite understand how to set the number of output neurons though. The `topology` array seems to only apply to the hidden layers. I have seen some tests of MNIST on your code though, so I was curious how this was done?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-61757723
  
    With regards to performance tests, I am going to use mnist8m
    (http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multiclass.html) as suggested by @mengxr. I have set up a cluster with Apache Spark that has five machines with 4-core Xeon 3.3Ghz 8MB cache with 16GB RAM.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-61590143
  
      [Test build #22851 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22851/consoleFull) for   PR 1290 at commit [`73759d5`](https://github.com/apache/spark/commit/73759d5bc587216c2484e9ebbfaac1fe0ab78bfb).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-57077703
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20930/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-54104764
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19573/consoleFull) for   PR 1290 at commit [`f12165a`](https://github.com/apache/spark/commit/f12165a031b387bf640bb9ab05c106699c88ee88).
     * This patch **passes** unit tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `abstract class GeneralizedModel(val weights: Vector )`
      * `class ParallelANN(`
      * `class LeastSquaresGradientANN(`
      * `class ANNUpdater extends Updater `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-50572266
  
    QA tests have started for PR 1290. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17412/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56343909
  
    Changed optimiser to LBFGS. Works much faster, but has the disadvantage (due to the increased convergence speed per iteration) that it also starts to exhibit overfitting earlier (after much fewer iterations).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-68241121
  
    I have compared the ANN with Support Vector Machine (SVM) and Logistic Regression.
    
    I have tested using a master "local(5)" configuration, and applied the MNIST dataset, using 60000 training examples and 10000 test examples.
    
    Since SVM and Logistic Regression are binary classifiers, I applied two methods to convert them to a multinary classifier: majority vote and ad-hoc tree.
    
    For the majority vote, I trained 10 different models, each to distinguish a single class from the rest. The classification was done by looking at which model gives the highest positive output. I performed 100 iterations per class, leading to 1000 iterations in total.
    
    For ANN, I used a single hidden layer with 32 nodes (not counting the bias nodes). I performed 100 iterations.
    
    For LBFGS I used tolerance 1e-5.
    
    Because of the poor performance of SVM+SGD, I re-ran it with 1000 iterations per class (10000 in total). The performance was similar.
    
    I found the following results for the test set:
    
    ```
      Algorithm                     Accuracy   Time        # correct   # incorrect
    +-----------------------------+----------+-----------+-----------+-------------+
    | ANN (LBFGS)                 |    95.1% |      665s |      9510 |         490 |
    +-----------------------------+----------+-----------+-----------+-------------+
    | Logistic Regression (SGD)   |    72.0% |     1325s |      7202 |        2798 |
    +-----------------------------+----------+-----------+-----------+-------------+
    | Logistic Regression (LBFGS) |    86.6% |     1635s |      8658 |        1342 |
    +-----------------------------+----------+-----------+-----------+-------------+
    | SVM (SGD)                   |    18.6% |     1294s |      1860 |        8140 |
    +-----------------------------+----------+-----------+-----------+-------------+
    | (SVM (SGD) 1000 iterations) |    18.5% |    12658s |      1850 |        8150 |
    +-----------------------------+----------+-----------+-----------+-------------+
    | SVM (LBFGS)                 |    86.2% |     1453s |      8622 |        1378 |
    +-----------------------------+----------+-----------+-----------+-------------+
    ```
    
    I also created an ad-hoc tree model. This separates the collection of training examples in two approximately equal size partitions, where I tried to separate the numbers based on how different they look. I continued with the two separated partitions, until each output class corresponded to a single number.
    
    The partioning choice was made manually and intuitively, as follows:
    
    0123456789 -> (04689, 12357)
    04689 -> (068, 49)
    068 -> (0, 68)
    68 -> (6, 8)
    49 -> (4, 9)
    12357 -> (17, 235)
    17 -> (1, 7)
    235 -> (2, 35)
    35 -> (3, 5)
    
    Notice that this leads to only nine classification runs, not ten as in the voting scheme.
    
    After training, I used the trained models to classify the test set. I got the following results (same parameters as with the voting scheme):
    
    ```
      Algorithm                     Accuracy   Time        # correct   # incorrect
    +-----------------------------+----------+-----------+-----------+-------------+
    | ANN (LBFGS)                 |    95.1% |      665s |      9510 |         490 |
    +-----------------------------+----------+-----------+-----------+-------------+
    | Logistic Regression (SGD)   |    82.3% |     1146s |      8228 |        1772 |
    +-----------------------------+----------+-----------+-----------+-------------+
    | Logistic Regression (LBFGS) |    87.2% |     1273s |      8719 |        1281 |
    +-----------------------------+----------+-----------+-----------+-------------+
    | SVM (SGD)                   |    61.1% |     1148s |      6113 |        3887 |
    +-----------------------------+----------+-----------+-----------+-------------+
    | SVM (LBFGS)                 |    87.5% |     1182s |      8753 |        1247 |
    +-----------------------------+----------+-----------+-----------+-------------+
    ```
    
    Notice that I left ANN in the table because this is to compare ANN with other algorithms. Since ANN is a multinary classifier by nature, it didn't use the ad-hoc tree.
    
    It would be great if someone could verify of my results. I am particularly amazed of the low performance of SVM+SGD with voting, and the increase with the ad-hoc tree. I used the same code for SGD and LBFGS, and only changed the optimiser and related parameters.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-50859737
  
    QA tests have started for PR 1290. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17665/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-57077701
  
      [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20930/consoleFull) for   PR 1290 at commit [`804c07a`](https://github.com/apache/spark/commit/804c07a3abd6a0e81d0f04b4a08f88df29cad357).
     * This patch **passes** unit tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OutputCanvas2D(wd: Int, ht: Int) extends Canvas `
      * `class OutputFrame2D( title: String ) extends Frame( title ) `
      * `class OutputCanvas3D(wd: Int, ht: Int, shadowFrac: Double) extends Canvas `
      * `class OutputFrame3D(title: String, shadowFrac: Double) extends Frame(title) `
      * `class IndexedRecordToJavaConverter extends Converter[IndexedRecord, JMap[String, Any]]`
      * `class AvroWrapperToJavaConverter extends Converter[Any, Any] `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-66876255
  
    @ankurdave  
    #3677 should be able to help improve this PR performance


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67908486
  
      [Test build #24718 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24718/consoleFull) for   PR 1290 at commit [`a59f170`](https://github.com/apache/spark/commit/a59f170ac6c671718f9e27ee0db8bbed16f6447a).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-55154549
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20109/consoleFull) for   PR 1290 at commit [`770ea2a`](https://github.com/apache/spark/commit/770ea2a078575ae8203f11068cc50526d31c1792).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67260887
  
    It is an option, however there will be confusion which optimizer to pass, because not all optimizers fit and the trait `Optimizer` is too vague, is not it? Additionally, we will still need few function that uses specific optimizer just to make ANN more user-friendly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67912532
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24718/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-52883816
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19040/consoleFull) for   PR 1290 at commit [`767071f`](https://github.com/apache/spark/commit/767071f409bdd932aada2748644a126700b1ec13).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-53031648
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19079/consoleFull) for   PR 1290 at commit [`78dde56`](https://github.com/apache/spark/commit/78dde5638c5615ad2d97adfa0df9504481ece51c).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r21603916
  
    --- Diff: docs/mllib-ann.md ---
    @@ -0,0 +1,239 @@
    +---
    +layout: global
    +title: Artificial Neural Networks - MLlib
    +displayTitle: <a href="mllib-guide.html">MLlib</a> - Artificial Neural Networks
    +---
    +
    +# Introduction
    +
    +This document describes the MLlib's Artificial Neural Network (ANN) implementation.
    +
    +The implementation currently consist of the following files:
    +
    +* 'ArtificialNeuralNetwork.scala': implements the ANN
    +* 'ANNSuite': implements automated tests for the ANN and its gradient
    +* 'ANNDemo': a demo that approximates three functions and shows a graphical representation of
    +the result
    +
    +# Summary of usage
    +
    +The "ArtificialNeuralNetwork" object is used as an interface to the neural network. It is
    +called as follows:
    +
    +```
    +val annModel = ArtificialNeuralNetwork.train(rdd, hiddenLayersTopology, maxNumIterations)
    +```
    +
    +where
    +
    +* `rdd` is an RDD of type (Vector,Vector), the first element containing the input vector and
    +the second the associated output vector.
    +* `hiddenLayersTopology` is an array of integers (Array[Int]), which contains the number of
    +nodes per hidden layer, starting with the layer that takes inputs from the input layer, and
    +finishing with the layer that outputs to the output layer. The bias nodes are not counted.
    +* `maxNumIterations` is an upper bound to the number of iterations to be performed.
    +* `ANNmodel` contains the trained ANN parameters, and can be used to calculated the ANNs
    +approximation to arbitrary input values.
    +
    +The approximations can be calculated as follows:
    +
    +```
    +val v_out = annModel.predict(v_in)
    +```
    +
    +where v_in is either a Vector or an RDD of Vectors, and v_out respectively a Vector or RDD of
    +(Vector,Vector) pairs, corresponding to input and output values.
    +
    +Further details and other calling options will be elaborated upon below.
    +
    +# Architecture and Notation
    +
    +The file ArtificialNeuralNetwork.scala implements the ANN. The following picture shows the
    +architecture of a 3-layer ANN:
    +
    +```
    + +-------+
    + |       |
    + | N_0,0 |
    + |       | 
    + +-------+        +-------+
    +                  |       |
    + +-------+        | N_0,1 |       +-------+
    + |       |        |       |       |       |
    + | N_1,0 |-       +-------+     ->| N_0,2 |
    + |       | \ Wij1              /  |       |
    + +-------+  --    +-------+  --   +-------+
    +               \  |       | / Wjk2
    +     :          ->| N_1,1 |-      +-------+
    +     :            |       |       |       |
    +     :            +-------+       | N_1,2 |
    +     :                            |       |
    +     :                :           +-------+
    +     :                :
    +     :                :                :
    +     :                : 
    +     :                :           +-------+
    +     :                :           |       |
    +     :                :           |N_K-1,2|
    +     :                            |       |
    +     :            +-------+       +-------+
    +     :            |       |
    +     :            |N_J-1,1|
    +                  |       |
    + +-------+        +-------+
    + |       | 
    + |N_I-1,0|  
    + |       |
    + +-------+
    +
    + +-------+        +--------+
    + |       |        |        |
    + |   -1  |        |   -1   |
    + |       |        |        |
    + +-------+        +--------+
    +
    +INPUT LAYER      HIDDEN LAYER    OUTPUT LAYER
    +```
    +
    +The i-th node in layer l is denoted by N_{i,l}, both i and l starting with 0. The weight
    +between node i in layer l-1 and node j in layer l is denoted by Wijl. Layer 0 is the input
    +layer, whereas layer L is the output layer.
    +
    +The ANN also implements bias units. These are nodes that always output the value -1. The bias
    +units are in all layers except the output layer. They act similar to other nodes, but do not
    +have input.
    +
    +The value of node N_{j,l} is calculated  as follows:
    +
    +`$N_{j,l} = g( \sum_{i=0}^{topology_l} W_{i,j,l)*N_{i,l-1} )$`
    +
    +Where g is the sigmoid function
    +
    +`$g(t) = \frac{e^{\beta t} }{1+e^{\beta t}}$`
    +
    +# LBFGS
    +
    +MLlib's ANN implementation uses the LBFGS optimisation algorithm for training. It minimises the
    +following error function:
    +
    +`$E = \sum_{k=0}^{K-1} (N_{k,L} - Y_k)^2$`
    +
    +where Y_k is the target output given inputs N_{0,0} ... N_{I-1,0}.
    +
    +# Implementation Details
    +
    +## The "ArtificialNeuralNetwork" class
    +
    +The "ArtificialNeuralNetwork" class has the following constructor:
    +
    +```
    +class ArtificialNeuralNetwork private(topology: Array[Int], maxNumIterations: Int,
    +convergenceTol: Double)
    +```
    +
    +* `topology` is an array of integers indicating then number of nodes per layer. For example, if
    +"topology" holds (3, 5, 1), it means that there are three input nodes, five nodes in a single
    +hidden layer and 1 output node.
    +* `maxNumIterations` indicates the number of iterations after which the LBFGS algorithm must
    +have stopped.
    +* `convergenceTol` indicates the acceptable error, and if reached the LBFGS algorithm will
    +stop. A lower value of "convergenceTol" will give a higher precision.
    +
    +## The "ArtificialNeuralNetwork" object
    +
    +The object "ArtificialNeuralNetwork" is the interface to the "ArtificialNeuralNetwork" class.
    +The object contains the training function. There are six different instances of the training
    +function, each for use with different parameters. All take as the first parameter the RDD
    +"input", which contains pairs of input and output vectors.
    +
    +In addition, there are three functions for generating random weights. Two take a fixed seed,
    +which is useful for testing if one wants to start with the same weights in every test.
    +
    +* `def train(trainingRDD: RDD[(Vector, Vector)], hiddenLayersTopology: Array[Int], maxNumIterations:
    +Int): ArtificialNeuralNetworkModel`: starts training with random initial weights, and a default
    +convergenceTol=1e-4.
    +* `def train(trainingRDD: RDD[(Vector, Vector)], model: ArtificialNeuralNetworkModel,
    +maxNumIterations: Int): ArtificialNeuralNetworkModel`: resumes training given an earlier
    +calculated model, and a default convergenceTol=1e-4.
    +* `def train(trainingRDD: RDD[(Vector,Vector)], hiddenLayersTopology: Array[Int],
    +initialWeights: Vector, maxNumIterations: Int): ArtificialNeuralNetworkModel`: Trains an ANN
    +with given initial weights, and a default convergenceTol=1e-4.
    +* `def train(trainingRDD: RDD[(Vector, Vector)], hiddenLayersTopology: Array[Int], maxNumIterations:
    +Int, convergenceTol: Double): ArtificialNeuralNetworkModel`: starts training with random
    +initial weights. Allows setting a customised "convergenceTol".
    +* `def train(trainingRDD: RDD[(Vector, Vector)], model: ArtificialNeuralNetworkModel,
    +maxNumIterations: Int, convergenceTol: Double): ArtificialNeuralNetworkModel`: resumes training
    +given an earlier calculated model. Allows setting a customised "convergenceTol".
    +* `def train(trainingRDD: RDD[(Vector,Vector)], hiddenLayersTopology: Array[Int],
    +initialWeights: Vector, maxNumIterations: Int, convergenceTol: Double): 
    +ArtificialNeuralNetworkModel`: Trains an ANN with given initial weights. Allows setting a
    +customised "convergenceTol".
    +* `def randomWeights(trainingRDD: RDD[(Vector,Vector)], hiddenLayersTopology: Array[Int]):
    +Vector`: Generates a random weights vector.
    +*`def randomWeights(trainingRDD: RDD[(Vector,Vector)], hiddenLayersTopology: Array[Int],
    +seed: Int): Vector`: Generates a random weights vector with given seed.
    +*`def randomWeights(inputLayerSize: Int, outputLayerSize: Int, hiddenLayersTopology: Array[Int],
    +seed: Int): Vector`: Generates a random weights vector, using given random seed, input layer
    +size, hidden layers topology and output layer size.
    +
    +Notice that the "hiddenLayersTopology" differs from the "topology" array. The
    +"hiddenLayersTopology" does not include the number of nodes in the input and output layers. The
    +number of nodes in input and output layers is calculated from the first element of the training
    +RDD. For example, the "topology" array (3, 5, 7, 1) would have a "hiddenLayersTopology" (5, 7),
    +the values 3 and 1 are deduced from the training data. The rationale for having these different
    +arrays is that future methods may have a different mapping between input values and input nodes
    +or output values and output nodes.
    +
    +## The "ArtificialNeuralNetworkModel" class
    +
    +All training functions return the trained ANN using the class "ArtificialNeuralNetworkModel".
    +This class has the following function:
    +
    +* `predict(testData: Vector): Vector` calculates the output vector given input vector
    +"testData".
    +* `predict(testData: RDD[Vector]): RDD[(Vector,Vector)]` returns (input, output) vector pairs,
    +using input vector pairs in "testData".
    +
    +The weights used by "predict" come from the model.
    +
    +## Training
    +
    +We have chosen to implement the ANN with LBFGS as optimiser function. We compared it with
    +Statistical Gradient Descent. LBGFS was much faster, but in accordance is also earlier with
    --- End diff --
    
    Stochastic of course. Changed it. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-49122868
  
    Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-55546270
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20315/consoleFull) for   PR 1290 at commit [`3a72fae`](https://github.com/apache/spark/commit/3a72fae62d674d8bc85646ab5a8ab3acaa7d98e7).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [spark-2352] Implementation of an 1-hidden lay...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-47962409
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-63371703
  
    @jkbradley (a) I found few papers with timing results, however with it was with SVM. Moreover, the results differ from minutes to hours. Could you suggest a paper with performance results of ANN on mnist8m? (b) I can compare against single threaded C++ implementation. I think it will be much faster. Probably the size of data is not enough to run it in distributed mode. Probably, we need some dataset that cannot be used on a single machine. (c) Do you mean comparing with other machine learners in Spark MLlib? To the best of my knowledge, there is only one multiclass machine learner in MLlib, it is Bayesian. It is no supposed to work with negative features.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-68004911
  
    I did few measurements of the recent performance optimizations of `ANNLeastSquaresGradient`. I used the cluster of 6 machines  (Xeon 3.3GHz 4 cores, 16GB RAM) with 12 workers total and mnist8m dataset and trained `ANNClassifier` for 40 iterations with no hidden layer, which is 784x10 topology. Error on mnist test was around 9%.
    
     |Before optimization | After optimization
    --------|------------ | -------------
    Total time | 00:47:55 | 00:16:58
    Avg step time| 51 s. | 23 s.
    
    It became ~3x faster. The optimization should be even more evident with bigger configurations, i.e larger weight matrices.
    
    Update: one can also take a look on the recent comparison with unreleased LogisticRegression https://github.com/apache/spark/pull/1379#issuecomment-68002991
    
    @jkbradley If we want to be consistent with the current MLlib API, we need to implement `ANNwithXXX` pattern as it is done with LogisticRegression. Though, I don't like it neither. BTW how can I restart the test other than with a new commit?
    @bgreeven I like case classes more than the other options considered.
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r19717937
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/ann/ArtificialNeuralNetwork.scala ---
    @@ -0,0 +1,528 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.mllib.ann
    +
    +import breeze.linalg.{DenseVector, Vector => BV, axpy => brzAxpy}
    +
    +import org.apache.spark.mllib.linalg.{Vector, Vectors}
    +import org.apache.spark.mllib.optimization._
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.random.XORShiftRandom
    +
    +/*
    + * Implements a Artificial Neural Network (ANN)
    + *
    + * The data consists of an input vector and an output vector, combined into a single vector
    + * as follows:
    + *
    + * [ ---input--- ---output--- ]
    + *
    + * NOTE: output values should be in the range [0,1]
    + *
    + * For a network of H hidden layers:
    + *
    + * hiddenLayersTopology(h) indicates the number of nodes in hidden layer h, excluding the bias
    + * node. h counts from 0 (first hidden layer, taking inputs from input layer) to H - 1 (last
    + * hidden layer, sending outputs to the output layer).
    + *
    + * hiddenLayersTopology is converted internally to topology, which adds the number of nodes
    + * in the input and output layers.
    + *
    + * noInput = topology(0), the number of input nodes
    + * noOutput = topology(L-1), the number of output nodes
    + *
    + * input = data( 0 to noInput-1 )
    + * output = data( noInput to noInput + noOutput - 1 )
    + *
    + * W_ijl is the weight from node i in layer l-1 to node j in layer l
    + * W_ijl goes to position ofsWeight(l) + j*(topology(l-1)+1) + i in the weights vector
    + *
    + * B_jl is the bias input of node j in layer l
    + * B_jl goes to position ofsWeight(l) + j*(topology(l-1)+1) + topology(l-1) in the weights vector
    + *
    + * error function: E( O, Y ) = sum( O_j - Y_j )
    + * (with O = (O_0, ..., O_(noOutput-1)) the output of the ANN,
    + * and (Y_0, ..., Y_(noOutput-1)) the input)
    + *
    + * node_jl is node j in layer l
    + * node_jl goes to position ofsNode(l) + j
    + *
    + * The weights gradient is defined as dE/dW_ijl and dE/dB_jl
    + * It has same mapping as W_ijl and B_jl
    + *
    + * For back propagation:
    + * delta_jl = dE/dS_jl, where S_jl the output of node_jl, but before applying the sigmoid
    + * delta_jl has the same mapping as node_jl
    + *
    + * Where E = ((estOutput-output),(estOutput-output)),
    + * the inner product of the difference between estimation and target output with itself.
    + *
    + */
    +
    +/**
    + * Artificial neural network (ANN) model
    + *
    + * @param weights the weights between the neurons in the ANN.
    + * @param topology array containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + */
    +class ArtificialNeuralNetworkModel private[mllib](val weights: Vector, val topology: Array[Int])
    +  extends Serializable with ANNHelper {
    +
    +  /**
    +   * Predicts values for a single data point using the trained model.
    +   *
    +   * @param testData represents a single data point.
    +   * @return prediction using the trained model.
    +   */
    +  def predict(testData: Vector): Vector = {
    +    Vectors.dense(computeValues(testData.toArray, weights.toArray))
    +  }
    +
    +  /**
    +   * Predict values for an RDD of data points using the trained model.
    +   *
    +   * @param testDataRDD RDD representing the input vectors.
    +   * @return RDD with predictions using the trained model as (input, output) pairs.
    +   */
    +  def predict(testDataRDD: RDD[Vector]): RDD[(Vector,Vector)] = {
    +    testDataRDD.map(T => (T, predict(T)) )
    +  }
    +
    +  private def computeValues(arrData: Array[Double], arrWeights: Array[Double]): Array[Double] = {
    +    val arrNodes = forwardRun(arrData, arrWeights)
    +    arrNodes.slice(arrNodes.size - topology(L), arrNodes.size)
    +  }
    +}
    +
    +/**
    + * Performs the training of an Artificial Neural Network (ANN)
    + *
    + * @param topology A vector containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + * @param maxNumIterations The maximum number of iterations for the training phase.
    + * @param convergenceTol Convergence tolerance for LBFGS. Smaller value for closer convergence.
    + */
    +class ArtificialNeuralNetwork private[mllib](
    +    topology: Array[Int],
    +    maxNumIterations: Int,
    +    convergenceTol: Double)
    +  extends Serializable {
    +
    +  private val gradient = new ANNLeastSquaresGradient(topology)
    +  private val updater = new ANNUpdater()
    +  private val optimizer = new LBFGS(gradient, updater).
    +    setConvergenceTol(convergenceTol).
    +    setMaxNumIterations(maxNumIterations)
    +
    + /**
    +   * Trains the ANN model.
    --- End diff --
    
    I think the doc is misaligned by one space.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-81842046
  
    Hi, @Zehao 
    Gradient class computes the delta of gradient `dG`, updater class updates the gradient, simply `G(i+1)=G(i) + r*dG`, where `r` is learning rate. Bias `b` of the layer works as follows: `y=f(A*x^T+b)`, where `f` is activation function, `x` is layer input. `A` and `b` are layer parameters or weights.
    
    We are currently working on the more generic interface for the artificial neural networks, which should be easily extensible with the other layer and network types: https://github.com/avulanov/spark/tree/ann-interface


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r19722711
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/ann/ArtificialNeuralNetwork.scala ---
    @@ -0,0 +1,528 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.mllib.ann
    +
    +import breeze.linalg.{DenseVector, Vector => BV, axpy => brzAxpy}
    +
    +import org.apache.spark.mllib.linalg.{Vector, Vectors}
    +import org.apache.spark.mllib.optimization._
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.random.XORShiftRandom
    +
    +/*
    + * Implements a Artificial Neural Network (ANN)
    + *
    + * The data consists of an input vector and an output vector, combined into a single vector
    + * as follows:
    + *
    + * [ ---input--- ---output--- ]
    + *
    + * NOTE: output values should be in the range [0,1]
    + *
    + * For a network of H hidden layers:
    + *
    + * hiddenLayersTopology(h) indicates the number of nodes in hidden layer h, excluding the bias
    + * node. h counts from 0 (first hidden layer, taking inputs from input layer) to H - 1 (last
    + * hidden layer, sending outputs to the output layer).
    + *
    + * hiddenLayersTopology is converted internally to topology, which adds the number of nodes
    + * in the input and output layers.
    + *
    + * noInput = topology(0), the number of input nodes
    + * noOutput = topology(L-1), the number of output nodes
    + *
    + * input = data( 0 to noInput-1 )
    + * output = data( noInput to noInput + noOutput - 1 )
    + *
    + * W_ijl is the weight from node i in layer l-1 to node j in layer l
    + * W_ijl goes to position ofsWeight(l) + j*(topology(l-1)+1) + i in the weights vector
    + *
    + * B_jl is the bias input of node j in layer l
    + * B_jl goes to position ofsWeight(l) + j*(topology(l-1)+1) + topology(l-1) in the weights vector
    + *
    + * error function: E( O, Y ) = sum( O_j - Y_j )
    + * (with O = (O_0, ..., O_(noOutput-1)) the output of the ANN,
    + * and (Y_0, ..., Y_(noOutput-1)) the input)
    + *
    + * node_jl is node j in layer l
    + * node_jl goes to position ofsNode(l) + j
    + *
    + * The weights gradient is defined as dE/dW_ijl and dE/dB_jl
    + * It has same mapping as W_ijl and B_jl
    + *
    + * For back propagation:
    + * delta_jl = dE/dS_jl, where S_jl the output of node_jl, but before applying the sigmoid
    + * delta_jl has the same mapping as node_jl
    + *
    + * Where E = ((estOutput-output),(estOutput-output)),
    + * the inner product of the difference between estimation and target output with itself.
    + *
    + */
    +
    +/**
    + * Artificial neural network (ANN) model
    + *
    + * @param weights the weights between the neurons in the ANN.
    + * @param topology array containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + */
    +class ArtificialNeuralNetworkModel private[mllib](val weights: Vector, val topology: Array[Int])
    +  extends Serializable with ANNHelper {
    +
    +  /**
    +   * Predicts values for a single data point using the trained model.
    +   *
    +   * @param testData represents a single data point.
    +   * @return prediction using the trained model.
    +   */
    +  def predict(testData: Vector): Vector = {
    +    Vectors.dense(computeValues(testData.toArray, weights.toArray))
    +  }
    +
    +  /**
    +   * Predict values for an RDD of data points using the trained model.
    +   *
    +   * @param testDataRDD RDD representing the input vectors.
    +   * @return RDD with predictions using the trained model as (input, output) pairs.
    +   */
    +  def predict(testDataRDD: RDD[Vector]): RDD[(Vector,Vector)] = {
    +    testDataRDD.map(T => (T, predict(T)) )
    +  }
    +
    +  private def computeValues(arrData: Array[Double], arrWeights: Array[Double]): Array[Double] = {
    +    val arrNodes = forwardRun(arrData, arrWeights)
    +    arrNodes.slice(arrNodes.size - topology(L), arrNodes.size)
    +  }
    +}
    +
    +/**
    + * Performs the training of an Artificial Neural Network (ANN)
    + *
    + * @param topology A vector containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + * @param maxNumIterations The maximum number of iterations for the training phase.
    + * @param convergenceTol Convergence tolerance for LBFGS. Smaller value for closer convergence.
    + */
    +class ArtificialNeuralNetwork private[mllib](
    +    topology: Array[Int],
    +    maxNumIterations: Int,
    +    convergenceTol: Double)
    +  extends Serializable {
    +
    +  private val gradient = new ANNLeastSquaresGradient(topology)
    +  private val updater = new ANNUpdater()
    +  private val optimizer = new LBFGS(gradient, updater).
    +    setConvergenceTol(convergenceTol).
    +    setMaxNumIterations(maxNumIterations)
    +
    + /**
    +   * Trains the ANN model.
    +   * Uses default convergence tolerance 1e-4 for LBFGS.
    --- End diff --
    
    The convergenceTol is used by the LBFGS algorithm. So the convergenceTol is taken from the ArtificialNeuralNetwork header and serves as input to the LBFGS algorithm. (setConvergenceTol accesses the LBFGS class, not the ArtificialNeuralNetwork class.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56343891
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20650/consoleFull) for   PR 1290 at commit [`8acd799`](https://github.com/apache/spark/commit/8acd799abc20a054a14ec99780b7609c87f3255f).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-60844377
  
      [Test build #22385 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22385/consoleFull) for   PR 1290 at commit [`79c433e`](https://github.com/apache/spark/commit/79c433ef84f59a999a080dd421ca4bc5856d24ca).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56470205
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20682/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-51796460
  
    Hung,
    You can merge the repository on your spark fork and you should be able to see the code..
     


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-91181467
  
    @debasish83 I'm interested in the Graphx-based NN and implemented a prototype. Yet I found both feed forward and back propagation are slow (unacceptably) due to `aggregateMessages` and `joinVertices`, which seems to be inevitable, and will become worse if layer count increases. 
    
    Is there any other Graph based attempt available? or maybe some suggestion. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-56971423
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20858/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-62254979
  
    I agree with what @debasish83 said. We should find a suitable solution to weight matrix distributed storage. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-61499597
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22818/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r20181004
  
    --- Diff: docs/mllib-ann.md ---
    @@ -0,0 +1,223 @@
    +---
    +layout: global
    +title: Artificial Neural Networks - MLlib
    +displayTitle: <a href="mllib-guide.html">MLlib</a> - Artificial Neural Networks
    +---
    +
    +# Introduction
    +
    +This document describes the MLlib's Artificial Neural Network (ANN) implementation.
    +
    +The implementation currently consist of the following files:
    +
    +* 'ArtificialNeuralNetwork.scala': implements the ANN
    +* 'ANNSuite': implements automated tests for the ANN and its gradient
    +* 'ANNDemo': a demo that approximates three functions and shows a graphical representation of
    +the result
    +
    +# Summary of usage
    +
    +The "ArtificialNeuralNetwork" object is used as an interface to the neural network. It is
    +called as follows:
    +
    +```
    +val annModel = ArtificialNeuralNetwork.train(rdd, hiddenLayersTopology, maxNumIterations)
    --- End diff --
    
    @manishamde  The upcoming API may make it a bit easier, but the current API here could support default parameters via a builder pattern for parameters.  I'll take a closer look at this PR!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67585200
  
    Addendum: notice that the ANNClassifier.train function has several instances, and the number of nodes in the hidden layer(s) is quite critical. Hence I would prefer using:
    ```
    randomWeights(data: RDD[LabeledPoint], hiddenLayersTopology: Array[Int])
    ```
    where hiddenLayersTopology indeed only indicates the nodes in the hidden layers, as number of input and output nodes are deduced from the data.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-53032066
  
      [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19080/consoleFull) for   PR 1290 at commit [`95a88c6`](https://github.com/apache/spark/commit/95a88c660e7c67894b8b86c11017a1da480a9182).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-52456503
  
    Thanks for your feedback. I'll write some documentation, and also add some comments. I'll try with similar size data.
    
    The internal data structure of the weights (and gradient) would have a dimension of (1001*500)+(501*18)=509518 floats. The weights are stored in a non-sparse vector, sometimes converted to Breeze. There may be an issue with that for this size of data. It should be possible though, so worth to have a look on how to fix it.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67883971
  
    @avulanov  I think we should support multiple optimizers too, but it should be done properly, in a way which will not change APIs unless absolutely necessary.  Otherwise, users will suddenly find their code breaks when they update Spark versions; some users will refuse to use modules which do not have stable APIs.  It would be great if we could split the optimizer issue into multiple PRs, where we only add support for more optimizers later on (after the optimizer API is stabilized).
    
    Also, with respect to trainWithX methods, it is really hard to come up with good ways to specify parameters right now because of the problems with Optimizer APIs.  My feeling is that the public API should be declared Experimental and kept as minimal as possible right now, even if it means limiting options.  Afterwards, optimization can be cleaned up, and then the ANN API can be updated and made non-Experimental.  Even if you don't allow all of the options you want at first, it would be valuable to get some other users testing the implementation.
    
    @bgreeven  +1 for only including hidden layer nodes in ```randomWeights()``` argument ```hiddenLayersTopology```.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67716554
  
      [Test build #24658 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24658/consoleFull) for   PR 1290 at commit [`ed6a7c6`](https://github.com/apache/spark/commit/ed6a7c67ad5393a28e45ef3cb3141f21ff53cd93).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OutputCanvas2D(wd: Int, ht: Int) extends Canvas `
      * `class OutputFrame2D( title: String ) extends Frame( title ) `
      * `class OutputCanvas3D(wd: Int, ht: Int, shadowFrac: Double) extends Canvas `
      * `class OutputFrame3D(title: String, shadowFrac: Double) extends Frame(title) `
      * `trait ANNClassifierHelper `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-60853271
  
      [Test build #22385 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22385/consoleFull) for   PR 1290 at commit [`79c433e`](https://github.com/apache/spark/commit/79c433ef84f59a999a080dd421ca4bc5856d24ca).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OutputCanvas2D(wd: Int, ht: Int) extends Canvas `
      * `class OutputFrame2D( title: String ) extends Frame( title ) `
      * `class OutputCanvas3D(wd: Int, ht: Int, shadowFrac: Double) extends Canvas `
      * `class OutputFrame3D(title: String, shadowFrac: Double) extends Frame(title) `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-82685857
  
    The MLLIB existing infrastructure(BSP) is not suitable for  large scale distributed neural net 
    There are JIRA  about the parameters server.
     [SPARK-4590](https://issues.apache.org/jira/browse/SPARK-4590)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-70021585
  
    @loachli @bgreeven We are thinking of changing the name of`ArtificialNeuralNetwork` and `ANNClassifier` objects to `ANNWithLBFGS` and `ANNClassifierWithLBFGS` to be in line with the naming convention in mllib. Are there any objections?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-60742124
  
      [Test build #22359 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22359/consoleFull) for   PR 1290 at commit [`6d167c5`](https://github.com/apache/spark/commit/6d167c52184582d58557fd16a462416391d29d82).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67718891
  
      [Test build #24660 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24660/consoleFull) for   PR 1290 at commit [`1d0ba3e`](https://github.com/apache/spark/commit/1d0ba3e3a0a43c8aa664657ac325d9a8a5d81227).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by hunggpham <gi...@git.apache.org>.
Github user hunggpham commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-51779387
  
    Hi Bert,
    
    I want to try your ANN on Spark but could not find it in the latest clone.   It's probably not there yet despite the successful tests and merge messages above (10 days ago).    How can I get a copy of your ANN code and try it out.
    
    Thanks,
    Hung Pham


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67256540
  
      [Test build #24517 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24517/consoleFull) for   PR 1290 at commit [`2891e94`](https://github.com/apache/spark/commit/2891e9488330a7538eac0a6ec14a40c8b70b37dd).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1290#discussion_r19733377
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/ann/ArtificialNeuralNetwork.scala ---
    @@ -0,0 +1,528 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.mllib.ann
    +
    +import breeze.linalg.{DenseVector, Vector => BV, axpy => brzAxpy}
    +
    +import org.apache.spark.mllib.linalg.{Vector, Vectors}
    +import org.apache.spark.mllib.optimization._
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.random.XORShiftRandom
    +
    +/*
    + * Implements a Artificial Neural Network (ANN)
    + *
    + * The data consists of an input vector and an output vector, combined into a single vector
    + * as follows:
    + *
    + * [ ---input--- ---output--- ]
    + *
    + * NOTE: output values should be in the range [0,1]
    + *
    + * For a network of H hidden layers:
    + *
    + * hiddenLayersTopology(h) indicates the number of nodes in hidden layer h, excluding the bias
    + * node. h counts from 0 (first hidden layer, taking inputs from input layer) to H - 1 (last
    + * hidden layer, sending outputs to the output layer).
    + *
    + * hiddenLayersTopology is converted internally to topology, which adds the number of nodes
    + * in the input and output layers.
    + *
    + * noInput = topology(0), the number of input nodes
    + * noOutput = topology(L-1), the number of output nodes
    + *
    + * input = data( 0 to noInput-1 )
    + * output = data( noInput to noInput + noOutput - 1 )
    + *
    + * W_ijl is the weight from node i in layer l-1 to node j in layer l
    + * W_ijl goes to position ofsWeight(l) + j*(topology(l-1)+1) + i in the weights vector
    + *
    + * B_jl is the bias input of node j in layer l
    + * B_jl goes to position ofsWeight(l) + j*(topology(l-1)+1) + topology(l-1) in the weights vector
    + *
    + * error function: E( O, Y ) = sum( O_j - Y_j )
    + * (with O = (O_0, ..., O_(noOutput-1)) the output of the ANN,
    + * and (Y_0, ..., Y_(noOutput-1)) the input)
    + *
    + * node_jl is node j in layer l
    + * node_jl goes to position ofsNode(l) + j
    + *
    + * The weights gradient is defined as dE/dW_ijl and dE/dB_jl
    + * It has same mapping as W_ijl and B_jl
    + *
    + * For back propagation:
    + * delta_jl = dE/dS_jl, where S_jl the output of node_jl, but before applying the sigmoid
    + * delta_jl has the same mapping as node_jl
    + *
    + * Where E = ((estOutput-output),(estOutput-output)),
    + * the inner product of the difference between estimation and target output with itself.
    + *
    + */
    +
    +/**
    + * Artificial neural network (ANN) model
    + *
    + * @param weights the weights between the neurons in the ANN.
    + * @param topology array containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + */
    +class ArtificialNeuralNetworkModel private[mllib](val weights: Vector, val topology: Array[Int])
    +  extends Serializable with ANNHelper {
    +
    +  /**
    +   * Predicts values for a single data point using the trained model.
    +   *
    +   * @param testData represents a single data point.
    +   * @return prediction using the trained model.
    +   */
    +  def predict(testData: Vector): Vector = {
    +    Vectors.dense(computeValues(testData.toArray, weights.toArray))
    +  }
    +
    +  /**
    +   * Predict values for an RDD of data points using the trained model.
    +   *
    +   * @param testDataRDD RDD representing the input vectors.
    +   * @return RDD with predictions using the trained model as (input, output) pairs.
    +   */
    +  def predict(testDataRDD: RDD[Vector]): RDD[(Vector,Vector)] = {
    +    testDataRDD.map(T => (T, predict(T)) )
    +  }
    +
    +  private def computeValues(arrData: Array[Double], arrWeights: Array[Double]): Array[Double] = {
    +    val arrNodes = forwardRun(arrData, arrWeights)
    +    arrNodes.slice(arrNodes.size - topology(L), arrNodes.size)
    +  }
    +}
    +
    +/**
    + * Performs the training of an Artificial Neural Network (ANN)
    + *
    + * @param topology A vector containing the number of nodes per layer in the network, including
    + * the nodes in the input and output layer, but excluding the bias nodes.
    + * @param maxNumIterations The maximum number of iterations for the training phase.
    + * @param convergenceTol Convergence tolerance for LBFGS. Smaller value for closer convergence.
    + */
    +class ArtificialNeuralNetwork private[mllib](
    +    topology: Array[Int],
    +    maxNumIterations: Int,
    +    convergenceTol: Double)
    +  extends Serializable {
    +
    +  private val gradient = new ANNLeastSquaresGradient(topology)
    +  private val updater = new ANNUpdater()
    +  private val optimizer = new LBFGS(gradient, updater).
    +    setConvergenceTol(convergenceTol).
    +    setMaxNumIterations(maxNumIterations)
    +
    + /**
    +   * Trains the ANN model.
    --- End diff --
    
    Fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by bgreeven <gi...@git.apache.org>.
Github user bgreeven commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-67584041
  
    > @bgreeven I have cloned your branch and am trying to run the MNIST dataset.
    > I can't quite understand how to set the number of output neurons though.
    > The topology array seems to only apply to the hidden layers. I have seen
    > some tests of MNIST on your code though, so I was curious how this was
    > done?
    
    The ANN learns the number of input nodes and number of output nodes from a sample from the data. This is done automatically when invoking the train function.
    
    The ArtificialNeuralNetwork object/class only does approximation, not classification. To do classification, you best use the ANNClassifier object/class.
    
    The simplest call to ANNClassifier is the following:
    ```
    val model = ANNClassifier.train(data)
    ```
    where `data: RDD[LabeledPoint]`
    
    In this case, the features of each LabeledPoint are the bitmap values, whereas the label could be one of the values 0, 1, ..., 9.
    
    Subsequent prediction can be done by
    ```
    val predictedLabel = model.predict(v)
    ```
    
    where v is a vector of features, and the output is one of the labels 0, 1, ...9.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-69854864
  
      [Test build #25491 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25491/consoleFull) for   PR 1290 at commit [`4b8d297`](https://github.com/apache/spark/commit/4b8d2978667b7b1d0dad37f0dbf36c7ac2935af9).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OutputCanvas2D(wd: Int, ht: Int) extends Canvas `
      * `class OutputFrame2D( title: String ) extends Frame( title ) `
      * `class OutputCanvas3D(wd: Int, ht: Int, shadowFrac: Double) extends Canvas `
      * `class OutputFrame3D(title: String, shadowFrac: Double) extends Frame(title) `
      * `trait ANNClassifierHelper `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...

Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-82767499
  
    This seems to be worth a try.  @avulanov what do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-63196649
  
    @avulanov  Thanks for running tests!  It would be great if we could calibrate tests somehow.  Some options would be: (a) trying to reproduce tests in a published paper which has timing results + info about the distributed system they used, (b) testing against another implementation (though I'm not sure about the availability of comparable distributed implementations), or (c) doing self-speedup tests to understand scalability.  What do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/1290#issuecomment-51798466
  
    SteepestDescend should be SteepestDescent ! 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org