small accumulator gives out of memory error

[AlexG <sw...@gmail.com>]

When I call the following minimal working example, the accumulator matrix is
32-by-100K, and each executor has 64G but I get an out of memory error:

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
exceeds VM limit

Here BDM is a Breeze DenseMatrix

object BDMAccumulatorParam extends AccumulatorParam[BDM[Double]] {
  def zero(initialValue: BDM[Double]): BDM[Double] = {
    BDM.zeros[Double](initialValue.rows, initialValue.cols)
  }

  def addInPlace(m1: BDM[Double], m2: BDM[Double]) : BDM[Double] = {
    m1 += m2
  }
}

def testfun(mat: IndexedRowMatrix, lhs: DenseMatrix) : DenseMatrix = {

   val accum =
mat.rows.context.accumulator(BDM.zeros[Double](lhs.numRows.toInt,
mat.numCols.toInt))(BDMAccumulatorParam)
   mat.rows.foreach(row => accum += BDM.ones[Double](lhs.numRows.toInt,
mat.numCols.toInt))
   fromBreeze(accum.value)
}

Any ideas or suggestions on how to avoid this error?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/small-accumulator-gives-out-of-memory-error-tp23864.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org