You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by jluan <ja...@gmail.com> on 2015/09/24 03:59:35 UTC

Spark ClosureCleaner or java serializer OOM when trying to grow

I have been stuck on this problem for the last few days:

I am attempting to run random forest from MLLIB, it gets through most of it,
but breaks when doing a mapPartition operation. The following stack trace is
shown:

: An error occurred while calling o94.trainRandomForestModel.
: java.lang.OutOfMemoryError
        at
java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
        at
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
        at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
        at
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
        at
java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
        at
java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
        at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189)
        at
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
        at
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)
        at
org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:84)
        at
org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301)
        at
org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)
        at org.apache.spark.SparkContext.clean(SparkContext.scala:2021)
        at
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:703)
        at
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:702)
        at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
        at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
        at org.apache.spark.rdd.RDD.mapPartitions(RDD.scala:702)
        at
org.apache.spark.mllib.tree.DecisionTree$.findBestSplits(DecisionTree.scala:625)
        at
org.apache.spark.mllib.tree.RandomForest.run(RandomForest.scala:235)
        at
org.apache.spark.mllib.tree.RandomForest$.trainClassifier(RandomForest.scala:291)
        at
org.apache.spark.mllib.api.python.PythonMLLibAPI.trainRandomForestModel(PythonMLLibAPI.scala:742)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
        at
py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
        at py4j.Gateway.invoke(Gateway.java:259)
        at
py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:207)
        at java.lang.Thread.run(Thread.java:745)

It seems to me that it's trying to serialize the mapPartitions closure, but
runs out of space doing so. However I don't understand how it could run out
of space when I gave the driver ~190GB for a file that's 45MB.

I have a cluster setup on AWS such that my master is a r3.8xlarge along with
two r3.4xlarge workers. I have the following configurations:

spark version: 1.5.0
-----------------------------------
spark.executor.memory 32000m
spark.driver.memory 230000m
spark.driver.cores 10
spark.executor.cores 5
spark.executor.instances 17
spark.driver.maxResultSize 0
spark.storage.safetyFraction 1
spark.storage.memoryFraction 0.9
spark.storage.shuffleFraction 0.05
spark.default.parallelism 128

The master machine has approximately 240 GB of ram and each worker has about
120GB of ram. 

I load in a relatively tiny RDD of MLLIB LabeledPoint objects, with each
holding sparse vectors inside. This RDD has a total size of roughly 45MB. My
sparse vector has a total length of ~15 million while only about 3000 or so
are non-zeros.






--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-ClosureCleaner-or-java-serializer-OOM-when-trying-to-grow-tp24796.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Spark ClosureCleaner or java serializer OOM when trying to grow

Posted by Ted Yu <yu...@gmail.com>.
Please decrease spark.serializer.objectStreamReset for your queries. The
default value is 100.

I logged SPARK-10787 for improvement.

Cheers

On Wed, Sep 23, 2015 at 6:59 PM, jluan <ja...@gmail.com> wrote:

> I have been stuck on this problem for the last few days:
>
> I am attempting to run random forest from MLLIB, it gets through most of
> it,
> but breaks when doing a mapPartition operation. The following stack trace
> is
> shown:
>
> : An error occurred while calling o94.trainRandomForestModel.
> : java.lang.OutOfMemoryError
>         at
> java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
>         at
> java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
>         at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
>         at
> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
>         at
>
> java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
>         at
>
> java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
>         at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189)
>         at
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
>         at
>
> org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)
>         at
>
> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:84)
>         at
>
> org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301)
>         at
>
> org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294)
> at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)
>         at org.apache.spark.SparkContext.clean(SparkContext.scala:2021)
>         at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:703)
>         at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:702)
>         at
>
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
>         at
>
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
>         at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
>         at org.apache.spark.rdd.RDD.mapPartitions(RDD.scala:702)
>         at
>
> org.apache.spark.mllib.tree.DecisionTree$.findBestSplits(DecisionTree.scala:625)
>         at
> org.apache.spark.mllib.tree.RandomForest.run(RandomForest.scala:235)
>         at
>
> org.apache.spark.mllib.tree.RandomForest$.trainClassifier(RandomForest.scala:291)
>         at
>
> org.apache.spark.mllib.api.python.PythonMLLibAPI.trainRandomForestModel(PythonMLLibAPI.scala:742)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
>         at
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
>         at py4j.Gateway.invoke(Gateway.java:259)
>         at
> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
>         at py4j.commands.CallCommand.execute(CallCommand.java:79)
>         at py4j.GatewayConnection.run(GatewayConnection.java:207)
>         at java.lang.Thread.run(Thread.java:745)
>
> It seems to me that it's trying to serialize the mapPartitions closure, but
> runs out of space doing so. However I don't understand how it could run out
> of space when I gave the driver ~190GB for a file that's 45MB.
>
> I have a cluster setup on AWS such that my master is a r3.8xlarge along
> with
> two r3.4xlarge workers. I have the following configurations:
>
> spark version: 1.5.0
> -----------------------------------
> spark.executor.memory 32000m
> spark.driver.memory 230000m
> spark.driver.cores 10
> spark.executor.cores 5
> spark.executor.instances 17
> spark.driver.maxResultSize 0
> spark.storage.safetyFraction 1
> spark.storage.memoryFraction 0.9
> spark.storage.shuffleFraction 0.05
> spark.default.parallelism 128
>
> The master machine has approximately 240 GB of ram and each worker has
> about
> 120GB of ram.
>
> I load in a relatively tiny RDD of MLLIB LabeledPoint objects, with each
> holding sparse vectors inside. This RDD has a total size of roughly 45MB.
> My
> sparse vector has a total length of ~15 million while only about 3000 or so
> are non-zeros.
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-ClosureCleaner-or-java-serializer-OOM-when-trying-to-grow-tp24796.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Re: Spark ClosureCleaner or java serializer OOM when trying to grow

Posted by rohangpatil <ro...@gmail.com>.
Confirming that I am also hitting the same errors.

host: r3.8xlarge

configuration 

spark.serializer org.apache.spark.serializer.KryoSerializer     
spark.driver.memory 200g                                                        
spark.serializer.objectStreamReset 10                                           
spark.local.dir /mnt/rohanp-data2/                                              
spark.driver.maxResultSize 0

Error

15/11/14 08:03:49 INFO BlockManagerInfo: Added broadcast_5_piece32 in memory
on localhost:56741 (size: 1728.1 KB, free: 103.1 GB)                   
[29/1049]
15/11/14 08:03:49 INFO SparkContext: Created broadcast 5 from broadcast at
Word2Vec.scala:286
Exception in thread "main" java.lang.OutOfMemoryError
        at
java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
        at
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
        at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
        at
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
        at
java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1876)
        at
java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1785)
        at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1188)
        at
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
        at
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)
        at
org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:84)
        at
org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301)
        at
org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294)
        at
org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)
        at org.apache.spark.SparkContext.clean(SparkContext.scala:2032)
        at
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:707)
        at
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:706)
        at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
        at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
        at org.apache.spark.rdd.RDD.mapPartitions(RDD.scala:706)
        at org.apache.spark.mllib.feature.Word2Vec.fit(Word2Vec.scala:288)
        at Word2VecApp$.main(Word2VecApp.scala:13)
        at Word2VecApp.main(Word2VecApp.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
        at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15/11/14 08:04:21 INFO SparkContext: Invoking stop() from shutdown hook
15/11/14 08:04:21 INFO SparkUI: Stopped Spark web UI at
http://10.144.64.249:4040
15/11/14 08:04:21 INFO DAGScheduler: Stopping DAGScheduler

Code :

import org.apache.spark._
import org.apache.spark.rdd._
import org.apache.spark.SparkContext._
import org.apache.spark.mllib.feature.{Word2Vec, Word2VecModel}
import org.apache.spark.SparkConf

object Word2VecApp {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("Word2Vec Application")
    val sc = new SparkContext(conf)
    val input =
sc.textFile("/mnt/rohanp-data1/data/training_queries.txt").map(line =>
line.split(" ").toSeq)
        val word2vec = new Word2Vec()
        val model = word2vec.fit(input)
        val synonyms = model.findSynonyms("china", 40)
        for((synonym, cosineSimilarity) <- synonyms) {
          println(s"$synonym $cosineSimilarity")
        }
  }
}

start command:
../spark-submit   --class "Word2VecApp"   --master local[30]  
target/scala-2.10/word2vec-project_2.10-1.0.jar



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-ClosureCleaner-or-java-serializer-OOM-when-trying-to-grow-tp24796p25383.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Spark ClosureCleaner or java serializer OOM when trying to grow

Posted by jluan <ja...@gmail.com>.
With spark.serializer.objectStreamReset set to 1, I ran a sample scala test
code which still seems to be crashing at the same place. If someone could
verify this independently, I would greatly appreciate it.

Scala Code:
----------------------------------------------------------------------
import scala.util.Random
import scala.collection.mutable.ArrayBuffer

import org.apache.spark.mllib.tree.RandomForest
import org.apache.spark.mllib.tree.model.RandomForestModel
import org.apache.spark.mllib.util.MLUtils

import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint

val r = Random

var size = 15000000
var count = 3000
val indptr = (1 to size by size/count).toArray
val data = Seq.fill(count)(r.nextDouble()).toArray

var dset = ArrayBuffer[LabeledPoint]()
for (i <- 1 to 10) {
	dset += LabeledPoint(r.nextInt(2), Vectors.sparse(size, indptr, data));
}

val distData = sc.parallelize(dset)
val splits = distData.randomSplit(Array(0.7, 0.3))
val (trainingData, testData) = (splits(0), splits(1))

// Train a RandomForest model.
//  Empty categoricalFeaturesInfo indicates all features are continuous.
val numClasses = 2
val categoricalFeaturesInfo = Map[Int, Int]()
val numTrees = 3 // Use more in practice.
val featureSubsetStrategy = "auto" // Let the algorithm choose.
val impurity = "gini"
val maxDepth = 4
val maxBins = 32

val model = RandomForest.trainClassifier(trainingData, numClasses,
categoricalFeaturesInfo,
  numTrees, featureSubsetStrategy, impurity, maxDepth, maxBins)



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-ClosureCleaner-or-java-serializer-OOM-when-trying-to-grow-tp24796p24818.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org