You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:16:24 UTC
[jira] [Resolved] (SPARK-20811) GBT Classifier failed with
mysterious StackOverflowError
[ https://issues.apache.org/jira/browse/SPARK-20811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-20811.
----------------------------------
Resolution: Incomplete
> GBT Classifier failed with mysterious StackOverflowError
> --------------------------------------------------------
>
> Key: SPARK-20811
> URL: https://issues.apache.org/jira/browse/SPARK-20811
> Project: Spark
> Issue Type: Bug
> Components: ML
> Affects Versions: 2.1.0
> Reporter: Nan Zhu
> Priority: Major
> Labels: bulk-closed
>
> I am running GBT Classifier over airline dataset (combining 2005-2008) and in total it's around 22M examples as training data
> code is simple
> {code:title=Bar.scala|borderStyle=solid}
> val gradientBoostedTrees = new GBTClassifier()
> gradientBoostedTrees.setMaxBins(1000)
> gradientBoostedTrees.setMaxIter(500)
> gradientBoostedTrees.setMaxDepth(6)
> gradientBoostedTrees.setStepSize(1.0)
> transformedTrainingSet.cache().foreach(_ => Unit)
> val startTime = System.nanoTime()
> val model = gradientBoostedTrees.fit(transformedTrainingSet)
> println(s"===training time cost: ${(System.nanoTime() - startTime) / 1000.0 / 1000.0} ms")
> val resultDF = model.transform(transformedTestset)
> val binaryClassificationEvaluator = new BinaryClassificationEvaluator()
> binaryClassificationEvaluator.setRawPredictionCol("prediction").setLabelCol("label")
> println(s"=====test AUC: ${binaryClassificationEvaluator.evaluate(resultDF)}======")
> {code}
> my training job always failed with
> {quote}
> 17/05/19 13:41:29 WARN TaskSetManager: Lost task 18.0 in stage 3907.0 (TID 137506, 10.0.0.13, executor 3): java.lang.StackOverflowError
> at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:3037)
> at java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:3061)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2234)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
> at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
> {quote}
> the above pattern repeated for many times
> Is it a bug or did I make something wrong when using GBTClassifier in ML?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org