You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hivemall.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2019/09/03 18:31:10 UTC
[jira] [Commented] (HIVEMALL-245) Refactor RandomForest for Sparse
Data handling
[ https://issues.apache.org/jira/browse/HIVEMALL-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921647#comment-16921647 ]
ASF GitHub Bot commented on HIVEMALL-245:
-----------------------------------------
myui commented on issue #198: [WIP][HIVEMALL-245] Refactor RandomForest for Sparse Data handling
URL: https://github.com/apache/incubator-hivemall/pull/198#issuecomment-527582440
```
ed by: org.apache.hadoop.hive.ql.metadata.HiveException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at hivemall.smile.classification.RandomForestClassifierUDTF.train(RandomForestClassifierUDTF.java:415)
at hivemall.smile.classification.RandomForestClassifierUDTF.close(RandomForestClassifierUDTF.java:353)
at org.apache.hadoop.hive.ql.exec.UDTFOperator.closeOp(UDTFOperator.java:152)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:464)
... 15 more
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at hivemall.smile.utils.SmileTaskExecutor.run(SmileTaskExecutor.java:81)
at hivemall.smile.classification.RandomForestClassifierUDTF.train(RandomForestClassifierUDTF.java:413)
... 22 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at hivemall.utils.collections.arrays.SparseIntArray.forEach(SparseIntArray.java:286)
at hivemall.smile.classification.DecisionTree.partitionArray(DecisionTree.java:1028)
at hivemall.smile.classification.DecisionTree.access$1700(DecisionTree.java:119)
at hivemall.smile.classification.DecisionTree$TrainNode$4.accept(DecisionTree.java:997)
at hivemall.smile.utils.VariableOrder.eachRow(VariableOrder.java:41)
at hivemall.smile.classification.DecisionTree$TrainNode.partitionOrder(DecisionTree.java:994)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:896)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
at hivemall.smile.classification.DecisionTree$TrainNode.split(DecisionTree.java:924)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Refactor RandomForest for Sparse Data handling
> ----------------------------------------------
>
> Key: HIVEMALL-245
> URL: https://issues.apache.org/jira/browse/HIVEMALL-245
> Project: Hivemall
> Issue Type: Improvement
> Affects Versions: 0.5.2
> Reporter: Makoto Yui
> Assignee: Makoto Yui
> Priority: Major
> Fix For: 0.6.0
>
>
> * Fix attribute to use RoaringBitmap instead of AttributeType[]
> * Support pruning of redundant decision tree nodes
> * Support the default value for missing values
> * Fix split handling of sparse numeric values
> ** problem: split does not occur when column values have a single value
> *** if(x<=1.0) \{ .. } else \{ ... } never split where sparse x is already 1.0
> ** if there are only a single value for a column, then treat it as nominal value (?)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)