You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hivemall.apache.org by myui <gi...@git.apache.org> on 2017/07/18 09:58:18 UTC
[GitHub] incubator-hivemall pull request #102: [HIVEMALL-127][Improvement] Added tree...
GitHub user myui opened a pull request:
https://github.com/apache/incubator-hivemall/pull/102
[HIVEMALL-127][Improvement] Added tree_predict_v1 UDF for RandomForest backward compatibility
## What changes were proposed in this pull request?
Added `tree_predict_v1` UDF for RandomForest backward compatibility
## What type of PR is it?
Improvement
## What is the Jira issue?
https://issues.apache.org/jira/browse/HIVEMALL-127
## How was this patch tested?
unit tests, manual tests
## How to use this feature?
```sql
set hivevar:classification=true;
create table predicted
as
SELECT
rowid,
rf_ensemble(predicted) as predicted
FROM (
SELECT
rowid,
tree_predict_v1(p.model_id, p.model_type, p.pred_model, t.features, ${classification}) as predicted
FROM
model p
LEFT OUTER JOIN -- CROSS JOIN
training t
) t1
group by
rowid;
```
## Checklist
- [x] Did you apply source code formatter, i.e., `mvn formatter:format`, for your commit?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/myui/incubator-hivemall HIVEMALL-127
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hivemall/pull/102.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #102
----
commit 7f3ca6b301df2af94055e1024dc28b9c3b2d1a2f
Author: Makoto Yui <my...@apache.org>
Date: 2017-07-18T09:52:45Z
Added tree_predict_v1 UDF for RandomForest backward compatibility
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #102: [HIVEMALL-127] Added tree_predict_v1 UDF for ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/102
[](https://coveralls.io/builds/12455059)
Coverage increased (+0.4%) to 41.137% when pulling **39334d77761e44757ef6e69304013397dda4a2b8 on myui:HIVEMALL-127** into **11bd1f83e68a7fbd2e0cc7143303e35e32edf692 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #102: [HIVEMALL-127] Added tree_predict_v1 UDF for ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/102
[](https://coveralls.io/builds/12455797)
Coverage increased (+0.09%) to 40.832% when pulling **1fbc6d15d3f4ba903ae555fdda22a103979b367c on myui:HIVEMALL-127** into **11bd1f83e68a7fbd2e0cc7143303e35e32edf692 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #102: [HIVEMALL-127] Added tree_predict_v1 UDF for ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/102
[](https://coveralls.io/builds/12456313)
Coverage increased (+0.08%) to 40.829% when pulling **2dd6d00a1b44f7b5a238b2aff8a205edd1c01a14 on myui:HIVEMALL-127** into **11bd1f83e68a7fbd2e0cc7143303e35e32edf692 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #102: [HIVEMALL-127] Added tree_predict_v1 UDF for ...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/102
Need to fix the following error:
```sql
create table predicted_rf_old
as
SELECT
passengerid,
predicted.label,
predicted.probability,
predicted.probabilities
FROM (
SELECT
passengerid,
rf_ensemble(predicted) as predicted
-- hivemall v0.5-rc.1 or later
-- rf_ensemble(predicted.value, predicted.posteriori, model_weight) as predicted
-- rf_ensemble(predicted.value, predicted.posteriori) as predicted -- avoid OOB accuracy (i.e., model_weight)
FROM (
SELECT
t.passengerid,
-- hivemall v0.4.1-alpha.2 or before
-- tree_predict(p.model, t.features, ${classification}) as predicted
-- hivemall v0.4.1-alpha.3 or later
-- tree_predict(p.model_id, p.model_type, p.pred_model, t.features, ${classification}) as predicted
-- hivemall v0.5-rc.1 or later
-- p.model_weight,
-- tree_predict(p.model_id, p.model, t.features, ${classification}) as predicted
tree_predict_v1(p.model_id, p.model_type, p.pred_model, t.features, ${classification}) as predicted -- to use the old model in v0.5-rc.1 or later
FROM (
SELECT
-- model_id, model
-- hivemall v0.4.1-alpha.3 or later
model_id, -3 as model_type, model as pred_model
-- hivemall v0.5-rc.1 or later
-- model_id, model_weight, model
FROM
model_rf
DISTRIBUTE BY rand(1)
) p
LEFT OUTER JOIN test_rf t
) t1
group by
passengerid
) t2
;
```
```
Caused by: java.lang.NullPointerException
at hivemall.smile.tools.RandomForestEnsembleUDAF$RfEvaluatorV1.merge(RandomForestEnsembleUDAF.java:186)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:191)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:619)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:794)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:700)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:768)
... 28 more
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #102: [HIVEMALL-127] Added tree_predict_v1 UDF for ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/102
[](https://coveralls.io/builds/12456457)
Coverage increased (+0.08%) to 40.831% when pulling **2dd6d00a1b44f7b5a238b2aff8a205edd1c01a14 on myui:HIVEMALL-127** into **11bd1f83e68a7fbd2e0cc7143303e35e32edf692 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #102: [HIVEMALL-127] Added tree_predict_v1 UDF for ...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/102
- [ ] need to fix `rf_ensemble()` to accept the old style (1 argument)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #102: [HIVEMALL-127] Added tree_predict_v1 UDF for ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/102
[](https://coveralls.io/builds/12456694)
Coverage increased (+0.08%) to 40.831% when pulling **2dd6d00a1b44f7b5a238b2aff8a205edd1c01a14 on myui:HIVEMALL-127** into **11bd1f83e68a7fbd2e0cc7143303e35e32edf692 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #102: [HIVEMALL-127] Added tree_predict_v1 UDF for ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/102
[](https://coveralls.io/builds/12438067)
Coverage increased (+0.5%) to 41.279% when pulling **7f3ca6b301df2af94055e1024dc28b9c3b2d1a2f on myui:HIVEMALL-127** into **11bd1f83e68a7fbd2e0cc7143303e35e32edf692 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #102: [HIVEMALL-127] Added tree_predict_v1 U...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-hivemall/pull/102
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---