You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hivemall.apache.org by myui <gi...@git.apache.org> on 2017/07/11 07:07:13 UTC
[GitHub] incubator-hivemall pull request #98: [WIP][HIVEMALL-108-2] Support `-iter` o...
GitHub user myui opened a pull request:
https://github.com/apache/incubator-hivemall/pull/98
[WIP][HIVEMALL-108-2] Support `-iter` option in generic predictors
## What changes were proposed in this pull request?
Support -iter option in both generic classifier and regression by buffering input samples, with an auxiliary -tol option.
(Ref. #87 #79 )
This PR is based on #87 .
## What type of PR is it?
Improvement
## What is the Jira issue?
https://issues.apache.org/jira/browse/HIVEMALL-108
## How was this patch tested?
unit tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/myui/incubator-hivemall HIVEMALL-108-2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hivemall/pull/98.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #98
----
commit 5b1af8fea4659ec853b9eceebde90899e734071a
Author: Takuya Kitazawa <k....@gmail.com>
Date: 2017-07-06T09:53:52Z
Close #87: [HIVEMALL-108] Support '-iter' option in generic predictors
commit 7bd60a3f1da934721f390f86873d08bc5393b199
Author: Makoto Yui <my...@apache.org>
Date: 2017-07-11T07:00:56Z
Refactored read/writeFeatureValue to be more straightfoward
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #98: [HIVEMALL-108-2] Support `-iter` option in gen...
Posted by takuti <gi...@git.apache.org>.
Github user takuti commented on the issue:
https://github.com/apache/incubator-hivemall/pull/98
I double-checked the code with `-iter 1` option :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #98: [WIP][HIVEMALL-108-2] Support `-iter` option i...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/98
[](https://coveralls.io/builds/12336290)
Coverage increased (+0.3%) to 40.585% when pulling **7bd60a3f1da934721f390f86873d08bc5393b199 on myui:HIVEMALL-108-2** into **e9c66f0a1f3789cabd06e3b9fcaa56d52f05c2a0 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #98: [WIP][HIVEMALL-108-2] Support `-iter` option i...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/98
Need to fix output OI.
```
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to org.apache.hadoop.io.Text
at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.copyObject(WritableStringObjectInspector.java:36)
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:380)
at org.apache.hadoop.hive.ql.exec.KeyWrapperFactory$TextKeyWrapper.copyKey(KeyWrapperFactory.java:221)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:786)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:700)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:768)
```
Also, validation is required for Feature input OI when `-densemodel` option is specified (only INT or BIGINT is expected).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #98: [WIP][HIVEMALL-108-2] Support `-iter` option i...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/98
[](https://coveralls.io/builds/12393734)
Coverage increased (+0.5%) to 40.72% when pulling **902ca08efda5310ffd14be8cbffc3dccb7e6f4b6 on myui:HIVEMALL-108-2** into **e9c66f0a1f3789cabd06e3b9fcaa56d52f05c2a0 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #98: [HIVEMALL-108-2] Support `-iter` option...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-hivemall/pull/98
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #98: [WIP][HIVEMALL-108-2] Support `-iter` option i...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/98
still work in progress. https://github.com/apache/incubator-hivemall/pull/98/commits/5b1af8fea4659ec853b9eceebde90899e734071a is completely based on https://github.com/apache/incubator-hivemall/pull/87
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #98: [WIP][HIVEMALL-108-2] Support `-iter` option i...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/98
[](https://coveralls.io/builds/12391090)
Coverage increased (+0.6%) to 40.905% when pulling **1997c39652a55a7fde7874ff137e2d311f3677f3 on myui:HIVEMALL-108-2** into **e9c66f0a1f3789cabd06e3b9fcaa56d52f05c2a0 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #98: [WIP][HIVEMALL-108-2] Support `-iter` option i...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/98
Confirmed that the following system test passes successfully on EMR.
https://github.com/apache/incubator-hivemall/pull/79
```sql
set hivevar:n_samples=16281;
set hivevar:total_steps=32562;
drop table if exists logress_model;
create table logress_model as
select
feature,
avg(weight) as weight
from
(
select
train_classifier(add_bias(features), label, '-loss logloss -opt SGD -reg no -eta simple -total_steps ${total_steps}') as (feature, weight)
-- logress(add_bias(features), label, '-total_steps ${total_steps}') as (feature, weight)
from
train_x3
) t
group by feature;
WITH test_exploded as (
select
rowid,
label,
extract_feature(feature) as feature,
extract_weight(feature) as value
from
test LATERAL VIEW explode(add_bias(features)) t AS feature
),
predict as (
select
t.rowid,
sigmoid(sum(m.weight * t.value)) as prob,
CAST((case when sigmoid(sum(m.weight * t.value)) >= 0.5 then 1.0 else 0.0 end) as FLOAT) as label
from
test_exploded t LEFT OUTER JOIN
logress_model m ON (t.feature = m.feature)
group by
t.rowid
),
submit as (
select
t.label as actual,
pd.label as predicted,
pd.prob as probability
from
test t JOIN predict pd
on (t.rowid = pd.rowid)
)
select count(1) / ${n_samples} from submit
where actual = predicted;
```
```sql
drop table if exists adagrad_model;
create table adagrad_model as
select
feature,
avg(weight) as weight
from
(
select
-- train_adagrad_regr(features, label) as (feature, weight)
train_regression(features, label, '-loss squaredloss -opt AdaGrad -reg no') as (feature, weight)
from
train_x3
) t
group by feature;
WITH test_exploded as (
select
rowid,
label,
extract_feature(feature) as feature,
extract_weight(feature) as value
from
test LATERAL VIEW explode(add_bias(features)) t AS feature
),
predict as (
select
t.rowid,
sigmoid(sum(m.weight * t.value)) as prob
from
test_exploded t LEFT OUTER JOIN
adagrad_model m ON (t.feature = m.feature)
group by
t.rowid
),
submit as (
select
t.label as actual,
pd.prob as probability
from
test t JOIN predict pd
on (t.rowid = pd.rowid)
)
select rmse(probability, actual) from submit;
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #98: [WIP][HIVEMALL-108-2] Support `-iter` option i...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/98
Need to revise about this CI error.
```
test(hivemall.regression.GeneralRegressionUDTFTest) Time elapsed: 0.1 sec <<< ERROR!
org.apache.hadoop.hive.ql.exec.UDFArgumentException: Only INT or BIGINT is allowed for the element of feature vector when -densemodel option is specified: STRING
at hivemall.GeneralLearnerBaseUDTF.getFeatureOutputOI(GeneralLearnerBaseUDTF.java:249)
at hivemall.GeneralLearnerBaseUDTF.initialize(GeneralLearnerBaseUDTF.java:162)
at hivemall.regression.GeneralRegressionUDTFTest.run(GeneralRegressionUDTFTest.java:254)
at hivemall.regression.GeneralRegressionUDTFTest.test(GeneralRegressionUDTFTest.java:320)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #98: [WIP][HIVEMALL-108-2] Support `-iter` option i...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/98
Need to fix a CI error.
```
Running hivemall.model.FeatureValueTest
Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.122 sec <<< FAILURE!
testParseExpectingNumberFormatException(hivemall.model.FeatureValueTest) Time elapsed: 0.017 sec <<< ERROR!
java.lang.Exception: Unexpected exception, expected<java.lang.NumberFormatException> but was<java.lang.IllegalArgumentException>
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1250)
at java.lang.Double.parseDouble(Double.java:540)
at hivemall.model.FeatureValue.parse(FeatureValue.java:112)
at hivemall.model.FeatureValue.parse(FeatureValue.java:94)
at hivemall.model.FeatureValueTest.testParseExpectingNumberFormatException(FeatureValueTest.java:59)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---