You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hivemall.apache.org by maropu <gi...@git.apache.org> on 2017/10/15 08:36:52 UTC
[GitHub] incubator-hivemall pull request #122: [HIVEMALL-147][Spark] Support all Hive...
GitHub user maropu opened a pull request:
https://github.com/apache/incubator-hivemall/pull/122
[HIVEMALL-147][Spark] Support all Hivemall functions of v0.5-rc.1 in Spark Dataframe
## What changes were proposed in this pull request?
This pr added more Hivemall functions for Spark DataFrame. However, some of the functions are not supported here because Spark simply cannot handle them (e.g., unsupported types, returned types depending on options, ...).
## What type of PR is it?
Feature
## What is the Jira issue?
https://issues.apache.org/jira/browse/HIVEMALL-147
## How was this patch tested?
Added tests in `HivemallOpsWithFeatureSuite`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maropu/incubator-hivemall HIVEMALL-147-2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hivemall/pull/122.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #122
----
commit 4963c2e71279c095759ba4f545cbbb47cff667b7
Author: Takeshi Yamamuro <ya...@apache.org>
Date: 2017-10-14T15:11:19Z
Support all Hivemall functions of v0.5-rc.1 in Spark Dataframe
----
---
[GitHub] incubator-hivemall issue #122: [HIVEMALL-147][Spark] Support all Hivemall fu...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/122
[](https://coveralls.io/builds/13722141)
Coverage decreased (-0.01%) to 41.098% when pulling **4963c2e71279c095759ba4f545cbbb47cff667b7 on maropu:HIVEMALL-147-2** into **d4f4ab9bafec43d29a366bf955e10a62bb9e603d on apache:master**.
---
[GitHub] incubator-hivemall pull request #122: [HIVEMALL-147][Spark] Support all Hive...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/122#discussion_r144753835
--- Diff: core/src/main/java/hivemall/evaluation/HitRateUDAF.java ---
@@ -71,9 +71,6 @@
+ " - Returns HitRate")
public final class HitRateUDAF extends AbstractGenericUDAFResolver {
- // prevent instantiation
- private HitRateUDAF() {}
-
--- End diff --
This prevents Spark from loading UDAFs by using reflection. Can we remove this?
---
[GitHub] incubator-hivemall pull request #122: [HIVEMALL-147][Spark] Support all Hive...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/122#discussion_r144765390
--- Diff: core/src/main/java/hivemall/evaluation/HitRateUDAF.java ---
@@ -71,9 +71,6 @@
+ " - Returns HitRate")
public final class HitRateUDAF extends AbstractGenericUDAFResolver {
- // prevent instantiation
- private HitRateUDAF() {}
-
--- End diff --
sure.
---
[GitHub] incubator-hivemall pull request #122: [HIVEMALL-147][Spark] Support all Hive...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/122#discussion_r144753777
--- Diff: core/src/main/java/hivemall/evaluation/AUCUDAF.java ---
@@ -110,7 +110,7 @@ public ClassificationEvaluator() {}
@Override
public ObjectInspector init(Mode mode, ObjectInspector[] parameters) throws HiveException {
- assert (parameters.length == 2 || parameters.length == 3) : parameters.length;
+ assert (0 < parameters.length && parameters.length <= 3) : parameters.length;
--- End diff --
In Spark, this assertion fails because Spark passes a single parameter in `parameters` here for final output (IIUC [`AUC` finally outputs a single double-typed value for each group](https://github.com/apache/incubator-hivemall/pull/122/files#diff-9d758588c8fad559a15d0b2362e757b2R1134)). In Hive, does this work well?
---
[GitHub] incubator-hivemall issue #122: [HIVEMALL-147][Spark] Support all Hivemall fu...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/122
LGTM. Merged! Thanks.
---
[GitHub] incubator-hivemall pull request #122: [HIVEMALL-147][Spark] Support all Hive...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/122#discussion_r144765287
--- Diff: core/src/main/java/hivemall/evaluation/AUCUDAF.java ---
@@ -110,7 +110,7 @@ public ClassificationEvaluator() {}
@Override
public ObjectInspector init(Mode mode, ObjectInspector[] parameters) throws HiveException {
- assert (parameters.length == 2 || parameters.length == 3) : parameters.length;
+ assert (0 < parameters.length && parameters.length <= 3) : parameters.length;
--- End diff --
It would be a bug and should use Preconditions. I guess enable assertion `-ea` is not enabled (cc: @takuti )
---
[GitHub] incubator-hivemall pull request #122: [HIVEMALL-147][Spark] Support all Hive...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-hivemall/pull/122
---