You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hivemall.apache.org by maropu <gi...@git.apache.org> on 2018/11/13 23:20:42 UTC
[GitHub] incubator-hivemall pull request #171: [SPARK][HOTFIX][WIP] Fix existing test...
GitHub user maropu opened a pull request:
https://github.com/apache/incubator-hivemall/pull/171
[SPARK][HOTFIX][WIP] Fix existing test failures in spark-2.3
## What changes were proposed in this pull request?
This pr is to fix the test failures for spark-2.3.
## How was this patch tested?
Run the existing tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maropu/incubator-hivemall HOTFIX-20181114
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hivemall/pull/171.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #171
----
commit cde6fa6d11c2d6e23d52c3db282b9d66b69f4ee9
Author: Takeshi Yamamuro <ya...@...>
Date: 2018-11-13T23:14:33Z
Fix existing issues in spark-2.3
----
---
[GitHub] incubator-hivemall pull request #171: [SPARK][HOTFIX][WIP] Fix existing test...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/171#discussion_r233324312
--- Diff: spark/spark-2.3/src/test/scala/org/apache/spark/sql/hive/XGBoostSuite.scala ---
@@ -77,6 +77,7 @@ final class XGBoostSuite extends VectorQueryTest {
val model = hiveContext.sparkSession.read.format("libxgboost").load(tempDir)
val predict = model.join(mllibTestDf)
.xgboost_predict($"rowid", $"features", $"model_id", $"pred_model")
--- End diff --
BTW, could you paste Stacktrace of the exception?
---
[GitHub] incubator-hivemall pull request #171: [SPARK][HOTFIX][WIP] Fix existing test...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/171#discussion_r233262624
--- Diff: spark/spark-2.3/src/test/scala/org/apache/spark/sql/hive/XGBoostSuite.scala ---
@@ -77,6 +77,7 @@ final class XGBoostSuite extends VectorQueryTest {
val model = hiveContext.sparkSession.read.format("libxgboost").load(tempDir)
val predict = model.join(mllibTestDf)
.xgboost_predict($"rowid", $"features", $"model_id", $"pred_model")
--- End diff --
When invoking `xgboost_predict`, an assesion inside the xgboost library fails. I'm looking into this failure now though, I think we could skip the xgboost support for spark-2.3 in the upcoming release.
---
[GitHub] incubator-hivemall pull request #171: [SPARK][HOTFIX] Fix existing test fail...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/171#discussion_r233399519
--- Diff: spark/spark-2.3/src/test/scala/org/apache/spark/sql/hive/XGBoostSuite.scala ---
@@ -77,6 +77,7 @@ final class XGBoostSuite extends VectorQueryTest {
val model = hiveContext.sparkSession.read.format("libxgboost").load(tempDir)
val predict = model.join(mllibTestDf)
.xgboost_predict($"rowid", $"features", $"model_id", $"pred_model")
--- End diff --
Since it seems the test fails in JNI, we have no stacktrace;
```
XGBoostSuite:
- resolve libxgboost
- check XGBoost options
AssertError:read can not have position excceed buffer length
...
```
---
[GitHub] incubator-hivemall pull request #171: [SPARK][HOTFIX][WIP] Fix existing test...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/171#discussion_r233287092
--- Diff: spark/spark-2.3/src/test/scala/org/apache/spark/sql/hive/XGBoostSuite.scala ---
@@ -77,6 +77,7 @@ final class XGBoostSuite extends VectorQueryTest {
val model = hiveContext.sparkSession.read.format("libxgboost").load(tempDir)
val predict = model.join(mllibTestDf)
.xgboost_predict($"rowid", $"features", $"model_id", $"pred_model")
--- End diff --
Let's disable xgboost for spark-2.3.
---
[GitHub] incubator-hivemall pull request #171: [SPARK][HOTFIX][WIP] Fix existing test...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/171#discussion_r233288446
--- Diff: spark/spark-2.3/src/main/scala/org/apache/spark/sql/hive/HivemallOps.scala ---
@@ -1935,18 +1935,6 @@ object HivemallOps {
)
}
- /**
- * @see [[hivemall.tools.array.SubarrayUDF]]
- * @group tools.array
- */
- def subarray(original: Column, fromIndex: Column, toIndex: Column): Column = withExpr {
- planHiveUDF(
- "hivemall.tools.array.SubarrayUDF",
- "subarray",
- original :: fromIndex :: toIndex :: Nil
- )
- }
--- End diff --
I'll check
---
[GitHub] incubator-hivemall pull request #171: [SPARK][HOTFIX][WIP] Fix existing test...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/171#discussion_r233287740
--- Diff: spark/spark-2.3/src/main/scala/org/apache/spark/sql/hive/HivemallOps.scala ---
@@ -1935,18 +1935,6 @@ object HivemallOps {
)
}
- /**
- * @see [[hivemall.tools.array.SubarrayUDF]]
- * @group tools.array
- */
- def subarray(original: Column, fromIndex: Column, toIndex: Column): Column = withExpr {
- planHiveUDF(
- "hivemall.tools.array.SubarrayUDF",
- "subarray",
- original :: fromIndex :: toIndex :: Nil
- )
- }
--- End diff --
Replacing SubarrayUDF with ArraySliceUDF is not easy?
```
def subarray(original: Column, fromIndex: Column, length: Column): Column = withExpr {
planHiveUDF(
"hivemall.tools.array.ArraySliceUDF",
```
---
[GitHub] incubator-hivemall pull request #171: [SPARK][HOTFIX] Fix the existing test ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-hivemall/pull/171
---
[GitHub] incubator-hivemall pull request #171: [SPARK][HOTFIX][WIP] Fix existing test...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/171#discussion_r233262036
--- Diff: spark/spark-2.3/src/main/scala/org/apache/spark/sql/hive/HivemallOps.scala ---
@@ -1935,18 +1935,6 @@ object HivemallOps {
)
}
- /**
- * @see [[hivemall.tools.array.SubarrayUDF]]
- * @group tools.array
- */
- def subarray(original: Column, fromIndex: Column, toIndex: Column): Column = withExpr {
- planHiveUDF(
- "hivemall.tools.array.SubarrayUDF",
- "subarray",
- original :: fromIndex :: toIndex :: Nil
- )
- }
--- End diff --
Probably, we need to support brickhouse functions for spark in a following separate pr.
---
[GitHub] incubator-hivemall pull request #171: [SPARK][HOTFIX][WIP] Fix existing test...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/171#discussion_r233261637
--- Diff: spark/pom.xml ---
@@ -52,6 +52,12 @@
<artifactId>hivemall-core</artifactId>
<version>${project.version}</version>
<scope>compile</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-all</artifactId>
+ </exclusion>
--- End diff --
Because the `netty` version conflicts with the one in spark.
---
[GitHub] incubator-hivemall issue #171: [SPARK][HOTFIX] Fix the existing test failure...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/171
Merged. Thanks!
---
[GitHub] incubator-hivemall pull request #171: [SPARK][HOTFIX][WIP] Fix existing test...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/171#discussion_r233288186
--- Diff: spark/pom.xml ---
@@ -52,6 +52,12 @@
<artifactId>hivemall-core</artifactId>
<version>${project.version}</version>
<scope>compile</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-all</artifactId>
+ </exclusion>
--- End diff --
ah... I see.
---
[GitHub] incubator-hivemall issue #171: [SPARK][HOTFIX][WIP] Fix existing test failur...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/incubator-hivemall/pull/171
Not finished yet (I'm still working).
---