You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hivemall.apache.org by maropu <gi...@git.apache.org> on 2017/01/31 02:09:46 UTC
[GitHub] incubator-hivemall pull request #33: [HIVEMALL-44][SAPRK] Implement a protot...
GitHub user maropu opened a pull request:
https://github.com/apache/incubator-hivemall/pull/33
[HIVEMALL-44][SAPRK] Implement a prototype of Join with TopK for DataFrame/Spark
## What changes were proposed in this pull request?
This pr implemented a prototype for Top-K joins. In Hivemall, `each_top_k` is useful for practical use cases. On the other hand, there are some cases we need to join tables then compute Top-K entries.You know we can compute this query by using regular joins + `each_top_k`. However, we have space to improve this query more; that is, we compute Top-K entries while processing joins. This optimization avoids a substantial amount of I/O for joins.
An example query is as follows;
```
val inputDf = Seq(
("user1", 1, 0.3, 0.5),
("user2", 2, 0.1, 0.1),
("user3", 3, 0.8, 0.0),
("user4", 1, 0.9, 0.9),
("user5", 3, 0.7, 0.2),
("user6", 1, 0.5, 0.4),
("user7", 2, 0.6, 0.8)
).toDF("userId", "group", "x", "y")
val masterDf = Seq(
(1, "pos1-1", 0.5, 0.1),
(1, "pos1-2", 0.0, 0.0),
(1, "pos1-3", 0.3, 0.3),
(2, "pos2-3", 0.1, 0.3),
(2, "pos2-3", 0.8, 0.8),
(3, "pos3-1", 0.1, 0.7),
(3, "pos3-1", 0.7, 0.1),
(3, "pos3-1", 0.9, 0.0),
(3, "pos3-1", 0.1, 0.3)
).toDF("group", "position", "x", "y")
// Compute top-1 rows for each group
val distance = sqrt(
pow(inputDf("x") - masterDf("x"), lit(2.0)) +
pow(inputDf("y") - masterDf("y"), lit(2.0))
)
val top1Df = inputDf.join_top_k(
lit(1), masterDf, inputDf("group") === masterDf("group"),
distance.as("score")
)
```
A quick benchmark is as follows;
```
/**
* Java HotSpot(TM) 64-Bit Server VM 1.8.0_31-b13 on Mac OS X 10.10.2
* Intel(R) Core(TM) i7-4578U CPU @ 3.00GHz
*
* top-k join (k=3): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
* -------------------------------------------------------------------------------
* join + rank 65959 / 71324 0.0 503223.9 1.0X
* join + each_top_k 66093 / 78864 0.0 504247.3 1.0X
* join_top_k 5013 / 5431 0.0 38249.3 13.2X
*/
```
The APIs this pr added is unstable and we might change them in upcoming activities.
## What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/HIVEMALL-44
## How was this patch tested?
Added tests in `HivemallOpsSuite`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maropu/incubator-hivemall HIVEMALL-44
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hivemall/pull/33.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #33
----
commit a7c70c962a716ec200871c4f4c52d9a61aae2018
Author: Takeshi YAMAMURO <li...@gmail.com>
Date: 2017-01-30T23:57:19Z
Implement a prototype of Join with TopK
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
output `top1Df` should be explained in the description.
BTW, I personally prefer `top_k_join` instead of `join_top_k`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
[](https://coveralls.io/builds/9950414)
Coverage remained the same at 35.842% when pulling **c7a9838a26b6b3f0de2945d934c884dae1114f58 on maropu:HIVEMALL-44** into **c837e51adcf840f478537d8bb7c9ca810b3df200 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
@maropu waiting for markdown to be included in this PR :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
[](https://coveralls.io/builds/9947771)
Coverage remained the same at 35.842% when pulling **c7a9838a26b6b3f0de2945d934c884dae1114f58 on maropu:HIVEMALL-44** into **c837e51adcf840f478537d8bb7c9ca810b3df200 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
[](https://coveralls.io/builds/9948075)
Coverage remained the same at 35.842% when pulling **c7a9838a26b6b3f0de2945d934c884dae1114f58 on maropu:HIVEMALL-44** into **c837e51adcf840f478537d8bb7c9ca810b3df200 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
@maropu Thanks! Merged.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
@myui Many Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #33: [HIVEMALL-44][SAPRK] Implement a protot...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-hivemall/pull/33
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
[](https://coveralls.io/builds/9947681)
Coverage remained the same at 35.842% when pulling **c7a9838a26b6b3f0de2945d934c884dae1114f58 on maropu:HIVEMALL-44** into **c837e51adcf840f478537d8bb7c9ca810b3df200 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
[](https://coveralls.io/builds/9947317)
Coverage remained the same at 35.842% when pulling **c7a9838a26b6b3f0de2945d934c884dae1114f58 on maropu:HIVEMALL-44** into **c837e51adcf840f478537d8bb7c9ca810b3df200 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
@myui Added a doc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
[](https://coveralls.io/builds/9947788)
Coverage remained the same at 35.842% when pulling **c7a9838a26b6b3f0de2945d934c884dae1114f58 on maropu:HIVEMALL-44** into **c837e51adcf840f478537d8bb7c9ca810b3df200 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
Will do
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
[](https://coveralls.io/builds/9947788)
Coverage remained the same at 35.842% when pulling **c7a9838a26b6b3f0de2945d934c884dae1114f58 on maropu:HIVEMALL-44** into **c837e51adcf840f478537d8bb7c9ca810b3df200 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
[](https://coveralls.io/builds/9947706)
Coverage remained the same at 35.842% when pulling **c7a9838a26b6b3f0de2945d934c884dae1114f58 on maropu:HIVEMALL-44** into **c837e51adcf840f478537d8bb7c9ca810b3df200 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
[](https://coveralls.io/builds/9918006)
Coverage increased (+0.3%) to 36.14% when pulling **5ed10f2d38b7312202d478b93ad3b3caf34e230b on maropu:HIVEMALL-44** into **ed0d139bea229183b286d36bf29cd027eb889634 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
[](https://coveralls.io/builds/9917473)
Coverage remained the same at 35.842% when pulling **a7c70c962a716ec200871c4f4c52d9a61aae2018 on maropu:HIVEMALL-44** into **ed0d139bea229183b286d36bf29cd027eb889634 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
[](https://coveralls.io/builds/9947338)
Coverage increased (+0.3%) to 36.14% when pulling **c7a9838a26b6b3f0de2945d934c884dae1114f58 on maropu:HIVEMALL-44** into **c837e51adcf840f478537d8bb7c9ca810b3df200 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
@myui ping
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
[](https://coveralls.io/builds/9918006)
Coverage increased (+0.3%) to 36.14% when pulling **5ed10f2d38b7312202d478b93ad3b3caf34e230b on maropu:HIVEMALL-44** into **ed0d139bea229183b286d36bf29cd027eb889634 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #33: [HIVEMALL-44][SAPRK] Implement a prototype of ...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/33
[](https://coveralls.io/builds/9947771)
Coverage remained the same at 35.842% when pulling **c7a9838a26b6b3f0de2945d934c884dae1114f58 on maropu:HIVEMALL-44** into **c837e51adcf840f478537d8bb7c9ca810b3df200 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---