You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yanboliang <gi...@git.apache.org> on 2015/04/21 17:26:03 UTC
[GitHub] spark pull request: [SPARK-6827] [mllib] Wrap FPGrowthModel.freqIt...
GitHub user yanboliang opened a pull request:
https://github.com/apache/spark/pull/5614
[SPARK-6827] [mllib] Wrap FPGrowthModel.freqItemsets and make it consistent with Java API
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yanboliang/spark spark-6827
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5614.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5614
----
commit 5532e78b7228b624a3f0d1b09673b813b5909fc0
Author: Yanbo Liang <yb...@gmail.com>
Date: 2015-04-21T15:24:37Z
Wrap FPGrowthModel.freqItemsets and make it consistent with Java API
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6827] [mllib] Wrap FPGrowthModel.freqIt...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5614#issuecomment-95011515
[Test build #30723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30723/consoleFull) for PR 5614 at commit [`da8c404`](https://github.com/apache/spark/commit/da8c404223b69ec1c3e76dad7cf6603ccfe2b8ca).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6827] [mllib] Wrap FPGrowthModel.freqIt...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/5614#issuecomment-95030241
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6827] [mllib] Wrap FPGrowthModel.freqIt...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5614#issuecomment-95042909
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30726/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6827] [mllib] Wrap FPGrowthModel.freqIt...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/5614
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6827] [mllib] Wrap FPGrowthModel.freqIt...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/5614#issuecomment-95011704
@mengxr Thank you for your comments and help, I have merged your PR to this PR.
I will investigate the pickle/unpickle problems and file another PR to resolve it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6827] [mllib] Wrap FPGrowthModel.freqIt...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/5614#issuecomment-94891329
@yanboliang Sent you a PR at https://github.com/yanboliang/spark/pull/2 for using namedtuples.
I'm also thinking about pickling the items into byte strings on the Python side before training. Then on the JVM side, the items are all strings and we don't need to worry about the compatibility of SerDe. When we map the frequent itemsets back, we can unpickle the byte strings. Maybe we can try this in another PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6827] [mllib] Wrap FPGrowthModel.freqIt...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/5614#issuecomment-95374026
LGTM. Merged into master. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6827] [mllib] Wrap FPGrowthModel.freqIt...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5614#issuecomment-94839005
[Test build #30678 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30678/consoleFull) for PR 5614 at commit [`5532e78`](https://github.com/apache/spark/commit/5532e78b7228b624a3f0d1b09673b813b5909fc0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6827] [mllib] Wrap FPGrowthModel.freqIt...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5614#issuecomment-94864215
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30678/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6827] [mllib] Wrap FPGrowthModel.freqIt...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5614#issuecomment-95031269
[Test build #30726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30726/consoleFull) for PR 5614 at commit [`da8c404`](https://github.com/apache/spark/commit/da8c404223b69ec1c3e76dad7cf6603ccfe2b8ca).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6827] [mllib] Wrap FPGrowthModel.freqIt...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5614#issuecomment-95028469
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30723/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6827] [mllib] Wrap FPGrowthModel.freqIt...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/5614#issuecomment-95374557
That is different. In FPGrowth, we don't really care about the item type as long as they are serializable. So it is not necessary to map Python objects into their equivalent JVM objects through SerDes. Instead, we can pickle the items on Python side and treat all items as strings on the JVM side. I'm not sure whether it is worth doing this optimization. Maybe we should wait and see whether there are issues with the current implementation first.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6827] [mllib] Wrap FPGrowthModel.freqIt...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5614#issuecomment-95028436
[Test build #30723 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30723/consoleFull) for PR 5614 at commit [`da8c404`](https://github.com/apache/spark/commit/da8c404223b69ec1c3e76dad7cf6603ccfe2b8ca).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` class FreqItemset(namedtuple("FreqItemset", ["items", "freq"])):`
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6827] [mllib] Wrap FPGrowthModel.freqIt...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5614#issuecomment-94864198
[Test build #30678 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30678/consoleFull) for PR 5614 at commit [`5532e78`](https://github.com/apache/spark/commit/5532e78b7228b624a3f0d1b09673b813b5909fc0).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6827] [mllib] Wrap FPGrowthModel.freqIt...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5614#issuecomment-95042898
[Test build #30726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30726/consoleFull) for PR 5614 at commit [`da8c404`](https://github.com/apache/spark/commit/da8c404223b69ec1c3e76dad7cf6603ccfe2b8ca).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` class FreqItemset(namedtuple("FreqItemset", ["items", "freq"])):`
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org