You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by VinceShieh <gi...@git.apache.org> on 2016/08/15 02:53:18 UTC
[GitHub] spark pull request #14640: [SPARK-17055] add labelKFold to CrossValidator
GitHub user VinceShieh opened a pull request:
https://github.com/apache/spark/pull/14640
[SPARK-17055] add labelKFold to CrossValidator
## What changes were proposed in this pull request?
This patch improves the CrossValidator by adding a new training/validation split method -labelKFold, which splits data based on data labels and makes sure that the same label is not in both testing and training sets.
This is necessary, for example when data is gathered from different subjects by testing and training on different subjects, i.e., learning cat specific features, and it can avoid over-fitting.
## How was this patch tested?
Unit test was added to MLUtilsSuite.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/VinceShieh/spark labelKFold2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14640.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14640
----
commit cbb78bce4022bfc46f570264de4087a01a84b281
Author: Vincent Xie <vi...@intel.com>
Date: 2016-08-08T13:28:08Z
Add labelKFold to cross validation
Currently, only KFold is supported in cross validation. But in cases
when data is gathered from different subjects and we want to avoid
over-fitting. labelKFold is a variation of k-fold which ensures that
the same label is not in both testing and training sets.
Unit test -'test labelKFold', is also added in MLUtilsSuite
Signed-off-by: Vincent Xie <vi...@intel.com>
Signed-off-by: VinceShieh <vi...@intel.com>
commit 461d696aa6aa41818be31dc1628e3282e560854a
Author: VinceShieh <vi...@intel.com>
Date: 2016-08-15T01:53:51Z
Merge remote-tracking branch 'origin/master' into labelKFold2
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14640: [SPARK-17055] [MLLIB] add groupKFold to CrossValidator
Posted by VinceShieh <gi...@git.apache.org>.
Github user VinceShieh commented on the issue:
https://github.com/apache/spark/pull/14640
@rdelassus Agree. There are a number of folding methods, so some code refractoring should be done if more folding methods are to be supported in the future. But for now, I guess we will just align with what we currently have in mllib. Thanks for your comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14640: [SPARK-17055] [MLLIB] add groupKFold to CrossValidator
Posted by rdelassus <gi...@git.apache.org>.
Github user rdelassus commented on the issue:
https://github.com/apache/spark/pull/14640
There is an infinite number of ways to make folds. Until now we had the mlutils kfold. You want to add the groupedKfold. But I don't think we should add one by one every folding method that can be useful, thus adding (like you did) "if my method else if this othermethod [...] else kfold". It would be far better to make the folding method independant from the crossvalidator class, and pass it as an argument for example.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14640: [SPARK-17055] [MLLIB] add groupKFold to CrossValidator
Posted by VinceShieh <gi...@git.apache.org>.
Github user VinceShieh commented on the issue:
https://github.com/apache/spark/pull/14640
@finleyb indeed, thank you for pointing it out. I have put it right and added a test to guard this issue. Many thanks. And feel free to let us know if you have any problem with this class or any requirement. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14640: [SPARK-17055] [MLLIB] add labelKFold to CrossValidator
Posted by amueller <gi...@git.apache.org>.
Github user amueller commented on the issue:
https://github.com/apache/spark/pull/14640
Do you guys thing "label" is a good name for this? Or did you just take it from us? See the issue linked to above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14640: [SPARK-17055] add labelKFold to CrossValidator
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14640
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14640: [SPARK-17055] [MLLIB] add labelKFold to CrossValidator
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/14640
@VinceShieh there are kfold exists in Spark ML as well, and this PR could maybe go instead of trying to add it on to mllib (for which we don't plan to add new features anymore rather bug fixes only).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14640: [SPARK-17055] [MLLIB] add labelKFold to CrossValidator
Posted by hqzizania <gi...@git.apache.org>.
Github user hqzizania commented on the issue:
https://github.com/apache/spark/pull/14640
This work may be similar with [SPARK-8971](https://github.com/apache/spark/pull/14321) which is another variation of KFold, and very significant in some cases. I suppose it is okay to add to .mllib like the latter PR, but we could add its use to CrossValidator in .ml. @sethah @MLnick @yanboliang
BTW, fortunately, it seems to be easier to implement than the kFoldStratified, as it does not need to change underlying codes, such as in rdd/PairRDDFunctions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14640: [SPARK-17055] [MLLIB] add groupKFold to CrossValidator
Posted by finleyb <gi...@git.apache.org>.
Github user finleyb commented on the issue:
https://github.com/apache/spark/pull/14640
@VinceShieh I was wondering if require in the groupKFold method of MLUtils should be a greater than or equal rather than less than or equal? I was testing this branch because I need this functionality for a ML task I am performing and I ran into the require. Thanks for implementing this!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14640: [SPARK-17055] [MLLIB] add labelKFold to CrossValidator
Posted by MechCoder <gi...@git.apache.org>.
Github user MechCoder commented on the issue:
https://github.com/apache/spark/pull/14640
Just FYI, we plan to rename "LabelKFold" to "GroupKFold" in the next version of sklearn as a label can mean several things. (including the target label)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14640: [SPARK-17055] [MLLIB] add labelKFold to CrossValidator
Posted by VinceShieh <gi...@git.apache.org>.
Github user VinceShieh commented on the issue:
https://github.com/apache/spark/pull/14640
Updates:
1. code refactoring. Rename the API to align with Sklearn changes
2. add implementation in CrossValidator
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14640: [SPARK-17055] [MLLIB] add labelKFold to CrossValidator
Posted by VinceShieh <gi...@git.apache.org>.
Github user VinceShieh commented on the issue:
https://github.com/apache/spark/pull/14640
if one understands the underlying ideas behind this method (labelKFold), it's easy to take it as a class/category of data, though I do think it's not that straightforward, even a bit confusing, when I saw it the first time. @amueller
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14640: [SPARK-17055] [MLLIB] add labelKFold to CrossValidator
Posted by VinceShieh <gi...@git.apache.org>.
Github user VinceShieh commented on the issue:
https://github.com/apache/spark/pull/14640
@holdenk thanks for your comments. :) You are right. But as you can see, this is a variant of kFold, so I think it's better to stay close to it, otherwise, it would seems confusing, dont you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14640: [SPARK-17055] [MLLIB] add labelKFold to CrossValidator
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/14640
Thanks for making this issue and PR :) The first thing before people are likely to have the bandwith to review this is we are switching all new ML development to Spark ML from MLlib so it might be good to retarget this on top of Spark ML.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14640: [SPARK-17055] [MLLIB] add labelKFold to CrossVali...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/14640#discussion_r75826211
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala ---
@@ -228,6 +229,30 @@ object MLUtils extends Logging {
}
/**
+ * Version of [[labelKFold()]] taking a PairRDD with labels.
+ */
+ @Since("2.0.0")
--- End diff --
(This can't be since 2.0.0)
Let me comment on the change itself in the JIRA again. I understand the purpose now, but this is not about labels actually. We'd have to change the name and this wouldn't go in .mllib.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14640: [SPARK-17055] [MLLIB] add groupKFold to CrossVali...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/14640
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14640: [SPARK-17055] [MLLIB] add groupKFold to CrossValidator
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/14640
OK let's close this one for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org