You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by krishnakalyan3 <gi...@git.apache.org> on 2016/06/24 14:14:48 UTC

[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

GitHub user krishnakalyan3 opened a pull request:

    https://github.com/apache/spark/pull/13894

    [SPARK-15254][DOC] Improve ML pipeline Cross Validation Scaladoc & PyDoc

    ## What changes were proposed in this pull request?
    Updated ML pipeline Cross Validation Scaladoc & PyDoc.
    
    
    ## How was this patch tested?
    
    (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
    
    
    (If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
    
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/krishnakalyan3/spark kfold-cv

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/13894.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #13894
    
----
commit d7ffe7a442a6e29b5f3afdf32894d108eb04bb35
Author: krishnakalyan3 <kr...@gmail.com>
Date:   2016-06-24T14:11:37Z

    Improve ML pipeline Cross Validation Scaladoc & PyDoc

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #61726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61726/consoleFull)** for PR 13894 at commit [`7a3a4fe`](https://github.com/apache/spark/commit/7a3a4fe4f7a3304eb528e01af5d40ef4698b6c8d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61921/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by krishnakalyan3 <gi...@git.apache.org>.
Github user krishnakalyan3 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13894#discussion_r69567260
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala ---
    @@ -57,7 +57,7 @@ private[ml] trait CrossValidatorParams extends ValidatorParams {
     /**
      * :: Experimental ::
      * CrossValidator begins by splitting the dataset into a set of non-overlapping randomly
    - * partitioned folds which are used as separate training and test datasets e.g., with k=3 folds,
    + * partitioned folds as separate training and test datasets e.g., with k=3 folds,
    --- End diff --
    
    @MLnick is the current description okay?.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #62412 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62412/consoleFull)** for PR 13894 at commit [`853d6fd`](https://github.com/apache/spark/commit/853d6fd840990070f0b680f0cb9d7b145b048d6e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by krishnakalyan3 <gi...@git.apache.org>.
Github user krishnakalyan3 commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    cc @holdenk 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13894#discussion_r69133212
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala ---
    @@ -56,7 +56,10 @@ private[ml] trait CrossValidatorParams extends ValidatorParams {
     
     /**
      * :: Experimental ::
    - * K-fold cross validation.
    + * CrossValidator begins by splitting the dataset into a set of folds which are 
    + * used as separate training and test datasets e.g., with k=3 folds, CrossValidator 
    + * will generate 3 (training, test) dataset pairs, each of which uses 2/3 of the data 
    + * for training and 1/3 for testing.
    --- End diff --
    
    Also, we can mention that the final cross-validation metric is the average of the test metric achieved for each (training, test) pair.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    @krishnakalyan3 think merge conflicts still need to be resolved - also the Python style issue. Subject to those this LGTM now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #62185 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62185/consoleFull)** for PR 13894 at commit [`013a1db`](https://github.com/apache/spark/commit/013a1dbe06c2babf68b597b4b025a156a9723b03).
     * This patch **fails PySpark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61726/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by krishnakalyan3 <gi...@git.apache.org>.
Github user krishnakalyan3 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13894#discussion_r69103201
  
    --- Diff: python/pyspark/ml/tuning.py ---
    @@ -266,7 +269,7 @@ class CrossValidatorModel(Model, ValidatorParams):
         """
         .. note:: Experimental
     
    -    Model from k-fold cross validation.
    +    CrossValidatorModel model returns the best set of estimators for your model.
    --- End diff --
    
    @holdenk thanks for the review, ran lint-python.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #62185 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62185/consoleFull)** for PR 13894 at commit [`013a1db`](https://github.com/apache/spark/commit/013a1dbe06c2babf68b597b4b025a156a9723b03).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #62631 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62631/consoleFull)** for PR 13894 at commit [`e78d311`](https://github.com/apache/spark/commit/e78d3110d584080a482751329f7e52a2c4661896).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by krishnakalyan3 <gi...@git.apache.org>.
Github user krishnakalyan3 commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    @MLnick @holdenk @jkbradley thanks for the reviews.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61754/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #61754 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61754/consoleFull)** for PR 13894 at commit [`ffc9576`](https://github.com/apache/spark/commit/ffc9576ea246fb8a1e4b2b6d58f4174500f47e91).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by krishnakalyan3 <gi...@git.apache.org>.
Github user krishnakalyan3 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13894#discussion_r69086192
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala ---
    @@ -194,8 +194,7 @@ object CrossValidator extends MLReadable[CrossValidator] {
     
     /**
      * :: Experimental ::
    - * Pipelines facilitate model selection by making it easy to tune an entire 
    - * Pipeline at once, rather than tuning each element in the Pipeline separately.
    + * CrossValidatorModel model returns the best set of estimators for your model.
    --- End diff --
    
    @holdenk I have updated doc. Could you please review it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    LGTM. @holdenk ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Really close, just wondering if we can maybe make the docs for the model a bit clearer since it does a bit more than just contain the best model but it looks pretty good. Thanks for taking this on @krishnakalyan3  and sorry for all the suggestions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #62187 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62187/consoleFull)** for PR 13894 at commit [`f9725cc`](https://github.com/apache/spark/commit/f9725ccf09268aa82beb57ce89ce04ff3be456e3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62187/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13894#discussion_r69892610
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala ---
    @@ -56,7 +56,10 @@ private[ml] trait CrossValidatorParams extends ValidatorParams {
     
     /**
      * :: Experimental ::
    - * K-fold cross validation.
    + * CrossValidator begins by splitting the dataset into a set of non-overlapping randomly
    + * partitioned folds as separate training and test datasets e.g., with k=3 folds,
    --- End diff --
    
    I think we can bring back the "folds, which are used as ..." part


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #62187 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62187/consoleFull)** for PR 13894 at commit [`f9725cc`](https://github.com/apache/spark/commit/f9725ccf09268aa82beb57ce89ce04ff3be456e3).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    @krishnakalyan3 still merge conflicts - could you rebase to current master?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #62591 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62591/consoleFull)** for PR 13894 at commit [`f40210c`](https://github.com/apache/spark/commit/f40210ce7d5ede3e94b7d0509ad593b11d5ea931).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13894#discussion_r69133294
  
    --- Diff: python/pyspark/ml/tuning.py ---
    @@ -266,7 +269,7 @@ class CrossValidatorModel(Model, ValidatorParams):
         """
         .. note:: Experimental
     
    -    Model from k-fold cross validation.
    +    CrossValidatorModel model returns the best set of estimators for your model.
    --- End diff --
    
    Java docs will be generated from the scala doc, so no need for anything special here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13894#discussion_r71392299
  
    --- Diff: python/pyspark/ml/tuning.py ---
    @@ -264,10 +264,10 @@ def copy(self, extra=None):
     
     class CrossValidatorModel(Model, ValidatorParams):
         """
    -    .. note:: Experimental
    -
    -    Model from k-fold cross validation.
    -
    +    CrossValidatorModel contains the model with the highest average cross-validation
    +    metric across folds and uses this model to transform input data. CrossValidatorModel
    +    also tracks the metrics for each param map evaluated.
    +    
    --- End diff --
    
    You've got some extra whitespace here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62412/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13894#discussion_r69524371
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala ---
    @@ -56,7 +56,10 @@ private[ml] trait CrossValidatorParams extends ValidatorParams {
     
     /**
      * :: Experimental ::
    - * K-fold cross validation.
    + * CrossValidator begins by splitting the dataset into a set of non-overlapping randomly
    --- End diff --
    
    sorry about this, but looking at it again I think we can clean up the description a bit more:
    
    `CrossValidator performs model selection by splitting the dataset into a set of non-overlapping randomly partitioned folds, which are used as separate training and test datasets. For example, with k=3 folds, ...`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62591/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by krishnakalyan3 <gi...@git.apache.org>.
Github user krishnakalyan3 commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    @holdenk @MLnick is the current update okay?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62185/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #61921 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61921/consoleFull)** for PR 13894 at commit [`9fb0562`](https://github.com/apache/spark/commit/9fb056293fdba45137c86716eae59fecb1735157).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    jenkins retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by krishnakalyan3 <gi...@git.apache.org>.
Github user krishnakalyan3 commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    cc @holdenk @MLnick @jkbradley. Does the current state look good?.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13894#discussion_r69089084
  
    --- Diff: python/pyspark/ml/tuning.py ---
    @@ -266,8 +269,8 @@ class CrossValidatorModel(Model, ValidatorParams):
         """
         .. note:: Experimental
     
    -    Model from k-fold cross validation.
    -
    +    CrossValidatorModel model returns the best set of estimators for your model.
    +	
    --- End diff --
    
    Seems like a possible formatting issue here - have you run ./dev/lint-python?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #62921 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62921/consoleFull)** for PR 13894 at commit [`e78d311`](https://github.com/apache/spark/commit/e78d3110d584080a482751329f7e52a2c4661896).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #61754 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61754/consoleFull)** for PR 13894 at commit [`ffc9576`](https://github.com/apache/spark/commit/ffc9576ea246fb8a1e4b2b6d58f4174500f47e91).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #62921 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62921/consoleFull)** for PR 13894 at commit [`e78d311`](https://github.com/apache/spark/commit/e78d3110d584080a482751329f7e52a2c4661896).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #62591 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62591/consoleFull)** for PR 13894 at commit [`f40210c`](https://github.com/apache/spark/commit/f40210ce7d5ede3e94b7d0509ad593b11d5ea931).
     * This patch passes all tests.
     * This patch **does not merge cleanly**.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13894#discussion_r69809083
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala ---
    @@ -191,7 +194,8 @@ object CrossValidator extends MLReadable[CrossValidator] {
     
     /**
      * :: Experimental ::
    - * Model from k-fold cross validation.
    + * CrossValidatorModel contains the model that achieved the highest average cross-validation
    --- End diff --
    
    I think we could maybe make this a bit clearer. Maybe something like:
    
    "CrossValidatorModel contains the model with the highest average cross validation metric across folds and uses this model to transform input data. CrossValidatorModel also tracks the metrics for each param map evaluated."
    
    This way its clear that it contains both the best model and the metrics, as well as uses the best model if asked to predict/transform any data. What do you think?
    (Of course if we change it here we would want to have the corresponding change occur in Python).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by krishnakalyan3 <gi...@git.apache.org>.
Github user krishnakalyan3 commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by krishnakalyan3 <gi...@git.apache.org>.
Github user krishnakalyan3 commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    @holdenk @MLnick sorry for so many changes. Newbie here. Please let me know if the current state is okay?. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/13894


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by krishnakalyan3 <gi...@git.apache.org>.
Github user krishnakalyan3 commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Updated the doc based on the reviews. Thanks for the review comments @holdenk and @MLnick. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13894#discussion_r69132570
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala ---
    @@ -56,7 +56,10 @@ private[ml] trait CrossValidatorParams extends ValidatorParams {
     
     /**
      * :: Experimental ::
    - * K-fold cross validation.
    + * CrossValidator begins by splitting the dataset into a set of folds which are 
    + * used as separate training and test datasets e.g., with k=3 folds, CrossValidator 
    + * will generate 3 (training, test) dataset pairs, each of which uses 2/3 of the data 
    + * for training and 1/3 for testing.
    --- End diff --
    
    It may be worth mentioning that each fold is non-overlapping, and that each fold will be used in test exactly once. See [wikipedia](https://en.wikipedia.org/wiki/Cross-validation_(statistics)#k-fold_cross-validation).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13894#discussion_r69132912
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala ---
    @@ -191,7 +194,7 @@ object CrossValidator extends MLReadable[CrossValidator] {
     
     /**
      * :: Experimental ::
    - * Model from k-fold cross validation.
    + * CrossValidatorModel model returns the best set of estimators for your model.
    --- End diff --
    
    Perhaps something like `CrossValidatorModel contains the model that achieved the highest cross-validation metric ( averaged across the number of folds).`, or something like that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62921/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by krishnakalyan3 <gi...@git.apache.org>.
Github user krishnakalyan3 commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    cc @MLnick @holdenk 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #61726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61726/consoleFull)** for PR 13894 at commit [`7a3a4fe`](https://github.com/apache/spark/commit/7a3a4fe4f7a3304eb528e01af5d40ef4698b6c8d).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13894#discussion_r69808182
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala ---
    @@ -56,7 +56,10 @@ private[ml] trait CrossValidatorParams extends ValidatorParams {
     
     /**
      * :: Experimental ::
    - * K-fold cross validation.
    + * CrossValidator begins by splitting the dataset into a set of non-overlapping randomly
    --- End diff --
    
    +1 - is good to mention why its splitting the dataset/what its for.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13894#discussion_r68857452
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala ---
    @@ -191,7 +194,8 @@ object CrossValidator extends MLReadable[CrossValidator] {
     
     /**
      * :: Experimental ::
    - * Model from k-fold cross validation.
    + * Pipelines facilitate model selection by making it easy to tune an entire 
    --- End diff --
    
    I'm not sure if this is the best place to mention that - it seems like it doesn't belong in the CrossValidatorModel scaladoc.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #61921 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61921/consoleFull)** for PR 13894 at commit [`9fb0562`](https://github.com/apache/spark/commit/9fb056293fdba45137c86716eae59fecb1735157).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13894#discussion_r70367155
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala ---
    @@ -56,7 +56,10 @@ private[ml] trait CrossValidatorParams extends ValidatorParams {
     
     /**
      * :: Experimental ::
    - * K-fold cross validation.
    + * CrossValidator begins by splitting the dataset into a set of non-overlapping randomly
    --- End diff --
    
    I like the improved description, but can we please keep the phrase "k-fold cross validation?"  It's a very common phrase and will be useful for people using keyword search.  Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Vali...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13894#discussion_r69281183
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala ---
    @@ -56,7 +56,10 @@ private[ml] trait CrossValidatorParams extends ValidatorParams {
     
     /**
      * :: Experimental ::
    - * K-fold cross validation.
    + * CrossValidator begins by splitting the dataset into a set of non-overlapping randomly
    + * partitioned folds which are used as separate training and test datasets e.g., with k=3 folds,
    + * CrossValidator will generate 3 (training, test) dataset pairs, each of which uses 2/3 of
    + * the data for training and 1/3 for testing. Each fold is used in the testing set exactly once.
    --- End diff --
    
    "used in the testing set" -> "used as the test set"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Merged to master. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #62412 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62412/consoleFull)** for PR 13894 at commit [`853d6fd`](https://github.com/apache/spark/commit/853d6fd840990070f0b680f0cb9d7b145b048d6e).
     * This patch **fails Python style tests**.
     * This patch **does not merge cleanly**.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    **[Test build #62631 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62631/consoleFull)** for PR 13894 at commit [`e78d311`](https://github.com/apache/spark/commit/e78d3110d584080a482751329f7e52a2c4661896).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62631/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13894
  
    Build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org