You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by JeremyNixon <gi...@git.apache.org> on 2016/02/14 20:31:25 UTC

[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

GitHub user JeremyNixon opened a pull request:

    https://github.com/apache/spark/pull/11202

    SPARK-10759 [MLlib] Add python example to model selection via cross-validation

    Python example in the same form as the Scala and Java examples. Successfully rendered on my local server and run through pyspark, results match those from the Scala example. 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/JeremyNixon/spark add_py_ex_ml-guide

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/11202.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #11202
    
----
commit c3e83f6e7fc96c77d121f333b27b326812546df8
Author: JeremyNixon <jn...@gmail.com>
Date:   2016-02-13T04:38:18Z

    add python example to model selection via cross-validation

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-187898355
  
    ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-187916328
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51791/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by JeremyNixon <gi...@git.apache.org>.
Github user JeremyNixon closed the pull request at:

    https://github.com/apache/spark/pull/11202


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by JeremyNixon <gi...@git.apache.org>.
Github user JeremyNixon commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-185260069
  
    yinxusen, I have a PR here with the solution to just this example that takes advantage of {% include_example %}, though accepting it would make this example inconsistent with the rest of the examples in this doc. 
    
    I'm also going to go ahead and create a branch with all of the examples fixed with {% include_example %} to make them consistent with one another and take advantage of the automatic testing and dryer code. Would you prefer that I submit that PR here, or create a new JIRA stating that all are fixed? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-210620887
  
    @JeremyNixon is this PR still alive?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-185040855
  
    Refer to https://github.com/apache/spark/pull/11126 and JIRA https://issues.apache.org/jira/browse/SPARK-11337


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-187916309
  
    **[Test build #51791 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51791/consoleFull)** for PR 11202 at commit [`14208ad`](https://github.com/apache/spark/commit/14208ad71cb14c90782c8aa9e6ea7171a798123f).
     * This patch **fails Scala style tests**.
     * This patch **does not merge cleanly**.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-187956836
  
    Yes. Can we close the issue? Since you have https://github.com/apache/spark/pull/11240


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-185016426
  
    One question is whether we should be using `{% include_example %}` when adding new examples to the documentation. We could separate it out into different PRs, but then we are duplicating code (this example already exists [here](https://github.com/apache/spark/blob/master/examples/src/main/python/ml/cross_validator.py)).
    
    @yinxusen could you advise?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-187959472
  
    @JeremyNixon Give a PR under JIRA https://issues.apache.org/jira/browse/SPARK-12877 of the train-validation-split in PySpark. You can add the train-validation-split along with that JIRA. I think we can merge https://github.com/apache/spark/pull/11240 first.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-185040255
  
    It's better to reuse current cross validator with `{% include_example %}` other than writing it directly in markdown file.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-187939971
  
    @JeremyNixon Could you merge the PR with the master?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by JeremyNixon <gi...@git.apache.org>.
Github user JeremyNixon commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-187957979
  
    @yinxusen Two things - 
    
    I finished an implementation of train-validation-split so that we could finish the examples for this doc here: https://github.com/apache/spark/compare/master...JeremyNixon:train_val_split_python, not sure if it belongs in this PR or should be done separately. #11240 requires one small update, which I can push now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by JeremyNixon <gi...@git.apache.org>.
Github user JeremyNixon commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-187940565
  
    @yinxusen Will do.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by JeremyNixon <gi...@git.apache.org>.
Github user JeremyNixon commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-185296342
  
    yinxusen, I went ahead and changed all of the examples in this doc to use {% include_example %} as well as updated the python cross validation example. Let me know if it's acceptable to make all of the changes within this JIRA, or if you'd prefer they be separated. 
    
    I also notice that a JIRA exists (https://issues.apache.org/jira/browse/SPARK-13012) for converting the original examples to use {% include_example %}. The PR makes the same changes that are here, except does not include the change to the python example for cross-validation that this JIRA is supposed to be for.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-183961986
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-187916321
  
    Build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by JeremyNixon <gi...@git.apache.org>.
Github user JeremyNixon commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-210657710
  
    @MLnick Dead PR, work completed with a combination of https://github.com/apache/spark/commit/230bbeaa614ed0ee87ecceece42355dd9a4bacb3 and https://github.com/apache/spark/commit/02b1fefffb00d50c1076a26f2f3f41f3c1fa0001. Thanks for the heads up.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-10759 [MLlib] Add python example to mode...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11202#issuecomment-187905272
  
    **[Test build #51791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51791/consoleFull)** for PR 11202 at commit [`14208ad`](https://github.com/apache/spark/commit/14208ad71cb14c90782c8aa9e6ea7171a798123f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org