You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ludatabricks <gi...@git.apache.org> on 2018/05/01 06:25:35 UTC

[GitHub] spark pull request #21204: [SPARK-24132][ML]Expand instrumentation for class...

GitHub user ludatabricks opened a pull request:

    https://github.com/apache/spark/pull/21204

    [SPARK-24132][ML]Expand instrumentation for classification

    ## What changes were proposed in this pull request?
    
    - Add OptionalInstrumentation as argument for getNumClasses in ml.classification.Classifier
    
    - Change the function call for getNumClasses in train() in ml.classification.DecisionTreeClassifier, ml.classification.RandomForestClassifier, and ml.classification.NaiveBayes
    
    - Modify the instrumentation creation in ml.classification.LinearSVC
    
    - Change the log call in ml.classification.OneVsRest and ml.classification.LinearSVC
    
    ## How was this patch tested?
    
    Manual.
    
    Please review http://spark.apache.org/contributing.html before opening a pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ludatabricks/spark-1 SPARK-23686

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21204.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21204
    
----
commit 7b75ed6aaa21faf68f5f2db04eaf550f2d468542
Author: Lu WANG <lu...@...>
Date:   2018-05-01T06:13:36Z

    Expand instrumentation for classification

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    **[Test build #90223 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90223/testReport)** for PR 21204 at commit [`e41869e`](https://github.com/apache/spark/commit/e41869ec2c0060f5ed345fdda463ad7f9da70434).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML]Expand instrumentation for classificati...

Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    ok to test


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90223/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...

Posted by ludatabricks <gi...@git.apache.org>.
Github user ludatabricks commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    LGTM Retest this please.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21204: [SPARK-24132][ML]Expand instrumentation for class...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21204#discussion_r185325228
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala ---
    @@ -97,9 +97,10 @@ class DecisionTreeClassifier @Since("1.4.0") (
       override def setSeed(value: Long): this.type = set(seed, value)
     
       override protected def train(dataset: Dataset[_]): DecisionTreeClassificationModel = {
    +    val instr = Instrumentation.create(this, dataset)
         val categoricalFeatures: Map[Int, Int] =
           MetadataUtils.getCategoricalFeatures(dataset.schema($(featuresCol)))
    -    val numClasses: Int = getNumClasses(dataset)
    +    val numClasses: Int = getNumClasses(dataset, instr = OptionalInstrumentation.create(instr))
    --- End diff --
    
    Also see what else you can log easily (e.g., numFeatures)


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML]Expand instrumentation for classificati...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    **[Test build #90070 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90070/testReport)** for PR 21204 at commit [`893c93c`](https://github.com/apache/spark/commit/893c93c9dcdcd674352b09bd370bb19e7e18ca29).
     * This patch **fails SparkR unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML]Expand instrumentation for classificati...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    **[Test build #90148 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90148/testReport)** for PR 21204 at commit [`e41869e`](https://github.com/apache/spark/commit/e41869ec2c0060f5ed345fdda463ad7f9da70434).
     * This patch **fails SparkR unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    **[Test build #90076 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90076/testReport)** for PR 21204 at commit [`e41869e`](https://github.com/apache/spark/commit/e41869ec2c0060f5ed345fdda463ad7f9da70434).
     * This patch **fails SparkR unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90076/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90148/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    LGTM. Merged into master. Thanks!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML]Expand instrumentation for classificati...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    **[Test build #4163 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4163/testReport)** for PR 21204 at commit [`7b75ed6`](https://github.com/apache/spark/commit/7b75ed6aaa21faf68f5f2db04eaf550f2d468542).
     * This patch **fails MiMa tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    **[Test build #90076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90076/testReport)** for PR 21204 at commit [`e41869e`](https://github.com/apache/spark/commit/e41869ec2c0060f5ed345fdda463ad7f9da70434).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    **[Test build #90148 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90148/testReport)** for PR 21204 at commit [`e41869e`](https://github.com/apache/spark/commit/e41869ec2c0060f5ed345fdda463ad7f9da70434).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21204: [SPARK-24132][ML]Expand instrumentation for class...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21204#discussion_r185325179
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala ---
    @@ -97,9 +97,10 @@ class DecisionTreeClassifier @Since("1.4.0") (
       override def setSeed(value: Long): this.type = set(seed, value)
     
       override protected def train(dataset: Dataset[_]): DecisionTreeClassificationModel = {
    +    val instr = Instrumentation.create(this, dataset)
         val categoricalFeatures: Map[Int, Int] =
           MetadataUtils.getCategoricalFeatures(dataset.schema($(featuresCol)))
    -    val numClasses: Int = getNumClasses(dataset)
    +    val numClasses: Int = getNumClasses(dataset, instr = OptionalInstrumentation.create(instr))
    --- End diff --
    
    In cases like this, you can use instr.logNumClasses() instead of relying on logging within the getNumClasses() method.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21204: [SPARK-24132][ML] Instrumentation improvement for...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/21204


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML]Expand instrumentation for classificati...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    **[Test build #4163 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4163/testReport)** for PR 21204 at commit [`7b75ed6`](https://github.com/apache/spark/commit/7b75ed6aaa21faf68f5f2db04eaf550f2d468542).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    **[Test build #90223 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90223/testReport)** for PR 21204 at commit [`e41869e`](https://github.com/apache/spark/commit/e41869ec2c0060f5ed345fdda463ad7f9da70434).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML]Expand instrumentation for classificati...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    **[Test build #90070 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90070/testReport)** for PR 21204 at commit [`893c93c`](https://github.com/apache/spark/commit/893c93c9dcdcd674352b09bd370bb19e7e18ca29).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML]Expand instrumentation for classificati...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML]Expand instrumentation for classificati...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90070/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21204: [SPARK-24132][ML]Expand instrumentation for class...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21204#discussion_r185324974
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/Classifier.scala ---
    @@ -103,7 +103,10 @@ abstract class Classifier[
        * @throws IllegalArgumentException  if metadata does not specify numClasses, and the
        *                                   actual numClasses exceeds maxNumClasses
        */
    -  protected def getNumClasses(dataset: Dataset[_], maxNumClasses: Int = 100): Int = {
    +  protected def getNumClasses(dataset: Dataset[_],
    --- End diff --
    
    Since we don't have Instrumentation readily available, I recommend using the old logging here.  (That will also avoid this API-breaking change which is causing the MiMA failure.)


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21204
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org