You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by WeichenXu123 <gi...@git.apache.org> on 2018/04/16 10:40:35 UTC

[GitHub] spark pull request #21078: [SPARK-23990][ML] Instruments logging improvement...

GitHub user WeichenXu123 opened a pull request:

    https://github.com/apache/spark/pull/21078

    [SPARK-23990][ML] Instruments logging improvements - ML regression package

    ## What changes were proposed in this pull request?
    
    Instruments logging improvements - ML regression package
    
    To be discussed:
     I add an `OptionalLogging` trait, used in `WeightedLeastSquares` and `IterativelyReweightedLeastSquares`, so that `logInfo/logDebug/logWarning` can prior to use `instrument` to log. if no instrument set, then use default logging.
    
    In order to allow `RandomForest` and `GradientBoostedTrees` to use it, I am planning to convert them into class (inheriting `OptionalLogging` trait and pass the `instrument` object via constructor)
    
    ## How was this patch tested?
    
    N/A

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/WeichenXu123/spark inst_reg

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21078.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21078
    
----
commit 617aa27074f98929db4123545fa4d08568b5ffd1
Author: WeichenXu <we...@...>
Date:   2018-04-16T10:02:48Z

    init pr

commit 78c4cd94a48a8c831484b3d51bde9f184f526c90
Author: WeichenXu <we...@...>
Date:   2018-04-16T10:26:51Z

    add optionalLogging trait

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89440 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89440/testReport)** for PR 21078 at commit [`5f60387`](https://github.com/apache/spark/commit/5f6038713e28b2cf1ad10f76f24cfa45e7a5af5f).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89440/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89390/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21078: [SPARK-23990][ML] Instruments logging improvement...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/21078


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89754 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89754/testReport)** for PR 21078 at commit [`492dc46`](https://github.com/apache/spark/commit/492dc4605b0779e5b337d0db41691e372adec45e).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    LGTM
    Merging with master
    Thanks @WeichenXu123 !


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21078: [SPARK-23990][ML] Instruments logging improvement...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21078#discussion_r182833013
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala ---
    @@ -150,3 +156,35 @@ private[spark] object Instrumentation {
       }
     
     }
    +
    +private[spark] class OptionalInstrument private (
    +    val instrument: Option[Instrumentation[_ <: Estimator[_]]],
    +    val className: String) extends Logging {
    +
    +  def this(instr: Instrumentation[_ <: Estimator[_]]) = this(Some(instr), "")
    +
    +  def this(clazz: Class[_]) = this(None, clazz.getName.stripSuffix("$"))
    +
    +  protected override def logName = className
    --- End diff --
    
    Fix IntelliJ style warning: add explicit type annotation


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89547/testReport)** for PR 21078 at commit [`cf649ce`](https://github.com/apache/spark/commit/cf649ce4f377c7efd7e33122678b5135542758f4).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2381/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2365/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89622 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89622/testReport)** for PR 21078 at commit [`1ab9ce0`](https://github.com/apache/spark/commit/1ab9ce03543ca5631117cd73e7b4ccf4fec1c33b).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89391/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89623 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89623/testReport)** for PR 21078 at commit [`ece1a3a`](https://github.com/apache/spark/commit/ece1a3a5eb898a31b868ce9cb754afcd3bbe2d3e).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89544 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89544/testReport)** for PR 21078 at commit [`bbc86b8`](https://github.com/apache/spark/commit/bbc86b813b279fe336ca17348587ae1c9b8486f3).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    @MrBago @jkbradley Thanks!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    @jkbradley Updated.
    I would like to split `RandomForest` and `GradientBoostedTrees` modification into another PR because it will change many methods in them.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89752/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89423/testReport)** for PR 21078 at commit [`7d286c4`](https://github.com/apache/spark/commit/7d286c4322bca3c95f678018dc85e476d77e48da).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89622 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89622/testReport)** for PR 21078 at commit [`1ab9ce0`](https://github.com/apache/spark/commit/1ab9ce03543ca5631117cd73e7b4ccf4fec1c33b).
     * This patch **fails Scala style tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89423 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89423/testReport)** for PR 21078 at commit [`7d286c4`](https://github.com/apache/spark/commit/7d286c4322bca3c95f678018dc85e476d77e48da).
     * This patch **fails to build**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21078: [SPARK-23990][ML] Instruments logging improvement...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21078#discussion_r183514966
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala ---
    @@ -157,34 +161,55 @@ private[spark] object Instrumentation {
     
     }
     
    -private[spark] class OptionalInstrument private (
    -    val instrument: Option[Instrumentation[_ <: Estimator[_]]],
    +/**
    + * A small wrapper that contains an optional `Instrumentation` object.
    + * Provide some log methods, if the containing `Instrumentation` object is defined,
    + * will log via it, otherwise will log via common logger.
    + */
    +private[spark] class OptionalInstrumentation private(
    +    val instrumentation: Option[Instrumentation[_ <: Estimator[_]]],
         val className: String) extends Logging {
     
    -  def this(instr: Instrumentation[_ <: Estimator[_]]) = this(Some(instr), "")
    -
    -  def this(clazz: Class[_]) = this(None, clazz.getName.stripSuffix("$"))
    -
    -  protected override def logName = className
    +  protected override def logName: String = className
     
       override def logInfo(msg: => String) {
    -    instrument match {
    +    instrumentation match {
           case Some(instr) => instr.logInfo(msg)
           case None => super.logInfo(msg)
         }
       }
     
       override def logWarning(msg: => String) {
    -    instrument match {
    +    instrumentation match {
           case Some(instr) => instr.logWarning(msg)
           case None => super.logWarning(msg)
         }
       }
     
       override def logError(msg: => String) {
    -    instrument match {
    +    instrumentation match {
           case Some(instr) => instr.logError(msg)
           case None => super.logError(msg)
         }
       }
     }
    +
    +private[spark] object OptionalInstrumentation {
    +
    +  /**
    +   * Creates an `OptionalInstrumentation` object from an existing `Instrumentation` object.
    +   */
    +  def create(instr: Instrumentation[_ <: Estimator[_]]): OptionalInstrumentation = {
    +    new OptionalInstrumentation(Some(instr),
    +      classOf[Instrumentation[_]].getName.stripSuffix("$"))
    --- End diff --
    
    This is going to log using className = Instrumentation, which is not very informative.  It should log using the Estimator class name.  It might be easier to separate logName from className.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89427/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21078: [SPARK-23990][ML] Instruments logging improvement...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21078#discussion_r182833420
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala ---
    @@ -150,3 +156,35 @@ private[spark] object Instrumentation {
       }
     
     }
    +
    +private[spark] class OptionalInstrument private (
    +    val instrument: Option[Instrumentation[_ <: Estimator[_]]],
    --- End diff --
    
    instrument -> either `instrumentation` or `instr`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2340/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21078: [SPARK-23990][ML] Instruments logging improvement...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21078#discussion_r182833731
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala ---
    @@ -150,3 +156,35 @@ private[spark] object Instrumentation {
       }
     
     }
    +
    +private[spark] class OptionalInstrument private (
    +    val instrument: Option[Instrumentation[_ <: Estimator[_]]],
    +    val className: String) extends Logging {
    +
    +  def this(instr: Instrumentation[_ <: Estimator[_]]) = this(Some(instr), "")
    --- End diff --
    
    Stick with conventions: Move these constructors to object methods called `create()`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89427/testReport)** for PR 21078 at commit [`2d7a8ca`](https://github.com/apache/spark/commit/2d7a8cadb81eb1adf896ef566c1d27f42f8b26ba).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89544/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89752 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89752/testReport)** for PR 21078 at commit [`5ee7e23`](https://github.com/apache/spark/commit/5ee7e233bcaf1e78afbbe6f9b877f48cecb03da6).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89623/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2468/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2341/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89427/testReport)** for PR 21078 at commit [`2d7a8ca`](https://github.com/apache/spark/commit/2d7a8cadb81eb1adf896ef566c1d27f42f8b26ba).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21078: [SPARK-23990][ML] Instruments logging improvement...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21078#discussion_r182833153
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala ---
    @@ -150,3 +156,35 @@ private[spark] object Instrumentation {
       }
     
     }
    +
    +private[spark] class OptionalInstrument private (
    --- End diff --
    
    Stick with existing naming conventions: OptionalInstrument -> OptionalInstrumentation


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89623 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89623/testReport)** for PR 21078 at commit [`ece1a3a`](https://github.com/apache/spark/commit/ece1a3a5eb898a31b868ce9cb754afcd3bbe2d3e).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2465/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89547 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89547/testReport)** for PR 21078 at commit [`cf649ce`](https://github.com/apache/spark/commit/cf649ce4f377c7efd7e33122678b5135542758f4).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89440 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89440/testReport)** for PR 21078 at commit [`5f60387`](https://github.com/apache/spark/commit/5f6038713e28b2cf1ad10f76f24cfa45e7a5af5f).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2369/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2522/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89423/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89754 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89754/testReport)** for PR 21078 at commit [`492dc46`](https://github.com/apache/spark/commit/492dc4605b0779e5b337d0db41691e372adec45e).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2523/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89752 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89752/testReport)** for PR 21078 at commit [`5ee7e23`](https://github.com/apache/spark/commit/5ee7e233bcaf1e78afbbe6f9b877f48cecb03da6).
     * This patch **fails to build**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89754/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89547/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89391 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89391/testReport)** for PR 21078 at commit [`2239856`](https://github.com/apache/spark/commit/2239856567770fbd2a04a9ef50ded8ff5b50b8e5).
     * This patch **fails to build**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2606/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89390 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89390/testReport)** for PR 21078 at commit [`78c4cd9`](https://github.com/apache/spark/commit/78c4cd94a48a8c831484b3d51bde9f184f526c90).
     * This patch **fails to build**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21078: [SPARK-23990][ML] Instruments logging improvement...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21078#discussion_r182836091
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala ---
    @@ -378,18 +378,24 @@ class LinearRegression @Since("1.3.0") (@Since("1.3.0") override val uid: String
     
         val yMean = ySummarizer.mean(0)
         val rawYStd = math.sqrt(ySummarizer.variance(0))
    +
    +    instr.logNamedValue(Instrumentation.loggerTags.numExamples, ySummarizer.count)
    --- End diff --
    
    It might be worth creating a logNumExamples() method in Instrumentation since I bet we'll add this elsewhere.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2605/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89622/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89544 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89544/testReport)** for PR 21078 at commit [`bbc86b8`](https://github.com/apache/spark/commit/bbc86b813b279fe336ca17348587ae1c9b8486f3).
     * This patch **fails to build**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89390 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89390/testReport)** for PR 21078 at commit [`78c4cd9`](https://github.com/apache/spark/commit/78c4cd94a48a8c831484b3d51bde9f184f526c90).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    **[Test build #89391 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89391/testReport)** for PR 21078 at commit [`2239856`](https://github.com/apache/spark/commit/2239856567770fbd2a04a9ef50ded8ff5b50b8e5).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21078: [SPARK-23990][ML] Instruments logging improvement...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21078#discussion_r182834828
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala ---
    @@ -150,3 +156,35 @@ private[spark] object Instrumentation {
       }
     
     }
    +
    +private[spark] class OptionalInstrument private (
    +    val instrument: Option[Instrumentation[_ <: Estimator[_]]],
    +    val className: String) extends Logging {
    +
    +  def this(instr: Instrumentation[_ <: Estimator[_]]) = this(Some(instr), "")
    --- End diff --
    
    Also, let's set logName in the constructor so that, when we create() this with an Instrumentation instance, we can match the Instrumentation instance's logName: instr.getClass.getName.stripSuffix(...)


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Thanks for thinking through the optional logging issue!  I responded in the JIRA to preserve the design discussion there: https://issues.apache.org/jira/browse/SPARK-23990


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2521/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21078: [SPARK-23990][ML] Instruments logging improvement...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21078#discussion_r182835834
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala ---
    @@ -326,7 +326,7 @@ class LinearRegression @Since("1.3.0") (@Since("1.3.0") override val uid: String
             Instance(label, weight, features)
         }
     
    -    val instr = Instrumentation.create(this, dataset)
    +    val instr = Instrumentation.create(this, instances)
    --- End diff --
    
    Why this change?  Is this to match other things inheriting from Predictor?  Let's keep dataset since I hope (maybe in 3.0) we can switch everything to create Instrumentation with Datasets instead of RDDs (since Datasets have more info available).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21078: [SPARK-23990][ML] Instruments logging improvements - ML ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21078
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org