[jira] [Commented] (SPARK-24747) Make spark.ml.util.Instrumentation class more flexible

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-24747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533998#comment-16533998 ] 

Apache Spark commented on SPARK-24747:
--------------------------------------

User 'MrBago' has created a pull request for this issue:
https://github.com/apache/spark/pull/21719

> Make spark.ml.util.Instrumentation class more flexible
> ------------------------------------------------------
>
>                 Key: SPARK-24747
>                 URL: https://issues.apache.org/jira/browse/SPARK-24747
>             Project: Spark
>          Issue Type: Improvement
>          Components: ML
>    Affects Versions: 2.3.1
>            Reporter: Bago Amirbekian
>            Priority: Major
>
> The Instrumentation class (which is an internal private class) is some what limited by it's current APIs. The class requires an estimator and dataset be passed to the constructor which limits how it can be used. Furthermore, the current APIs make it hard to intercept failures and record anything related to those failures.
> The following changes could make the instrumentation class easier to work with. All these changes are for private APIs and should not be visible to users.
> {code}
> // New no-argument constructor:
> Instrumentation()
> // New api to log previous constructor arguments.
> logTrainingContext(estimator: Estimator[_], dataset: Dataset[_])
> logFailure(e: Throwable): Unit
> // Log success with no arguments
> logSuccess(): Unit
> // Log result model explicitly instead of passing to logSuccess
> logModel(model: Model[_]): Unit
> // On Companion object
> Instrumentation.instrumented[T](body: (Instrumentation => T)): T
> // The above API will allow us to write instrumented methods more clearly and handle logging success and failure automatically:
> def someMethod(...): T = instrumented { instr =>
>   instr.logNamedValue(name, value)
>   // more code here
>   instr.logModel(model)
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org