You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by holdenk <gi...@git.apache.org> on 2017/12/04 11:27:30 UTC

[GitHub] spark pull request #19876: [WIP][ML][SPARK-11171] spark 11237 Add PMML expor...

GitHub user holdenk opened a pull request:

    https://github.com/apache/spark/pull/19876

    [WIP][ML][SPARK-11171] spark 11237 Add PMML export to Spark ML pipelines 

    ## What changes were proposed in this pull request?
    
    Adds PMML export support to Spark ML pipelines in the style of Spark's DataSource API to allow library authors to add their own model export formats.
    
    This is a WIP to see if this is the design we want to go with.
    
    ## How was this patch tested?
    
    Basic unit test.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/holdenk/spark SPARK-11171-SPARK-11237-Add-PMML-export-for-ML-KMeans-r2

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19876.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19876
    
----
commit 43ae30f08aed921178da07a5e982297b272c7c8f
Author: Holden Karau <ho...@google.com>
Date:   2017-11-24T14:00:16Z

    Initial attempt at allowing Spark ML writers to be slightly more pluggable

commit 9fec08fbd2dd1c980d5862f0b4521213e1e9349c
Author: Holden Karau <ho...@google.com>
Date:   2017-11-25T12:55:19Z

    The LinearRegression suite passes

commit 0075bf4776ecffa7fcb24a6f74c0e96161d6221c
Author: Holden Karau <ho...@google.com>
Date:   2017-11-25T13:00:18Z

    Add missing META-INFO for MLFormatRegister

commit c68880d6d982c56934f4b583263ed5cd4e8329d6
Author: Holden Karau <ho...@google.com>
Date:   2017-11-25T16:19:35Z

    Add a (untested) PMMLLinearRegressionModelWriter

commit c2108df2b499bd45dff0e8add789f01d8c3c2c48
Author: Holden Karau <ho...@google.com>
Date:   2017-12-04T10:00:56Z

    Basic PMML export test

commit de8619098eeb01ff86b54753f27c29729935bb94
Author: Holden Karau <ho...@google.com>
Date:   2017-12-04T11:27:03Z

    Add PMML testing utils for Spark ML that were accidently left out

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML expo...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r158126871
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -126,15 +180,69 @@ abstract class MLWriter extends BaseReadWrite with Logging {
         this
       }
     
    +  // override for Java compatibility
    +  override def session(sparkSession: SparkSession): this.type = super.session(sparkSession)
    +
    +  // override for Java compatibility
    +  override def context(sqlContext: SQLContext): this.type = super.session(sqlContext.sparkSession)
    +}
    +
    +/**
    + * A ML Writer which delegates based on the requested format.
    + */
    +class GeneralMLWriter(stage: PipelineStage) extends MLWriter with Logging {
    +  private var source: String = "internal"
    +
       /**
    -   * Overwrites if the output path already exists.
    +   * Specifies the format of ML export (e.g. PMML, internal, or
    +   * the fully qualified class name for export).
        */
    -  @Since("1.6.0")
    -  def overwrite(): this.type = {
    -    shouldOverwrite = true
    +  @Since("2.3.0")
    +  def format(source: String): this.type = {
    +    this.source = source
         this
       }
     
    +  /**
    +   * Dispatches the save to the correct MLFormat.
    +   */
    +  @Since("2.3.0")
    +  @throws[IOException]("If the input path already exists but overwrite is not enabled.")
    +  @throws[SparkException]("If multiple sources for a given short name format are found.")
    +  override protected def saveImpl(path: String) = {
    +    val loader = Utils.getContextOrSparkClassLoader
    +    val serviceLoader = ServiceLoader.load(classOf[MLFormatRegister], loader)
    +    val stageName = stage.getClass.getName
    +    val targetName = s"${source}+${stageName}"
    +    val formats = serviceLoader.asScala.toList
    +    val shortNames = formats.map(_.shortName())
    +    val writerCls = formats.filter(_.shortName().equalsIgnoreCase(targetName)) match {
    +      // requested name did not match any given registered alias
    +      case Nil =>
    +        Try(loader.loadClass(source)) match {
    +          case Success(writer) =>
    +            // Found the ML writer using the fully qualified path
    +            writer
    +          case Failure(error) =>
    +            throw new SparkException(
    +              s"Could not load requested format $source for $stageName ($targetName) had $formats" +
    +              s"supporting $shortNames", error)
    +        }
    +      case head :: Nil =>
    +        head.getClass
    +      case _ =>
    +        // Multiple sources
    +        throw new SparkException(
    +          s"Multiple writers found for $source+$stageName, try using the class name of the writer")
    +    }
    +    if (classOf[MLWriterFormat].isAssignableFrom(writerCls)) {
    +      val writer = writerCls.newInstance().asInstanceOf[MLWriterFormat]
    +      writer.write(path, sparkSession, optionMap, stage)
    +    } else {
    +      throw new SparkException("ML source $source is not a valid MLWriterFormat")
    --- End diff --
    
    Good catch, I've added a test for this error message.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    So to be clear this doesn't handle the `read` path at all? Would there be a plan to implement a similar read API?
    
    Overall I like the idea of an open API for plugging in model serialization formats. This might be a bit too much to put in for 2.3 though?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML expo...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r158113236
  
    --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala ---
    @@ -994,6 +998,38 @@ class LinearRegressionSuite
           LinearRegressionSuite.allParamSettings, checkModelData)
       }
     
    +  test("pmml export") {
    +    val lr = new LinearRegression()
    +    val model = lr.fit(datasetWithWeight)
    +    def checkModel(pmml: PMML): Unit = {
    +      val dd = pmml.getDataDictionary
    +      assert(dd.getNumberOfFields === 3)
    +      val fields = dd.getDataFields.asScala
    +      assert(fields(0).getName().toString === "field_0")
    +      assert(fields(0).getOpType() == OpType.CONTINUOUS)
    +      val pmmlRegressionModel = pmml.getModels().get(0).asInstanceOf[PMMLRegressionModel]
    +      val pmmlPredictors = pmmlRegressionModel.getRegressionTables.get(0).getNumericPredictors
    +      val pmmlWeights = pmmlPredictors.asScala.map(_.getCoefficient()).toList
    +      assert(pmmlWeights(0) ~== model.coefficients(0) relTol 1E-3)
    +      assert(pmmlWeights(1) ~== model.coefficients(1) relTol 1E-3)
    +    }
    +    testPMMLWrite(sc, model, checkModel)
    +  }
    +
    +  test("unsupported export format") {
    --- End diff --
    
    Sure, I'll put a dummy writer in test so it doesn't clog up our class space.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r161390600
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -126,15 +180,69 @@ abstract class MLWriter extends BaseReadWrite with Logging {
         this
       }
     
    +  // override for Java compatibility
    +  override def session(sparkSession: SparkSession): this.type = super.session(sparkSession)
    +
    +  // override for Java compatibility
    +  override def context(sqlContext: SQLContext): this.type = super.session(sqlContext.sparkSession)
    +}
    +
    +/**
    + * A ML Writer which delegates based on the requested format.
    + */
    +class GeneralMLWriter(stage: PipelineStage) extends MLWriter with Logging {
    +  private var source: String = "internal"
    +
       /**
    -   * Overwrites if the output path already exists.
    +   * Specifies the format of ML export (e.g. PMML, internal, or
    +   * the fully qualified class name for export).
        */
    -  @Since("1.6.0")
    -  def overwrite(): this.type = {
    -    shouldOverwrite = true
    +  @Since("2.3.0")
    +  def format(source: String): this.type = {
    +    this.source = source
         this
       }
     
    +  /**
    +   * Dispatches the save to the correct MLFormat.
    +   */
    +  @Since("2.3.0")
    +  @throws[IOException]("If the input path already exists but overwrite is not enabled.")
    +  @throws[SparkException]("If multiple sources for a given short name format are found.")
    +  override protected def saveImpl(path: String) = {
    +    val loader = Utils.getContextOrSparkClassLoader
    +    val serviceLoader = ServiceLoader.load(classOf[MLFormatRegister], loader)
    +    val stageName = stage.getClass.getName
    +    val targetName = s"${source}+${stageName}"
    +    val formats = serviceLoader.asScala.toList
    +    val shortNames = formats.map(_.shortName())
    +    val writerCls = formats.filter(_.shortName().equalsIgnoreCase(targetName)) match {
    +      // requested name did not match any given registered alias
    +      case Nil =>
    +        Try(loader.loadClass(source)) match {
    +          case Success(writer) =>
    +            // Found the ML writer using the fully qualified path
    +            writer
    +          case Failure(error) =>
    +            throw new SparkException(
    +              s"Could not load requested format $source for $stageName ($targetName) had $formats" +
    +              s"supporting $shortNames", error)
    +        }
    +      case head :: Nil =>
    +        head.getClass
    +      case _ =>
    +        // Multiple sources
    +        throw new SparkException(
    +          s"Multiple writers found for $source+$stageName, try using the class name of the writer")
    +    }
    +    if (classOf[MLWriterFormat].isAssignableFrom(writerCls)) {
    +      val writer = writerCls.newInstance().asInstanceOf[MLWriterFormat]
    --- End diff --
    
    True, we have the same issue with the DataFormat provider though. I don't think there is a way around this while keeping the DF like interface that lets us be pluggable in the way folks want (but if there is a way to require a 0 argument constructor in the concrete class with a trait I'm interested).
    
    I think given the folks who we general expect to be writing these formats that reasonable, but I'll add a comment about this in the doc?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #88534 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88534/testReport)** for PR 19876 at commit [`9075626`](https://github.com/apache/spark/commit/90756266667f708e271f3bf502a67c28c8810fa7).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r160461644
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -126,15 +180,69 @@ abstract class MLWriter extends BaseReadWrite with Logging {
         this
       }
     
    +  // override for Java compatibility
    +  override def session(sparkSession: SparkSession): this.type = super.session(sparkSession)
    +
    +  // override for Java compatibility
    +  override def context(sqlContext: SQLContext): this.type = super.session(sqlContext.sparkSession)
    +}
    +
    +/**
    + * A ML Writer which delegates based on the requested format.
    + */
    +class GeneralMLWriter(stage: PipelineStage) extends MLWriter with Logging {
    +  private var source: String = "internal"
    +
       /**
    -   * Overwrites if the output path already exists.
    +   * Specifies the format of ML export (e.g. PMML, internal, or
    +   * the fully qualified class name for export).
        */
    -  @Since("1.6.0")
    -  def overwrite(): this.type = {
    -    shouldOverwrite = true
    +  @Since("2.3.0")
    +  def format(source: String): this.type = {
    +    this.source = source
         this
       }
     
    +  /**
    +   * Dispatches the save to the correct MLFormat.
    +   */
    +  @Since("2.3.0")
    +  @throws[IOException]("If the input path already exists but overwrite is not enabled.")
    +  @throws[SparkException]("If multiple sources for a given short name format are found.")
    +  override protected def saveImpl(path: String) = {
    +    val loader = Utils.getContextOrSparkClassLoader
    +    val serviceLoader = ServiceLoader.load(classOf[MLFormatRegister], loader)
    +    val stageName = stage.getClass.getName
    +    val targetName = s"${source}+${stageName}"
    --- End diff --
    
    don't need brackets


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r160471845
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala ---
    @@ -710,15 +711,57 @@ class LinearRegressionModel private[ml] (
       }
     
       /**
    -   * Returns a [[org.apache.spark.ml.util.MLWriter]] instance for this ML instance.
    +   * Returns a [[org.apache.spark.ml.util.GeneralMLWriter]] instance for this ML instance.
        *
        * For [[LinearRegressionModel]], this does NOT currently save the training [[summary]].
        * An option to save [[summary]] may be added in the future.
        *
        * This also does not save the [[parent]] currently.
        */
       @Since("1.6.0")
    -  override def write: MLWriter = new LinearRegressionModel.LinearRegressionModelWriter(this)
    +  override def write: GeneralMLWriter = new GeneralMLWriter(this)
    +}
    +
    +/** A writer for LinearRegression that handles the "internal" (or default) format */
    +private class InternalLinearRegressionModelWriter()
    +  extends MLWriterFormat with MLFormatRegister {
    +
    +  override def shortName(): String =
    +    "internal+org.apache.spark.ml.regression.LinearRegressionModel"
    +
    +  private case class Data(intercept: Double, coefficients: Vector, scale: Double)
    +
    +  override def write(path: String, sparkSession: SparkSession,
    +    optionMap: mutable.Map[String, String], stage: PipelineStage): Unit = {
    +    val instance = stage.asInstanceOf[LinearRegressionModel]
    +    val sc = sparkSession.sparkContext
    +    // Save metadata and Params
    +    DefaultParamsWriter.saveMetadata(instance, path, sc)
    +    // Save model data: intercept, coefficients, scale
    +    val data = Data(instance.intercept, instance.coefficients, instance.scale)
    +    val dataPath = new Path(path, "data").toString
    +    sparkSession.createDataFrame(Seq(data)).repartition(1).write.parquet(dataPath)
    +  }
    +}
    +
    +/** A writer for LinearRegression that handles the "pmml" format */
    +private class PMMLLinearRegressionModelWriter()
    --- End diff --
    
    I could be wrong, but I think we prefer just omitting the `()`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    I'm going to update this tomorrow, but if no one has anything by EOW would folks be OK with this as an experimental developer API for 2.3? cc @JoshRosen ?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-23783][SPARK-11239] Add PMML export to...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r177202362
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala ---
    @@ -710,15 +711,58 @@ class LinearRegressionModel private[ml] (
       }
     
       /**
    -   * Returns a [[org.apache.spark.ml.util.MLWriter]] instance for this ML instance.
    +   * Returns a [[org.apache.spark.ml.util.GeneralMLWriter]] instance for this ML instance.
        *
        * For [[LinearRegressionModel]], this does NOT currently save the training [[summary]].
        * An option to save [[summary]] may be added in the future.
        *
        * This also does not save the [[parent]] currently.
        */
       @Since("1.6.0")
    -  override def write: MLWriter = new LinearRegressionModel.LinearRegressionModelWriter(this)
    +  override def write: GeneralMLWriter = new GeneralMLWriter(this)
    +}
    +
    +/** A writer for LinearRegression that handles the "internal" (or default) format */
    +private class InternalLinearRegressionModelWriter
    +  extends MLWriterFormat with MLFormatRegister {
    +
    +  override def format(): String = "internal"
    +  override def stageName(): String = "org.apache.spark.ml.regression.LinearRegressionModel"
    +
    +  private case class Data(intercept: Double, coefficients: Vector, scale: Double)
    +
    +  override def write(path: String, sparkSession: SparkSession,
    +    optionMap: mutable.Map[String, String], stage: PipelineStage): Unit = {
    +    val instance = stage.asInstanceOf[LinearRegressionModel]
    +    val sc = sparkSession.sparkContext
    +    // Save metadata and Params
    +    DefaultParamsWriter.saveMetadata(instance, path, sc)
    +    // Save model data: intercept, coefficients, scale
    +    val data = Data(instance.intercept, instance.coefficients, instance.scale)
    +    val dataPath = new Path(path, "data").toString
    +    sparkSession.createDataFrame(Seq(data)).repartition(1).write.parquet(dataPath)
    +  }
    +}
    +
    +/** A writer for LinearRegression that handles the "pmml" format */
    +private class PMMLLinearRegressionModelWriter
    +    extends MLWriterFormat with MLFormatRegister {
    --- End diff --
    
    I've included this in https://github.com/apache/spark/pull/20907


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #85204 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85204/testReport)** for PR 19876 at commit [`6e9cdc3`](https://github.com/apache/spark/commit/6e9cdc37d63af8631e2595a6e20b67f495758d39).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML expo...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r156370588
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -85,12 +87,55 @@ private[util] sealed trait BaseReadWrite {
       protected final def sc: SparkContext = sparkSession.sparkContext
     }
     
    +/**
    + * ML export formats for should implement this trait so that users can specify a shortname rather
    + * than the fully qualified class name of the exporter.
    + *
    + * A new instance of this class will be instantiated each time a DDL call is made.
    + *
    + * @since 2.3.0
    + */
    +@InterfaceStability.Evolving
    +trait MLFormatRegister {
    +  /**
    +   * The string that represents the format that this data source provider uses. This is
    +   * overridden by children to provide a nice alias for the data source. For example:
    +   *
    +   * {{{
    +   *   override def shortName(): String =
    +   *       "pmml+org.apache.spark.ml.regression.LinearRegressionModel"
    +   * }}}
    +   * Indicates that this format is capable of saving Spark's own LinearRegressionModel in pmml.
    +   *
    +   * Format discovery is done using a ServiceLoader so make sure to list your format in
    +   * META-INF/services.
    +   * @since 2.3.0
    +   */
    +  def shortName(): String
    +}
    +
    +/**
    + * Implemented by objects that provide ML exportability.
    + *
    + * A new instance of this class will be instantiated each time a DDL call is made.
    + *
    + * @since 2.3.0
    + */
    +@InterfaceStability.Evolving
    +trait MLWriterFormat {
    +  /**
    +   * Function write the provided pipeline stage out.
    +   */
    +  def write(path: String, session: SparkSession, optionMap: mutable.Map[String, String],
    +    stage: PipelineStage)
    --- End diff --
    
    return type?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #85207 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85207/testReport)** for PR 19876 at commit [`b8844c7`](https://github.com/apache/spark/commit/b8844c75b7b0278f19cd340c3c036935c43feef4).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML expo...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r158114844
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -126,15 +180,69 @@ abstract class MLWriter extends BaseReadWrite with Logging {
         this
       }
     
    +  // override for Java compatibility
    +  override def session(sparkSession: SparkSession): this.type = super.session(sparkSession)
    +
    +  // override for Java compatibility
    +  override def context(sqlContext: SQLContext): this.type = super.session(sqlContext.sparkSession)
    +}
    +
    +/**
    + * A ML Writer which delegates based on the requested format.
    + */
    +class GeneralMLWriter(stage: PipelineStage) extends MLWriter with Logging {
    --- End diff --
    
    So I don't think that belongs in the base GeneralMLWriter, but we could make a trait for writers which support PMML to mix in?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #84470 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84470/testReport)** for PR 19876 at commit [`8b1c752`](https://github.com/apache/spark/commit/8b1c7525cadd084686774b584ab376958ded2eb0).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #86228 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86228/testReport)** for PR 19876 at commit [`4047239`](https://github.com/apache/spark/commit/40472390fe2932900c4d788bf2e19bcfbff12c92).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r160463225
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -85,12 +87,55 @@ private[util] sealed trait BaseReadWrite {
       protected final def sc: SparkContext = sparkSession.sparkContext
     }
     
    +/**
    + * ML export formats for should implement this trait so that users can specify a shortname rather
    + * than the fully qualified class name of the exporter.
    + *
    + * A new instance of this class will be instantiated each time a DDL call is made.
    + *
    + * @since 2.3.0
    + */
    +@InterfaceStability.Evolving
    +trait MLFormatRegister {
    +  /**
    +   * The string that represents the format that this data source provider uses. This is
    +   * overridden by children to provide a nice alias for the data source. For example:
    --- End diff --
    
    "data source" -> "model format"?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84473/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r176821909
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -86,7 +88,80 @@ private[util] sealed trait BaseReadWrite {
     }
     
     /**
    - * Abstract class for utility classes that can save ML instances.
    + * Implemented by objects that provide ML exportability.
    + *
    + * A new instance of this class will be instantiated each time a save call is made.
    + *
    + * Must have a valid zero argument constructor which will be called to instantiate.
    + *
    + * @since 2.3.0
    + */
    +@InterfaceStability.Unstable
    +@Since("2.3.0")
    +trait MLWriterFormat {
    +  /**
    +   * Function to write the provided pipeline stage out.
    +   *
    +   * @param path  The path to write the result out to.
    +   * @param session  SparkSession associated with the write request.
    +   * @param optionMap  User provided options stored as strings.
    +   * @param stage  The pipeline stage to be saved.
    +   */
    +  @Since("2.3.0")
    +  def write(path: String, session: SparkSession, optionMap: mutable.Map[String, String],
    +    stage: PipelineStage): Unit
    +}
    +
    +/**
    + * ML export formats for should implement this trait so that users can specify a shortname rather
    + * than the fully qualified class name of the exporter.
    + *
    + * A new instance of this class will be instantiated each time a save call is made.
    --- End diff --
    
    Add a comment about zero arg constructor requirement


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85204/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-23783][SPARK-11239] Add PMML export to...

Posted by goungoun <gi...@git.apache.org>.
Github user goungoun commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r176911201
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala ---
    @@ -710,15 +711,58 @@ class LinearRegressionModel private[ml] (
       }
     
       /**
    -   * Returns a [[org.apache.spark.ml.util.MLWriter]] instance for this ML instance.
    +   * Returns a [[org.apache.spark.ml.util.GeneralMLWriter]] instance for this ML instance.
        *
        * For [[LinearRegressionModel]], this does NOT currently save the training [[summary]].
        * An option to save [[summary]] may be added in the future.
        *
        * This also does not save the [[parent]] currently.
        */
       @Since("1.6.0")
    -  override def write: MLWriter = new LinearRegressionModel.LinearRegressionModelWriter(this)
    +  override def write: GeneralMLWriter = new GeneralMLWriter(this)
    +}
    +
    +/** A writer for LinearRegression that handles the "internal" (or default) format */
    +private class InternalLinearRegressionModelWriter
    +  extends MLWriterFormat with MLFormatRegister {
    +
    +  override def format(): String = "internal"
    +  override def stageName(): String = "org.apache.spark.ml.regression.LinearRegressionModel"
    +
    +  private case class Data(intercept: Double, coefficients: Vector, scale: Double)
    +
    +  override def write(path: String, sparkSession: SparkSession,
    +    optionMap: mutable.Map[String, String], stage: PipelineStage): Unit = {
    +    val instance = stage.asInstanceOf[LinearRegressionModel]
    +    val sc = sparkSession.sparkContext
    +    // Save metadata and Params
    +    DefaultParamsWriter.saveMetadata(instance, path, sc)
    +    // Save model data: intercept, coefficients, scale
    +    val data = Data(instance.intercept, instance.coefficients, instance.scale)
    +    val dataPath = new Path(path, "data").toString
    +    sparkSession.createDataFrame(Seq(data)).repartition(1).write.parquet(dataPath)
    +  }
    +}
    +
    +/** A writer for LinearRegression that handles the "pmml" format */
    +private class PMMLLinearRegressionModelWriter
    +    extends MLWriterFormat with MLFormatRegister {
    --- End diff --
    
    Should be two space indentation
    `    extends MLWriterFormat with MLFormatRegister {` 



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r160484001
  
    --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala ---
    @@ -1044,6 +1056,50 @@ class LinearRegressionSuite extends MLTest with DefaultReadWriteTest {
           LinearRegressionSuite.allParamSettings, checkModelData)
       }
     
    +  test("pmml export") {
    +    val lr = new LinearRegression()
    +    val model = lr.fit(datasetWithWeight)
    +    def checkModel(pmml: PMML): Unit = {
    +      val dd = pmml.getDataDictionary
    +      assert(dd.getNumberOfFields === 3)
    +      val fields = dd.getDataFields.asScala
    +      assert(fields(0).getName().toString === "field_0")
    +      assert(fields(0).getOpType() == OpType.CONTINUOUS)
    +      val pmmlRegressionModel = pmml.getModels().get(0).asInstanceOf[PMMLRegressionModel]
    +      val pmmlPredictors = pmmlRegressionModel.getRegressionTables.get(0).getNumericPredictors
    +      val pmmlWeights = pmmlPredictors.asScala.map(_.getCoefficient()).toList
    +      assert(pmmlWeights(0) ~== model.coefficients(0) relTol 1E-3)
    +      assert(pmmlWeights(1) ~== model.coefficients(1) relTol 1E-3)
    +    }
    +    testPMMLWrite(sc, model, checkModel)
    +  }
    +
    +  test("unsupported export format") {
    +    val lr = new LinearRegression()
    +    val model = lr.fit(datasetWithWeight)
    +    intercept[SparkException] {
    --- End diff --
    
    Doesn't this and the one below it test the same thing? I think we could remove the first one.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r160462794
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -85,12 +87,55 @@ private[util] sealed trait BaseReadWrite {
       protected final def sc: SparkContext = sparkSession.sparkContext
     }
     
    +/**
    + * ML export formats for should implement this trait so that users can specify a shortname rather
    + * than the fully qualified class name of the exporter.
    + *
    + * A new instance of this class will be instantiated each time a DDL call is made.
    + *
    + * @since 2.3.0
    + */
    +@InterfaceStability.Evolving
    +trait MLFormatRegister {
    +  /**
    +   * The string that represents the format that this data source provider uses. This is
    +   * overridden by children to provide a nice alias for the data source. For example:
    +   *
    +   * {{{
    +   *   override def shortName(): String =
    +   *       "pmml+org.apache.spark.ml.regression.LinearRegressionModel"
    --- End diff --
    
    what about making a second abstract field `def stageName(): String`, instead of having it packed into one string?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #84474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84474/testReport)** for PR 19876 at commit [`b8362a4`](https://github.com/apache/spark/commit/b8362a463c92f902c12a6efcb7d82e5324e64e24).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML expo...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r156388871
  
    --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala ---
    @@ -994,6 +998,38 @@ class LinearRegressionSuite
           LinearRegressionSuite.allParamSettings, checkModelData)
       }
     
    +  test("pmml export") {
    +    val lr = new LinearRegression()
    +    val model = lr.fit(datasetWithWeight)
    +    def checkModel(pmml: PMML): Unit = {
    +      val dd = pmml.getDataDictionary
    +      assert(dd.getNumberOfFields === 3)
    +      val fields = dd.getDataFields.asScala
    +      assert(fields(0).getName().toString === "field_0")
    +      assert(fields(0).getOpType() == OpType.CONTINUOUS)
    +      val pmmlRegressionModel = pmml.getModels().get(0).asInstanceOf[PMMLRegressionModel]
    +      val pmmlPredictors = pmmlRegressionModel.getRegressionTables.get(0).getNumericPredictors
    +      val pmmlWeights = pmmlPredictors.asScala.map(_.getCoefficient()).toList
    +      assert(pmmlWeights(0) ~== model.coefficients(0) relTol 1E-3)
    +      assert(pmmlWeights(1) ~== model.coefficients(1) relTol 1E-3)
    +    }
    +    testPMMLWrite(sc, model, checkModel)
    +  }
    +
    +  test("unsupported export format") {
    --- End diff --
    
    Would be great to have a test that verifies that this works with third party implementations. Specifically, that something like `model.write.format("org.apache.spark.ml.MyDummyWriter").save(path)` works.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #84426 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84426/testReport)** for PR 19876 at commit [`de86190`](https://github.com/apache/spark/commit/de8619098eeb01ff86b54753f27c29729935bb94).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84426/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #86229 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86229/testReport)** for PR 19876 at commit [`cd330f3`](https://github.com/apache/spark/commit/cd330f3f37796f61b8b62a24c5600cf2868b6470).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86226/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Will do a last pass myself and merge on Friday if no one else has opinions.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    @MLnick I think the read path could follow a similar approach, but for models which we export from Spark and wish to load back into Spark the internal format is probably the best option.
    
    As far as options, this keeps the same stringly typed options interface as we have with Datasources, and individual writers are free to specialize and add their own special methods.
    
    I'm open to the idea of punting this, but we've punted PMML export since we introduced ML pipelines and users are still asking for general export support so the feature req hasn't gone away either. If we think the design needs another pass that's fine, but I'd like to suggest that this is probably a good starting block from which we can add more things on top off.
    
    If folks agree the foundation isn't bad I think putting it in for 2.3 would be useful (and it's not like we haven't had a long time to consider this design). This specific PR has been open since Dec 4th but we've been looking at variants on this idea for years.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r176821060
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -86,7 +88,80 @@ private[util] sealed trait BaseReadWrite {
     }
     
     /**
    - * Abstract class for utility classes that can save ML instances.
    + * Implemented by objects that provide ML exportability.
    + *
    + * A new instance of this class will be instantiated each time a save call is made.
    + *
    + * Must have a valid zero argument constructor which will be called to instantiate.
    + *
    + * @since 2.3.0
    --- End diff --
    
    Need to update since annotations to 2.4.0


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    @sethah So I'm hesitant to push an API without an implementation to make sure its actually usable for our goal. But I'm fine splitting it out into a separate PR.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r158434850
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala ---
    @@ -554,7 +555,49 @@ class LinearRegressionModel private[ml] (
        * This also does not save the [[parent]] currently.
        */
       @Since("1.6.0")
    -  override def write: MLWriter = new LinearRegressionModel.LinearRegressionModelWriter(this)
    +  override def write: GeneralMLWriter = new GeneralMLWriter(this)
    --- End diff --
    
    fixed


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #86228 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86228/testReport)** for PR 19876 at commit [`4047239`](https://github.com/apache/spark/commit/40472390fe2932900c4d788bf2e19bcfbff12c92).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86230/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    So now that it looks like 2.3 is pretty much wrapped up do folks have any thoughts? @mlnick @jkbradley @sethah ?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #84473 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84473/testReport)** for PR 19876 at commit [`72b509f`](https://github.com/apache/spark/commit/72b509ff1919c7c82ab2909ffb5c9a7596e52e6c).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r160503466
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -126,15 +180,69 @@ abstract class MLWriter extends BaseReadWrite with Logging {
         this
       }
     
    +  // override for Java compatibility
    +  override def session(sparkSession: SparkSession): this.type = super.session(sparkSession)
    --- End diff --
    
    since tags here


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r160461560
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -85,12 +87,55 @@ private[util] sealed trait BaseReadWrite {
       protected final def sc: SparkContext = sparkSession.sparkContext
     }
     
    +/**
    + * ML export formats for should implement this trait so that users can specify a shortname rather
    + * than the fully qualified class name of the exporter.
    + *
    + * A new instance of this class will be instantiated each time a DDL call is made.
    --- End diff --
    
    Was this supposed to be retained from the `DataSourceRegister`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    also maybe @dbtsai ?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r160502536
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -85,12 +87,55 @@ private[util] sealed trait BaseReadWrite {
       protected final def sc: SparkContext = sparkSession.sparkContext
     }
     
    +/**
    + * ML export formats for should implement this trait so that users can specify a shortname rather
    + * than the fully qualified class name of the exporter.
    + *
    + * A new instance of this class will be instantiated each time a DDL call is made.
    + *
    + * @since 2.3.0
    + */
    +@InterfaceStability.Evolving
    +trait MLFormatRegister {
    +  /**
    +   * The string that represents the format that this data source provider uses. This is
    +   * overridden by children to provide a nice alias for the data source. For example:
    +   *
    +   * {{{
    +   *   override def shortName(): String =
    +   *       "pmml+org.apache.spark.ml.regression.LinearRegressionModel"
    +   * }}}
    +   * Indicates that this format is capable of saving Spark's own LinearRegressionModel in pmml.
    +   *
    +   * Format discovery is done using a ServiceLoader so make sure to list your format in
    +   * META-INF/services.
    +   * @since 2.3.0
    +   */
    +  def shortName(): String
    +}
    +
    +/**
    + * Implemented by objects that provide ML exportability.
    + *
    + * A new instance of this class will be instantiated each time a DDL call is made.
    + *
    + * @since 2.3.0
    + */
    +@InterfaceStability.Evolving
    +trait MLWriterFormat {
    +  /**
    +   * Function write the provided pipeline stage out.
    --- End diff --
    
    Should add a full doc here with param annotations. Also should it be "Function to write ..."?
      


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-23783][SPARK-11239] Add PMML export to...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r177189453
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala ---
    @@ -710,15 +711,58 @@ class LinearRegressionModel private[ml] (
       }
     
       /**
    -   * Returns a [[org.apache.spark.ml.util.MLWriter]] instance for this ML instance.
    +   * Returns a [[org.apache.spark.ml.util.GeneralMLWriter]] instance for this ML instance.
        *
        * For [[LinearRegressionModel]], this does NOT currently save the training [[summary]].
        * An option to save [[summary]] may be added in the future.
        *
        * This also does not save the [[parent]] currently.
        */
       @Since("1.6.0")
    -  override def write: MLWriter = new LinearRegressionModel.LinearRegressionModelWriter(this)
    +  override def write: GeneralMLWriter = new GeneralMLWriter(this)
    +}
    +
    +/** A writer for LinearRegression that handles the "internal" (or default) format */
    +private class InternalLinearRegressionModelWriter
    +  extends MLWriterFormat with MLFormatRegister {
    +
    +  override def format(): String = "internal"
    +  override def stageName(): String = "org.apache.spark.ml.regression.LinearRegressionModel"
    +
    +  private case class Data(intercept: Double, coefficients: Vector, scale: Double)
    +
    +  override def write(path: String, sparkSession: SparkSession,
    +    optionMap: mutable.Map[String, String], stage: PipelineStage): Unit = {
    +    val instance = stage.asInstanceOf[LinearRegressionModel]
    +    val sc = sparkSession.sparkContext
    +    // Save metadata and Params
    +    DefaultParamsWriter.saveMetadata(instance, path, sc)
    +    // Save model data: intercept, coefficients, scale
    +    val data = Data(instance.intercept, instance.coefficients, instance.scale)
    +    val dataPath = new Path(path, "data").toString
    +    sparkSession.createDataFrame(Seq(data)).repartition(1).write.parquet(dataPath)
    +  }
    +}
    +
    +/** A writer for LinearRegression that handles the "pmml" format */
    +private class PMMLLinearRegressionModelWriter
    +    extends MLWriterFormat with MLFormatRegister {
    --- End diff --
    
    Thanks for pointing this out, I'll fix it in a follow up.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    So @MLnick / @sethah I'd like to put something like this in before we roll 2.3 RC1. Do y'all have any feedback before that?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #84426 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84426/testReport)** for PR 19876 at commit [`de86190`](https://github.com/apache/spark/commit/de8619098eeb01ff86b54753f27c29729935bb94).
     * This patch **fails to generate documentation**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `trait PMMLReadWriteTest extends TempDirectory `


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r161391241
  
    --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala ---
    @@ -1044,6 +1056,50 @@ class LinearRegressionSuite extends MLTest with DefaultReadWriteTest {
           LinearRegressionSuite.allParamSettings, checkModelData)
       }
     
    +  test("pmml export") {
    +    val lr = new LinearRegression()
    +    val model = lr.fit(datasetWithWeight)
    +    def checkModel(pmml: PMML): Unit = {
    +      val dd = pmml.getDataDictionary
    +      assert(dd.getNumberOfFields === 3)
    +      val fields = dd.getDataFields.asScala
    +      assert(fields(0).getName().toString === "field_0")
    +      assert(fields(0).getOpType() == OpType.CONTINUOUS)
    +      val pmmlRegressionModel = pmml.getModels().get(0).asInstanceOf[PMMLRegressionModel]
    +      val pmmlPredictors = pmmlRegressionModel.getRegressionTables.get(0).getNumericPredictors
    +      val pmmlWeights = pmmlPredictors.asScala.map(_.getCoefficient()).toList
    +      assert(pmmlWeights(0) ~== model.coefficients(0) relTol 1E-3)
    +      assert(pmmlWeights(1) ~== model.coefficients(1) relTol 1E-3)
    +    }
    +    testPMMLWrite(sc, model, checkModel)
    +  }
    +
    +  test("unsupported export format") {
    +    val lr = new LinearRegression()
    +    val model = lr.fit(datasetWithWeight)
    +    intercept[SparkException] {
    +      model.write.format("boop").save("boop")
    +    }
    +    intercept[SparkException] {
    +      model.write.format("com.holdenkarau.boop").save("boop")
    +    }
    +    withClue("ML source org.apache.spark.SparkContext is not a valid MLWriterFormat") {
    +      intercept[SparkException] {
    +        model.write.format("org.apache.spark.SparkContext").save("boop2")
    +      }
    +    }
    +  }
    +
    +  test("dummy export format is called") {
    +    val lr = new LinearRegression()
    +    val model = lr.fit(datasetWithWeight)
    +    withClue("Dummy writer doesn't write") {
    +      intercept[Exception] {
    --- End diff --
    
    good point


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Do we want to think about an `options` / `option` interface too? I'm not that familiar with whether there could be important options for PMML export, but custom user formats may need it (I could envisage a format for exporting to plain JSON, H2O say, TensorFlow, etc etc that may need to customize the behavior through options)


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #86230 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86230/testReport)** for PR 19876 at commit [`41312e7`](https://github.com/apache/spark/commit/41312e7fcd6d706323fecb828776fced5e5a769c).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    @holdenk Do you mind leaving some comments on the intentions/benefits of this new API for the benefit of other reviewers? For example, what use cases may exist - adding third party PFA support (other third party export tools?), and also why we need to add PMML support when there are already tools that do this [jpmml-sparkml](https://github.com/jpmml/jpmml-sparkml). 
    
    Also, this is two changes in one PR: adding an API for generic model export and adding PMML to LinearRegression. I think it makes sense to separate the two, and just focus on the new API here. What do you think?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    You can see some of the past discussion on https://github.com/apache/spark/pull/9207 cc @sethah @MLnick 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #86226 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86226/testReport)** for PR 19876 at commit [`6411054`](https://github.com/apache/spark/commit/6411054d12ea3d4b98bdc7ad409b392a8cc16a8f).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86229/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    @sethah: want to publish the comments from when we were chatting in Singapore? :p 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r160506592
  
    --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala ---
    @@ -1044,6 +1056,50 @@ class LinearRegressionSuite extends MLTest with DefaultReadWriteTest {
           LinearRegressionSuite.allParamSettings, checkModelData)
       }
     
    +  test("pmml export") {
    +    val lr = new LinearRegression()
    +    val model = lr.fit(datasetWithWeight)
    +    def checkModel(pmml: PMML): Unit = {
    +      val dd = pmml.getDataDictionary
    +      assert(dd.getNumberOfFields === 3)
    +      val fields = dd.getDataFields.asScala
    +      assert(fields(0).getName().toString === "field_0")
    +      assert(fields(0).getOpType() == OpType.CONTINUOUS)
    +      val pmmlRegressionModel = pmml.getModels().get(0).asInstanceOf[PMMLRegressionModel]
    +      val pmmlPredictors = pmmlRegressionModel.getRegressionTables.get(0).getNumericPredictors
    +      val pmmlWeights = pmmlPredictors.asScala.map(_.getCoefficient()).toList
    +      assert(pmmlWeights(0) ~== model.coefficients(0) relTol 1E-3)
    +      assert(pmmlWeights(1) ~== model.coefficients(1) relTol 1E-3)
    +    }
    +    testPMMLWrite(sc, model, checkModel)
    +  }
    +
    +  test("unsupported export format") {
    +    val lr = new LinearRegression()
    +    val model = lr.fit(datasetWithWeight)
    +    intercept[SparkException] {
    +      model.write.format("boop").save("boop")
    +    }
    +    intercept[SparkException] {
    +      model.write.format("com.holdenkarau.boop").save("boop")
    +    }
    +    withClue("ML source org.apache.spark.SparkContext is not a valid MLWriterFormat") {
    +      intercept[SparkException] {
    +        model.write.format("org.apache.spark.SparkContext").save("boop2")
    +      }
    +    }
    +  }
    +
    +  test("dummy export format is called") {
    --- End diff --
    
    We can also add tests for the `MLFormatRegister` similar to `DDLSourceLoadSuite`. Just add a `META-INF/services/` directory to `src/test/resources/`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84470/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #84470 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84470/testReport)** for PR 19876 at commit [`8b1c752`](https://github.com/apache/spark/commit/8b1c7525cadd084686774b584ab376958ded2eb0).
     * This patch **fails to generate documentation**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r160463657
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -126,15 +180,69 @@ abstract class MLWriter extends BaseReadWrite with Logging {
         this
       }
     
    +  // override for Java compatibility
    +  override def session(sparkSession: SparkSession): this.type = super.session(sparkSession)
    +
    +  // override for Java compatibility
    +  override def context(sqlContext: SQLContext): this.type = super.session(sqlContext.sparkSession)
    +}
    +
    +/**
    + * A ML Writer which delegates based on the requested format.
    + */
    +class GeneralMLWriter(stage: PipelineStage) extends MLWriter with Logging {
    +  private var source: String = "internal"
    +
       /**
    -   * Overwrites if the output path already exists.
    +   * Specifies the format of ML export (e.g. PMML, internal, or
    --- End diff --
    
    change to `e.g. "pmml", "internal", or the fully qualified class name for export)."`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #84474 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84474/testReport)** for PR 19876 at commit [`b8362a4`](https://github.com/apache/spark/commit/b8362a463c92f902c12a6efcb7d82e5324e64e24).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    March 15th is soon, any thoughts  @MLnick @jkbradley @sethah ?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML expo...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r156389238
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -126,15 +180,69 @@ abstract class MLWriter extends BaseReadWrite with Logging {
         this
       }
     
    +  // override for Java compatibility
    +  override def session(sparkSession: SparkSession): this.type = super.session(sparkSession)
    +
    +  // override for Java compatibility
    +  override def context(sqlContext: SQLContext): this.type = super.session(sqlContext.sparkSession)
    +}
    +
    +/**
    + * A ML Writer which delegates based on the requested format.
    + */
    +class GeneralMLWriter(stage: PipelineStage) extends MLWriter with Logging {
    --- End diff --
    
    Perhaps for another PR, but maybe we could add a method here:
    
    ```scala
      def pmml(path: String): Unit = {
        this.source = "pmml"
        save(path)
      }
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85207/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #85204 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85204/testReport)** for PR 19876 at commit [`6e9cdc3`](https://github.com/apache/spark/commit/6e9cdc37d63af8631e2595a6e20b67f495758d39).
     * This patch **fails Scala style tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    re-ping @MLnick + ping @jkbradley , thoughts?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #86230 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86230/testReport)** for PR 19876 at commit [`41312e7`](https://github.com/apache/spark/commit/41312e7fcd6d706323fecb828776fced5e5a769c).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML expo...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r155157785
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -126,15 +180,69 @@ abstract class MLWriter extends BaseReadWrite with Logging {
         this
       }
     
    +  // override for Java compatibility
    +  override def session(sparkSession: SparkSession): this.type = super.session(sparkSession)
    +
    +  // override for Java compatibility
    +  override def context(sqlContext: SQLContext): this.type = super.session(sqlContext.sparkSession)
    +}
    +
    +/**
    + * A ML Writer which delegates based on the requested format.
    + */
    +class GeneralMLWriter(stage: PipelineStage) extends MLWriter with Logging {
    +  private var source: String = "internal"
    +
       /**
    -   * Overwrites if the output path already exists.
    +   * Specifies the format of ML export (e.g. PMML, internal, or
    +   * the fully qualified class name for export).
        */
    -  @Since("1.6.0")
    -  def overwrite(): this.type = {
    -    shouldOverwrite = true
    +  @Since("2.3.0")
    +  def format(source: String): this.type = {
    +    this.source = source
         this
       }
     
    +  /**
    +   * Dispatches the save to the correct MLFormat.
    +   */
    +  @Since("2.3.0")
    +  @throws[IOException]("If the input path already exists but overwrite is not enabled.")
    +  @throws[SparkException]("If multiple sources for a given short name format are found.")
    +  override protected def saveImpl(path: String) = {
    +    val loader = Utils.getContextOrSparkClassLoader
    +    val serviceLoader = ServiceLoader.load(classOf[MLFormatRegister], loader)
    +    val stageName = stage.getClass.getName
    +    val targetName = s"${source}+${stageName}"
    +    val formats = serviceLoader.asScala.toList
    +    val shortNames = formats.map(_.shortName())
    +    val writerCls = formats.filter(_.shortName().equalsIgnoreCase(targetName)) match {
    +      // requested name did not match any given registered alias
    +      case Nil =>
    +        Try(loader.loadClass(source)) match {
    +          case Success(writer) =>
    +            // Found the ML writer using the fully qualified path
    +            writer
    +          case Failure(error) =>
    +            throw new SparkException(
    +              s"Could not load requested format $source for $stageName ($targetName) had $formats" +
    +              s"supporting $shortNames", error)
    +        }
    +      case head :: Nil =>
    +        head.getClass
    +      case _ =>
    +        // Multiple sources
    +        throw new SparkException(
    +          s"Multiple writers found for $source+$stageName, try using the class name of the writer")
    +    }
    +    if (classOf[MLWriterFormat].isAssignableFrom(writerCls)) {
    +      val writer = writerCls.newInstance().asInstanceOf[MLWriterFormat]
    +      writer.write(path, sparkSession, optionMap, stage)
    +    } else {
    +      throw new SparkException("ML source $source is not a valid MLWriterFormat")
    --- End diff --
    
    nit: need string interpolation here


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    LGTM pending Jenkins will merge.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #88534 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88534/testReport)** for PR 19876 at commit [`9075626`](https://github.com/apache/spark/commit/90756266667f708e271f3bf502a67c28c8810fa7).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-23783][SPARK-11239] Add PMML export to Spark ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #88546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88546/testReport)** for PR 19876 at commit [`cb6fd70`](https://github.com/apache/spark/commit/cb6fd70d0c61b6477f7514431ee2e1c097ec0aff).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #86229 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86229/testReport)** for PR 19876 at commit [`cd330f3`](https://github.com/apache/spark/commit/cd330f3f37796f61b8b62a24c5600cf2868b6470).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r160503640
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -85,12 +87,55 @@ private[util] sealed trait BaseReadWrite {
       protected final def sc: SparkContext = sparkSession.sparkContext
     }
     
    +/**
    + * ML export formats for should implement this trait so that users can specify a shortname rather
    + * than the fully qualified class name of the exporter.
    + *
    + * A new instance of this class will be instantiated each time a DDL call is made.
    + *
    + * @since 2.3.0
    + */
    +@InterfaceStability.Evolving
    +trait MLFormatRegister {
    +  /**
    +   * The string that represents the format that this data source provider uses. This is
    +   * overridden by children to provide a nice alias for the data source. For example:
    +   *
    +   * {{{
    +   *   override def shortName(): String =
    +   *       "pmml+org.apache.spark.ml.regression.LinearRegressionModel"
    +   * }}}
    +   * Indicates that this format is capable of saving Spark's own LinearRegressionModel in pmml.
    +   *
    +   * Format discovery is done using a ServiceLoader so make sure to list your format in
    +   * META-INF/services.
    +   * @since 2.3.0
    +   */
    +  def shortName(): String
    +}
    +
    +/**
    + * Implemented by objects that provide ML exportability.
    + *
    + * A new instance of this class will be instantiated each time a DDL call is made.
    + *
    + * @since 2.3.0
    + */
    +@InterfaceStability.Evolving
    +trait MLWriterFormat {
    --- End diff --
    
    do we need the actual since annotations here, though?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1727/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r159018510
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -85,12 +87,55 @@ private[util] sealed trait BaseReadWrite {
       protected final def sc: SparkContext = sparkSession.sparkContext
     }
     
    +/**
    + * ML export formats for should implement this trait so that users can specify a shortname rather
    + * than the fully qualified class name of the exporter.
    + *
    + * A new instance of this class will be instantiated each time a DDL call is made.
    + *
    + * @since 2.3.0
    + */
    +@InterfaceStability.Evolving
    +trait MLFormatRegister {
    +  /**
    +   * The string that represents the format that this data source provider uses. This is
    +   * overridden by children to provide a nice alias for the data source. For example:
    +   *
    +   * {{{
    +   *   override def shortName(): String =
    +   *       "pmml+org.apache.spark.ml.regression.LinearRegressionModel"
    +   * }}}
    +   * Indicates that this format is capable of saving Spark's own LinearRegressionModel in pmml.
    +   *
    +   * Format discovery is done using a ServiceLoader so make sure to list your format in
    +   * META-INF/services.
    +   * @since 2.3.0
    +   */
    +  def shortName(): String
    +}
    +
    +/**
    + * Implemented by objects that provide ML exportability.
    + *
    + * A new instance of this class will be instantiated each time a DDL call is made.
    + *
    + * @since 2.3.0
    + */
    +@InterfaceStability.Evolving
    +trait MLWriterFormat {
    +  /**
    +   * Function write the provided pipeline stage out.
    +   */
    +  def write(path: String, session: SparkSession, optionMap: mutable.Map[String, String],
    +    stage: PipelineStage): Unit
    +}
    +
     /**
      * Abstract class for utility classes that can save ML instances.
      */
    +@deprecated("Use GeneralMLWriter instead. Will be removed in Spark 3.0.0", "2.3.0")
    --- End diff --
    
    I'm debating if this should be deprecated in 2.4 and just have this as a new option in 2.3. What do you think @sethah / @MLnick ?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r176826923
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -86,7 +88,80 @@ private[util] sealed trait BaseReadWrite {
     }
     
     /**
    - * Abstract class for utility classes that can save ML instances.
    + * Implemented by objects that provide ML exportability.
    + *
    + * A new instance of this class will be instantiated each time a save call is made.
    + *
    + * Must have a valid zero argument constructor which will be called to instantiate.
    + *
    + * @since 2.3.0
    + */
    +@InterfaceStability.Unstable
    +@Since("2.3.0")
    +trait MLWriterFormat {
    +  /**
    +   * Function to write the provided pipeline stage out.
    +   *
    +   * @param path  The path to write the result out to.
    +   * @param session  SparkSession associated with the write request.
    +   * @param optionMap  User provided options stored as strings.
    +   * @param stage  The pipeline stage to be saved.
    +   */
    +  @Since("2.3.0")
    +  def write(path: String, session: SparkSession, optionMap: mutable.Map[String, String],
    +    stage: PipelineStage): Unit
    +}
    +
    +/**
    + * ML export formats for should implement this trait so that users can specify a shortname rather
    + * than the fully qualified class name of the exporter.
    + *
    + * A new instance of this class will be instantiated each time a save call is made.
    --- End diff --
    
    done


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    See some related thoughts in how to support Spark in kubeflow: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/119


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88534/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84474/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #85207 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85207/testReport)** for PR 19876 at commit [`b8844c7`](https://github.com/apache/spark/commit/b8844c75b7b0278f19cd340c3c036935c43feef4).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-23783][SPARK-11239] Add PMML export to Spark ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML expo...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r156381361
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala ---
    @@ -554,7 +555,49 @@ class LinearRegressionModel private[ml] (
        * This also does not save the [[parent]] currently.
        */
       @Since("1.6.0")
    -  override def write: MLWriter = new LinearRegressionModel.LinearRegressionModelWriter(this)
    +  override def write: GeneralMLWriter = new GeneralMLWriter(this)
    --- End diff --
    
    The doc above this is wrong.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r160501723
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -126,15 +180,69 @@ abstract class MLWriter extends BaseReadWrite with Logging {
         this
       }
     
    +  // override for Java compatibility
    +  override def session(sparkSession: SparkSession): this.type = super.session(sparkSession)
    +
    +  // override for Java compatibility
    +  override def context(sqlContext: SQLContext): this.type = super.session(sqlContext.sparkSession)
    +}
    +
    +/**
    + * A ML Writer which delegates based on the requested format.
    + */
    +class GeneralMLWriter(stage: PipelineStage) extends MLWriter with Logging {
    +  private var source: String = "internal"
    +
       /**
    -   * Overwrites if the output path already exists.
    +   * Specifies the format of ML export (e.g. PMML, internal, or
    +   * the fully qualified class name for export).
        */
    -  @Since("1.6.0")
    -  def overwrite(): this.type = {
    -    shouldOverwrite = true
    +  @Since("2.3.0")
    +  def format(source: String): this.type = {
    +    this.source = source
         this
       }
     
    +  /**
    +   * Dispatches the save to the correct MLFormat.
    +   */
    +  @Since("2.3.0")
    +  @throws[IOException]("If the input path already exists but overwrite is not enabled.")
    +  @throws[SparkException]("If multiple sources for a given short name format are found.")
    +  override protected def saveImpl(path: String) = {
    +    val loader = Utils.getContextOrSparkClassLoader
    +    val serviceLoader = ServiceLoader.load(classOf[MLFormatRegister], loader)
    +    val stageName = stage.getClass.getName
    +    val targetName = s"${source}+${stageName}"
    +    val formats = serviceLoader.asScala.toList
    +    val shortNames = formats.map(_.shortName())
    +    val writerCls = formats.filter(_.shortName().equalsIgnoreCase(targetName)) match {
    +      // requested name did not match any given registered alias
    +      case Nil =>
    +        Try(loader.loadClass(source)) match {
    +          case Success(writer) =>
    +            // Found the ML writer using the fully qualified path
    +            writer
    +          case Failure(error) =>
    +            throw new SparkException(
    +              s"Could not load requested format $source for $stageName ($targetName) had $formats" +
    +              s"supporting $shortNames", error)
    +        }
    +      case head :: Nil =>
    +        head.getClass
    +      case _ =>
    +        // Multiple sources
    +        throw new SparkException(
    +          s"Multiple writers found for $source+$stageName, try using the class name of the writer")
    +    }
    +    if (classOf[MLWriterFormat].isAssignableFrom(writerCls)) {
    +      val writer = writerCls.newInstance().asInstanceOf[MLWriterFormat]
    --- End diff --
    
    This will fail, non-intuitively, if anyone ever extends `MLWriterFormat` with a constructor that has more than zero arguments. Meaning:
    
    ```scala
    class DummyLinearRegressionWriter(someParam: Int) extends MLWriterFormat
    ```
    
    will raise `java.lang.NoSuchMethodException: org.apache.spark.ml.regression.DummyLinearRegressionWriter.<init>()`
      


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r160483562
  
    --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala ---
    @@ -1044,6 +1056,50 @@ class LinearRegressionSuite extends MLTest with DefaultReadWriteTest {
           LinearRegressionSuite.allParamSettings, checkModelData)
       }
     
    +  test("pmml export") {
    +    val lr = new LinearRegression()
    +    val model = lr.fit(datasetWithWeight)
    +    def checkModel(pmml: PMML): Unit = {
    +      val dd = pmml.getDataDictionary
    +      assert(dd.getNumberOfFields === 3)
    +      val fields = dd.getDataFields.asScala
    +      assert(fields(0).getName().toString === "field_0")
    +      assert(fields(0).getOpType() == OpType.CONTINUOUS)
    +      val pmmlRegressionModel = pmml.getModels().get(0).asInstanceOf[PMMLRegressionModel]
    +      val pmmlPredictors = pmmlRegressionModel.getRegressionTables.get(0).getNumericPredictors
    +      val pmmlWeights = pmmlPredictors.asScala.map(_.getCoefficient()).toList
    +      assert(pmmlWeights(0) ~== model.coefficients(0) relTol 1E-3)
    +      assert(pmmlWeights(1) ~== model.coefficients(1) relTol 1E-3)
    +    }
    +    testPMMLWrite(sc, model, checkModel)
    +  }
    +
    +  test("unsupported export format") {
    +    val lr = new LinearRegression()
    +    val model = lr.fit(datasetWithWeight)
    +    intercept[SparkException] {
    +      model.write.format("boop").save("boop")
    +    }
    +    intercept[SparkException] {
    +      model.write.format("com.holdenkarau.boop").save("boop")
    +    }
    +    withClue("ML source org.apache.spark.SparkContext is not a valid MLWriterFormat") {
    +      intercept[SparkException] {
    +        model.write.format("org.apache.spark.SparkContext").save("boop2")
    +      }
    +    }
    +  }
    +
    +  test("dummy export format is called") {
    +    val lr = new LinearRegression()
    +    val model = lr.fit(datasetWithWeight)
    +    withClue("Dummy writer doesn't write") {
    +      intercept[Exception] {
    --- End diff --
    
    this just catches any exception. Can we do something like 
    
    ```scala
    val thrown = intercept[Exception] {
            model.write.format("org.apache.spark.ml.regression.DummyLinearRegressionWriter").save("")
          }
          assert(thrown.getMessage.contains("Dummy writer doesn't write."))
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86228/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r161944495
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -126,15 +180,69 @@ abstract class MLWriter extends BaseReadWrite with Logging {
         this
       }
     
    +  // override for Java compatibility
    +  override def session(sparkSession: SparkSession): this.type = super.session(sparkSession)
    +
    +  // override for Java compatibility
    +  override def context(sqlContext: SQLContext): this.type = super.session(sqlContext.sparkSession)
    +}
    +
    +/**
    + * A ML Writer which delegates based on the requested format.
    + */
    +class GeneralMLWriter(stage: PipelineStage) extends MLWriter with Logging {
    --- End diff --
    
    The follow up issue to track this is https://issues.apache.org/jira/browse/SPARK-11241


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1716/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r160496808
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -126,15 +180,69 @@ abstract class MLWriter extends BaseReadWrite with Logging {
         this
       }
     
    +  // override for Java compatibility
    +  override def session(sparkSession: SparkSession): this.type = super.session(sparkSession)
    +
    +  // override for Java compatibility
    +  override def context(sqlContext: SQLContext): this.type = super.session(sqlContext.sparkSession)
    +}
    +
    +/**
    + * A ML Writer which delegates based on the requested format.
    + */
    +class GeneralMLWriter(stage: PipelineStage) extends MLWriter with Logging {
    +  private var source: String = "internal"
    +
       /**
    -   * Overwrites if the output path already exists.
    +   * Specifies the format of ML export (e.g. PMML, internal, or
    +   * the fully qualified class name for export).
        */
    -  @Since("1.6.0")
    -  def overwrite(): this.type = {
    -    shouldOverwrite = true
    +  @Since("2.3.0")
    +  def format(source: String): this.type = {
    +    this.source = source
         this
       }
     
    +  /**
    +   * Dispatches the save to the correct MLFormat.
    +   */
    +  @Since("2.3.0")
    +  @throws[IOException]("If the input path already exists but overwrite is not enabled.")
    +  @throws[SparkException]("If multiple sources for a given short name format are found.")
    +  override protected def saveImpl(path: String) = {
    --- End diff --
    
    return type


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    re-ping folks ?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-23783][SPARK-11239] Add PMML export to...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/19876


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-23783][SPARK-11239] Add PMML export to...

Posted by goungoun <gi...@git.apache.org>.
Github user goungoun commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r178432400
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala ---
    @@ -710,15 +711,58 @@ class LinearRegressionModel private[ml] (
       }
     
       /**
    -   * Returns a [[org.apache.spark.ml.util.MLWriter]] instance for this ML instance.
    +   * Returns a [[org.apache.spark.ml.util.GeneralMLWriter]] instance for this ML instance.
        *
        * For [[LinearRegressionModel]], this does NOT currently save the training [[summary]].
        * An option to save [[summary]] may be added in the future.
        *
        * This also does not save the [[parent]] currently.
        */
       @Since("1.6.0")
    -  override def write: MLWriter = new LinearRegressionModel.LinearRegressionModelWriter(this)
    +  override def write: GeneralMLWriter = new GeneralMLWriter(this)
    +}
    +
    +/** A writer for LinearRegression that handles the "internal" (or default) format */
    +private class InternalLinearRegressionModelWriter
    +  extends MLWriterFormat with MLFormatRegister {
    +
    +  override def format(): String = "internal"
    +  override def stageName(): String = "org.apache.spark.ml.regression.LinearRegressionModel"
    +
    +  private case class Data(intercept: Double, coefficients: Vector, scale: Double)
    +
    +  override def write(path: String, sparkSession: SparkSession,
    +    optionMap: mutable.Map[String, String], stage: PipelineStage): Unit = {
    +    val instance = stage.asInstanceOf[LinearRegressionModel]
    +    val sc = sparkSession.sparkContext
    +    // Save metadata and Params
    +    DefaultParamsWriter.saveMetadata(instance, path, sc)
    +    // Save model data: intercept, coefficients, scale
    +    val data = Data(instance.intercept, instance.coefficients, instance.scale)
    +    val dataPath = new Path(path, "data").toString
    +    sparkSession.createDataFrame(Seq(data)).repartition(1).write.parquet(dataPath)
    +  }
    +}
    +
    +/** A writer for LinearRegression that handles the "pmml" format */
    +private class PMMLLinearRegressionModelWriter
    +    extends MLWriterFormat with MLFormatRegister {
    --- End diff --
    
    Thanks!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19876#discussion_r160503322
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala ---
    @@ -126,15 +180,69 @@ abstract class MLWriter extends BaseReadWrite with Logging {
         this
       }
     
    +  // override for Java compatibility
    +  override def session(sparkSession: SparkSession): this.type = super.session(sparkSession)
    +
    +  // override for Java compatibility
    +  override def context(sqlContext: SQLContext): this.type = super.session(sqlContext.sparkSession)
    +}
    +
    +/**
    + * A ML Writer which delegates based on the requested format.
    + */
    +class GeneralMLWriter(stage: PipelineStage) extends MLWriter with Logging {
    --- End diff --
    
    need `@Since("2.3.0")` here?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-23783][SPARK-11239] Add PMML export to Spark ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88546/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [WIP][ML][SPARK-11171][SPARK-11239] Add PMML export to S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #84473 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84473/testReport)** for PR 19876 at commit [`72b509f`](https://github.com/apache/spark/commit/72b509ff1919c7c82ab2909ffb5c9a7596e52e6c).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #86226 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86226/testReport)** for PR 19876 at commit [`6411054`](https://github.com/apache/spark/commit/6411054d12ea3d4b98bdc7ad409b392a8cc16a8f).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    **[Test build #88546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88546/testReport)** for PR 19876 at commit [`cb6fd70`](https://github.com/apache/spark/commit/cb6fd70d0c61b6477f7514431ee2e1c097ec0aff).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/19876
  
    So if no one has comments by March 15th I'm going update the tags & push forward with this API since we need something and this is the design most folks seems to be interested in from the last proposal.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org