[GitHub] spark pull request #21525: [SPARK-24513][ML] Attribute support in UnaryTrans...

[viirya <gi...@git.apache.org>]

Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21525#discussion_r215461652
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/Transformer.scala ---
    @@ -116,10 +116,17 @@ abstract class UnaryTransformer[IN, OUT, T <: UnaryTransformer[IN, OUT, T]]
         StructType(outputFields)
       }
     
    +  /**
    +   * Returns [[Metadata]] to be attached to the output column.
    +   */
    +  protected def outputMetadata(outputSchema: StructType, dataset: Dataset[_]): Metadata =
    +    Metadata.empty
    +
       override def transform(dataset: Dataset[_]): DataFrame = {
    -    transformSchema(dataset.schema, logging = true)
    +    val outputSchema = transformSchema(dataset.schema, logging = true)
         val transformUDF = udf(this.createTransformFunc, outputDataType)
    -    dataset.withColumn($(outputCol), transformUDF(dataset($(inputCol))))
    +    val metadata = outputMetadata(outputSchema, dataset)
    --- End diff --
    
    `HashingTF` is an example that the metadata is created in `transformSchema` and attached to `outputSchema`. So my question is, do we need an extra API `outputMetadata` to do this?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org