[jira] [Comment Edited] (SPARK-8418) Add single- and multi-value support to ML Transformers

["Yanbo Liang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959978#comment-14959978 ] 

Yanbo Liang edited comment on SPARK-8418 at 10/16/15 1:23 AM:
--------------------------------------------------------------

[~josephkb] I don't think RFormula is the best way to resolve this issue because it still use the pipeline chained transformers one by one to encode multiple columns which is low performance.
I vote for strategy 2 of [~nburoojy] proposed. But I think we don't need to reimplement all transformers to support a multi-value implementation because of some feature transformers not needed.
Brief design doc:
* How input and output columns will be specified
/** @group setParam */
  def setInputCols(value: Array[String]): this.type = set(inputCols, value)
/** @group setParam */
  def setOutputCols(value: Array[String]): this.type = set(outputCols, value)
* Schema validation
Make transformSchema adaptive to multiple input and output columns.
* Code sharing to reduce duplication
For backwards compatibility, we must not modify current Params, we add a new one for multiple inputs (and check for conflicting settings when running). Reimplement transformers to support multi-value implementation and make the single-value interface a trivial invocation of the multi-value code. I think we should maximum reuse the transform function of a single-value to implement the multi-value one. 
I will firstly try to start sub-tasks with StringIndexer and OneHotEncoder which is mostly common used.
 


was (Author: yanboliang):
[~josephkb] I don't think RFormula is the best way to resolve this issue because it still use the pipeline chained transformers one by one to encode multiple columns which is low performance.
I vote for strategy 2 of [~nburoojy] proposed. But I think we don't need to reimplement all transformers to support a multi-value implementation because of some feature transformers not needed.
I will firstly try to start with OneHotEncoder which is mostly common used.
 

> Add single- and multi-value support to ML Transformers
> ------------------------------------------------------
>
>                 Key: SPARK-8418
>                 URL: https://issues.apache.org/jira/browse/SPARK-8418
>             Project: Spark
>          Issue Type: Sub-task
>          Components: ML
>            Reporter: Joseph K. Bradley
>
> It would be convenient if all feature transformers supported transforming columns of single values and multiple values, specifically:
> * one column with one value (e.g., type {{Double}})
> * one column with multiple values (e.g., {{Array[Double]}} or {{Vector}})
> We could go as far as supporting multiple columns, but that may not be necessary since VectorAssembler could be used to handle that.
> Estimators under {{ml.feature}} should also support this.
> This will likely require a short design doc to describe:
> * how input and output columns will be specified
> * schema validation
> * code sharing to reduce duplication



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org