You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yanbo Liang (JIRA)" <ji...@apache.org> on 2015/10/16 03:05:05 UTC
[jira] [Commented] (SPARK-8418) Add single- and multi-value support
to ML Transformers
[ https://issues.apache.org/jira/browse/SPARK-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959978#comment-14959978 ]
Yanbo Liang commented on SPARK-8418:
------------------------------------
[~josephkb] I don't think RFormula is the best way to resolve this issue because it still use the pipeline chained transformers one by one to encode multiple columns which is low performance.
I vote for strategy 2 of [~nburoojy] proposed. But I think we don't need to reimplement all transformers to support a multi-value implementation because of some feature transformers not needed.
I will firstly try to start with OneHotEncoder which is mostly common used.
> Add single- and multi-value support to ML Transformers
> ------------------------------------------------------
>
> Key: SPARK-8418
> URL: https://issues.apache.org/jira/browse/SPARK-8418
> Project: Spark
> Issue Type: Sub-task
> Components: ML
> Reporter: Joseph K. Bradley
>
> It would be convenient if all feature transformers supported transforming columns of single values and multiple values, specifically:
> * one column with one value (e.g., type {{Double}})
> * one column with multiple values (e.g., {{Array[Double]}} or {{Vector}})
> We could go as far as supporting multiple columns, but that may not be necessary since VectorAssembler could be used to handle that.
> Estimators under {{ml.feature}} should also support this.
> This will likely require a short design doc to describe:
> * how input and output columns will be specified
> * schema validation
> * code sharing to reduce duplication
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org