[jira] [Created] (SPARK-6915) VectorIndexer improvements

["Joseph K. Bradley (JIRA)" <ji...@apache.org>]

Joseph K. Bradley created SPARK-6915:
----------------------------------------

             Summary: VectorIndexer improvements
                 Key: SPARK-6915
                 URL: https://issues.apache.org/jira/browse/SPARK-6915
             Project: Spark
          Issue Type: Improvement
          Components: ML
    Affects Versions: 1.4.0
            Reporter: Joseph K. Bradley
            Priority: Minor


This covers several improvements to VectorIndexer.  They could be handled separately or in 1 PR.

*Preserve metadata*

Currently, it does not preserve ML-specific input metadata in the output column.  If a feature is already marked as categorical or continuous, we should preserve that metadata (rather than recomputing it).  We should also check that the input data is valid for that metadata.

*Allow unknown categories*

Add option for allowing unknown categories, probably via a parameter like "allowUnknownCategories."
If true, then handle unknown categories during transform by assigning them to an extra category index.

*Index particular features*

Add option for limiting indexing to particular features.
This could be specified by an option, or we could handle it via the "Preserve metadata" task above, where users would denote features as continuous in order to have VectorIndexer ignore them.

*Performance optimizations*

See the TODO items within VectorIndexer.scala




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org