[jira] [Comment Edited] (SPARK-19962) add DictVectorizor for DataFrame

["yu peng (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-19962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15926723#comment-15926723 ] 

yu peng edited comment on SPARK-19962 at 3/15/17 6:42 PM:
----------------------------------------------------------

yeah, with one indexer that performing like mutually exclusive  n "indexers". 
i can have a sparse version of the output  features fairly easy.

let's say if i want to train a classifer for gender based on age, country, hobbies. 
with DictVectorizor, it's going to be 

vec = DictVectorizor(sparse=True, inputColumns=['age', 'country', 'hobbies'], outputColumn='feature')

df= vec.fit_transform(df)
df= Binarizer(threshold=1.0, inputCol="_", outputCol="label").fit_transform(StringIndexer(inputColumn='gender', outputCol='_').fit_transform(df['gender'])
rf = RandomForestClassifier(labelCol="label", featuresCol="fetures", numTrees=10)
rf.fit(df)




was (Author: yupbank):
yeah, with one indexer that performing like mutually exclusive  n "indexers". 
i can have a sparse version of the output  features fairly easy.

let's say if i want to train a classifer for gender based on age, country, hobbies. 
with DictVectorizor, it's going to be 

vec = DictVectorizor(sparse=True)

df['fetures'] = vec.fit_transform(df[['age', 'country', 'hobbies']])
df['label'] = OnehotEncoder().fit_transform(StringIndexer().fit_transform(df['gender'])
rf = RandomForestClassifier(labelCol="label", featuresCol="fetures", numTrees=10)
rf.fit(df)



> add DictVectorizor for DataFrame
> --------------------------------
>
>                 Key: SPARK-19962
>                 URL: https://issues.apache.org/jira/browse/SPARK-19962
>             Project: Spark
>          Issue Type: Wish
>          Components: ML
>    Affects Versions: 2.1.0
>            Reporter: yu peng
>              Labels: features
>
> it's really useful to have something like sklearn.feature_extraction.DictVectorizor
> Since out features lives in json/data frame like format and classifier/regressors only take vector input. so there is a gap between them.
> something like 
> ```
> df = sqlCtx.createDataFrame([Row(age=1, gender='male', country='cn', hobbies=['sing', 'dance']),Row(age=3, gender='female', country='us',  hobbies=['sing']), ])
> df.show()
> |age|gender|country|hobbies|
> |1|male|cn|[sing, dance]|
> |3|female|us|[sing]|
> import DictVectorizor
> vec = DictVectorizor()
> matrix = vec.fit_transform(df)
> matrix.show()
> |features|
> |[1, 0, 1, 0, 1, 1, 1]|
> |[3, 1, 0, 1, 0, 1, 1]|
> vec.show()
> |feature_name| feature_dimension|
> |age|0|
> |gender=female|1|
> |gender=male|2|
> |country=us|3|
> |country=cn|4|
> |hobbies=sing|5|
> |hobbies=dance|6|
> ```



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org