You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by zml张明磊 <mi...@Ctrip.com> on 2015/12/17 09:00:57 UTC

Are there some solution to complete the transform category variables into dummy variable in scala or spark ?

Hi ,

         I am a new to scala and spark. Recently, I need to write a tool that transform category variables to dummy/indicator variables. I want to know are there some tools in scala and spark which support this transformation which like pandas.get_dummies in python ? Any example or study learning materials for me ?

Thanks,
Minglei.

Re: Are there some solution to complete the transform category variables into dummy variable in scala or spark ?

Posted by Yanbo Liang <yb...@gmail.com>.
Hi Minglei,

Spark ML provide a transformer named "OneHotEncoder" to map a column of
category indices to a column of binary vectors. It's similar with
pandas.get_dummies and OneHotEncoder of sklearn, but the output will be a
column of vector type rather than multiple columns.
You can refer the officially example
<https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/ml/OneHotEncoderExample.scala>
.

Yanbo

2015-12-17 16:00 GMT+08:00 zml张明磊 <mi...@ctrip.com>:

> Hi ,
>
>
>
>          I am a new to scala and spark. Recently, I need to write a tool
> that transform category variables to dummy/indicator variables. I want to
> know are there some tools in scala and spark which support this
> transformation which like *pandas.get_dummies* in python ? Any example or
> study learning materials for me ?
>
>
>
> Thanks,
>
> Minglei.
>