You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Eli Super <el...@gmail.com> on 2016/01/25 11:34:18 UTC

How to discretize Continuous Variable with Spark DataFrames

Hi

What is a best way to discretize Continuous Variable within  Spark
DataFrames ?

I want to discretize some variable 1) by equal frequency 2) by k-means

I usually use R  for this porpoises

_http://www.inside-r.org/packages/cran/arules/docs/discretize

R code for example :

### equal frequency
table(discretize(data$some_column, "frequency", categories=10))


#k-means
table(discretize(data$some_column, "cluster", categories=10))

Thanks a lot !

Re: How to discretize Continuous Variable with Spark DataFrames

Posted by Jeff Zhang <zj...@gmail.com>.
It's very straightforward, please refer the document here
http://spark.apache.org/docs/latest/ml-features.html#bucketizer


On Mon, Jan 25, 2016 at 10:09 PM, Eli Super <el...@gmail.com> wrote:

> Thanks Joshua ,
>
> I can't understand what algorithm behind Bucketizer  , how discretization
> done ?
>
> Best Regards
>
>
> On Mon, Jan 25, 2016 at 3:40 PM, Joshua TAYLOR <jo...@gmail.com>
> wrote:
>
>> It sounds like you may want the Bucketizer in SparkML.  The overview docs
>> [1] include, "Bucketizer transforms a column of continuous features to a
>> column of feature buckets, where the buckets are specified by users."
>>
>> [1]: http://spark.apache.org/docs/latest/ml-features.html#bucketizer
>>
>> On Mon, Jan 25, 2016 at 5:34 AM, Eli Super <el...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> What is a best way to discretize Continuous Variable within  Spark
>>> DataFrames ?
>>>
>>> I want to discretize some variable 1) by equal frequency 2) by k-means
>>>
>>> I usually use R  for this porpoises
>>>
>>> _http://www.inside-r.org/packages/cran/arules/docs/discretize
>>>
>>> R code for example :
>>>
>>> ### equal frequency
>>> table(discretize(data$some_column, "frequency", categories=10))
>>>
>>>
>>> #k-means
>>> table(discretize(data$some_column, "cluster", categories=10))
>>>
>>> Thanks a lot !
>>>
>>
>>
>>
>> --
>> Joshua Taylor, http://www.cs.rpi.edu/~tayloj/
>>
>
>


-- 
Best Regards

Jeff Zhang

Re: How to discretize Continuous Variable with Spark DataFrames

Posted by Eli Super <el...@gmail.com>.
Thanks Joshua ,

I can't understand what algorithm behind Bucketizer  , how discretization
done ?

Best Regards


On Mon, Jan 25, 2016 at 3:40 PM, Joshua TAYLOR <jo...@gmail.com>
wrote:

> It sounds like you may want the Bucketizer in SparkML.  The overview docs
> [1] include, "Bucketizer transforms a column of continuous features to a
> column of feature buckets, where the buckets are specified by users."
>
> [1]: http://spark.apache.org/docs/latest/ml-features.html#bucketizer
>
> On Mon, Jan 25, 2016 at 5:34 AM, Eli Super <el...@gmail.com> wrote:
>
>> Hi
>>
>> What is a best way to discretize Continuous Variable within  Spark
>> DataFrames ?
>>
>> I want to discretize some variable 1) by equal frequency 2) by k-means
>>
>> I usually use R  for this porpoises
>>
>> _http://www.inside-r.org/packages/cran/arules/docs/discretize
>>
>> R code for example :
>>
>> ### equal frequency
>> table(discretize(data$some_column, "frequency", categories=10))
>>
>>
>> #k-means
>> table(discretize(data$some_column, "cluster", categories=10))
>>
>> Thanks a lot !
>>
>
>
>
> --
> Joshua Taylor, http://www.cs.rpi.edu/~tayloj/
>

Re: How to discretize Continuous Variable with Spark DataFrames

Posted by Joshua TAYLOR <jo...@gmail.com>.
It sounds like you may want the Bucketizer in SparkML.  The overview docs
[1] include, "Bucketizer transforms a column of continuous features to a
column of feature buckets, where the buckets are specified by users."

[1]: http://spark.apache.org/docs/latest/ml-features.html#bucketizer

On Mon, Jan 25, 2016 at 5:34 AM, Eli Super <el...@gmail.com> wrote:

> Hi
>
> What is a best way to discretize Continuous Variable within  Spark
> DataFrames ?
>
> I want to discretize some variable 1) by equal frequency 2) by k-means
>
> I usually use R  for this porpoises
>
> _http://www.inside-r.org/packages/cran/arules/docs/discretize
>
> R code for example :
>
> ### equal frequency
> table(discretize(data$some_column, "frequency", categories=10))
>
>
> #k-means
> table(discretize(data$some_column, "cluster", categories=10))
>
> Thanks a lot !
>



-- 
Joshua Taylor, http://www.cs.rpi.edu/~tayloj/