You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Soumya Simanta <so...@gmail.com> on 2014/10/01 23:18:32 UTC
Creating a feature vector from text before using with MLLib
I'm trying to understand the intuition behind the features method that
Aaron used in one of his demos. I believe this feature will just work for
detecting the character set (i.e., language used).
Can someone help ?
def featurize(s: String): Vector = {
val n = 1000
val result = new Array[Double](n)
val bigrams = s.sliding(2).toArray
for (h <- bigrams.map(_.hashCode % n)) {
result(h) += 1.0 / bigrams.length
}
Vectors.sparse(n, result.zipWithIndex.filter(_._1 != 0).map(_.swap))
}
Re: Creating a feature vector from text before using with MLLib
Posted by Xiangrui Meng <me...@gmail.com>.
Yes, the "bigram" in that demo only has two characters, which could
separate different character sets. -Xiangrui
On Wed, Oct 1, 2014 at 2:54 PM, Liquan Pei <li...@gmail.com> wrote:
> The program computes hashing bi-gram frequency normalized by total number of
> bigrams then filter out zero values. hashing is a effective trick of
> vectorizing features. Take a look at
> http://en.wikipedia.org/wiki/Feature_hashing
>
> Liquan
>
> On Wed, Oct 1, 2014 at 2:18 PM, Soumya Simanta <so...@gmail.com>
> wrote:
>>
>> I'm trying to understand the intuition behind the features method that
>> Aaron used in one of his demos. I believe this feature will just work for
>> detecting the character set (i.e., language used).
>>
>> Can someone help ?
>>
>>
>> def featurize(s: String): Vector = {
>> val n = 1000
>> val result = new Array[Double](n)
>> val bigrams = s.sliding(2).toArray
>>
>> for (h <- bigrams.map(_.hashCode % n)) {
>> result(h) += 1.0 / bigrams.length
>> }
>>
>> Vectors.sparse(n, result.zipWithIndex.filter(_._1 != 0).map(_.swap))
>> }
>>
>>
>>
>
>
>
> --
> Liquan Pei
> Department of Physics
> University of Massachusetts Amherst
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Creating a feature vector from text before using with MLLib
Posted by Liquan Pei <li...@gmail.com>.
The program computes hashing bi-gram frequency normalized by total number
of bigrams then filter out zero values. hashing is a effective trick of
vectorizing features. Take a look at
http://en.wikipedia.org/wiki/Feature_hashing
Liquan
On Wed, Oct 1, 2014 at 2:18 PM, Soumya Simanta <so...@gmail.com>
wrote:
> I'm trying to understand the intuition behind the features method that
> Aaron used in one of his demos. I believe this feature will just work for
> detecting the character set (i.e., language used).
>
> Can someone help ?
>
>
> def featurize(s: String): Vector = {
> val n = 1000
> val result = new Array[Double](n)
> val bigrams = s.sliding(2).toArray
>
> for (h <- bigrams.map(_.hashCode % n)) {
> result(h) += 1.0 / bigrams.length
> }
>
> Vectors.sparse(n, result.zipWithIndex.filter(_._1 != 0).map(_.swap))
> }
>
>
>
>
--
Liquan Pei
Department of Physics
University of Massachusetts Amherst