You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Yuming Wang <q7...@gmail.com> on 2015/11/10 06:08:30 UTC
ml.feature.Word2Vec.transform() very slow issue
Hi
I found org.apache.spark.ml.feature.Word2Vec.transform() very slow.
I think we should not read broadcast every sentence, so I fixed on my forked.
https://github.com/979969786/spark/commit/a9f894df3671bb8df2f342de1820dab3185598f3
I have use 20000 number rows test it. Original version consume *5 minutes*,
and my version just consume *22 seconds* on same data.
If I'm right, I will pull request.
Thanks
Re: ml.feature.Word2Vec.transform() very slow issue
Posted by Nick Pentreath <ni...@gmail.com>.
Seems a straightforward change that purely enhances efficiency, so yes
please submit a JIRA and PR for this
On Tue, Nov 10, 2015 at 8:56 AM, Sean Owen <so...@cloudera.com> wrote:
> Since it's a fairly expensive operation to build the Map, I tend to agree
> it should not happen in the loop.
>
> On Tue, Nov 10, 2015 at 5:08 AM, Yuming Wang <q7...@gmail.com> wrote:
>
>> Hi
>>
>>
>>
>> I found org.apache.spark.ml.feature.Word2Vec.transform() very slow.
>>
>> I think we should not read broadcast every sentence, so I fixed on my forked.
>>
>>
>>
>> https://github.com/979969786/spark/commit/a9f894df3671bb8df2f342de1820dab3185598f3
>>
>>
>>
>> I have use 20000 number rows test it. Original version consume *5 minutes*,
>>
>>
>>
>>
>> and my version just consume *22 seconds* on same data.
>>
>>
>>
>>
>>
>>
>>
>> If I'm right, I will pull request.
>>
>>
>>
>> Thanks
>>
>>
>
Re: ml.feature.Word2Vec.transform() very slow issue
Posted by Sean Owen <so...@cloudera.com>.
Since it's a fairly expensive operation to build the Map, I tend to agree
it should not happen in the loop.
On Tue, Nov 10, 2015 at 5:08 AM, Yuming Wang <q7...@gmail.com> wrote:
> Hi
>
>
>
> I found org.apache.spark.ml.feature.Word2Vec.transform() very slow.
>
> I think we should not read broadcast every sentence, so I fixed on my forked.
>
>
>
> https://github.com/979969786/spark/commit/a9f894df3671bb8df2f342de1820dab3185598f3
>
>
>
> I have use 20000 number rows test it. Original version consume *5 minutes*,
>
>
>
>
> and my version just consume *22 seconds* on same data.
>
>
>
>
>
>
>
> If I'm right, I will pull request.
>
>
>
> Thanks
>
>