You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Sasaki Kai <le...@me.com> on 2015/11/28 14:20:30 UTC
Export BLAS module on Spark MLlib
Hello
I'm developing a Spark package that manipulates Vector and Matrix for machine learning.
This package uses mllib.linalg.Vector and mllib.linalg.Matrix in order to achieve compatible interface to mllib itself. But mllib.linalg.BLAS module is private inside spark package. We cannot use BLAS from spark package.
Due to this, there is no way to manipulate mllib.linalg.{Vector, Matrix} from spark package side.
Is there any reason why BLAS module is not set public?
If we cannot use BLAS, what is the reasonable option to manipulate Vector and Matrix from spark package?
Regards
Kai Sasaki(@Lewuathe)
Re: Export BLAS module on Spark MLlib
Posted by DB Tsai <db...@dbtsai.com>.
I used reflection initially, but I found it's very slow especially in
a tight loop. Maybe caching the reflection can help which I never try.
Sincerely,
DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 0xAF08DF8D
On Mon, Nov 30, 2015 at 2:15 PM, Burak Yavuz <br...@gmail.com> wrote:
> Or you could also use reflection like in this Spark Package:
> https://github.com/brkyvz/lazy-linalg/blob/master/src/main/scala/com/brkyvz/spark/linalg/BLASUtils.scala
>
> Best,
> Burak
>
> On Mon, Nov 30, 2015 at 12:48 PM, DB Tsai <db...@dbtsai.com> wrote:
>>
>> The workaround is have your code in the same package, or write some
>> utility wrapper in the same package so you can use them in your code.
>> Mostly we implement those BLAS for our own need, and we don't have
>> general use-case in mind. As a result, if we open them up prematurely,
>> it will add our api maintenance cost. Once it's getting mature, and
>> people are asking for them, we will gradually make them public.
>>
>> Thanks.
>>
>> Sincerely,
>>
>> DB Tsai
>> ----------------------------------------------------------
>> Web: https://www.dbtsai.com
>> PGP Key ID: 0xAF08DF8D
>>
>>
>> On Sat, Nov 28, 2015 at 5:20 AM, Sasaki Kai <le...@me.com> wrote:
>> > Hello
>> >
>> > I'm developing a Spark package that manipulates Vector and Matrix for
>> > machine learning.
>> > This package uses mllib.linalg.Vector and mllib.linalg.Matrix in order
>> > to
>> > achieve compatible interface to mllib itself. But mllib.linalg.BLAS
>> > module
>> > is private inside spark package. We cannot use BLAS from spark package.
>> > Due to this, there is no way to manipulate mllib.linalg.{Vector, Matrix}
>> > from spark package side.
>> >
>> > Is there any reason why BLAS module is not set public?
>> > If we cannot use BLAS, what is the reasonable option to manipulate
>> > Vector
>> > and Matrix from spark package?
>> >
>> > Regards
>> > Kai Sasaki(@Lewuathe)
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Export BLAS module on Spark MLlib
Posted by Burak Yavuz <br...@gmail.com>.
Or you could also use reflection like in this Spark Package:
https://github.com/brkyvz/lazy-linalg/blob/master/src/main/scala/com/brkyvz/spark/linalg/BLASUtils.scala
Best,
Burak
On Mon, Nov 30, 2015 at 12:48 PM, DB Tsai <db...@dbtsai.com> wrote:
> The workaround is have your code in the same package, or write some
> utility wrapper in the same package so you can use them in your code.
> Mostly we implement those BLAS for our own need, and we don't have
> general use-case in mind. As a result, if we open them up prematurely,
> it will add our api maintenance cost. Once it's getting mature, and
> people are asking for them, we will gradually make them public.
>
> Thanks.
>
> Sincerely,
>
> DB Tsai
> ----------------------------------------------------------
> Web: https://www.dbtsai.com
> PGP Key ID: 0xAF08DF8D
>
>
> On Sat, Nov 28, 2015 at 5:20 AM, Sasaki Kai <le...@me.com> wrote:
> > Hello
> >
> > I'm developing a Spark package that manipulates Vector and Matrix for
> > machine learning.
> > This package uses mllib.linalg.Vector and mllib.linalg.Matrix in order to
> > achieve compatible interface to mllib itself. But mllib.linalg.BLAS
> module
> > is private inside spark package. We cannot use BLAS from spark package.
> > Due to this, there is no way to manipulate mllib.linalg.{Vector, Matrix}
> > from spark package side.
> >
> > Is there any reason why BLAS module is not set public?
> > If we cannot use BLAS, what is the reasonable option to manipulate Vector
> > and Matrix from spark package?
> >
> > Regards
> > Kai Sasaki(@Lewuathe)
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: Export BLAS module on Spark MLlib
Posted by DB Tsai <db...@dbtsai.com>.
The workaround is have your code in the same package, or write some
utility wrapper in the same package so you can use them in your code.
Mostly we implement those BLAS for our own need, and we don't have
general use-case in mind. As a result, if we open them up prematurely,
it will add our api maintenance cost. Once it's getting mature, and
people are asking for them, we will gradually make them public.
Thanks.
Sincerely,
DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 0xAF08DF8D
On Sat, Nov 28, 2015 at 5:20 AM, Sasaki Kai <le...@me.com> wrote:
> Hello
>
> I'm developing a Spark package that manipulates Vector and Matrix for
> machine learning.
> This package uses mllib.linalg.Vector and mllib.linalg.Matrix in order to
> achieve compatible interface to mllib itself. But mllib.linalg.BLAS module
> is private inside spark package. We cannot use BLAS from spark package.
> Due to this, there is no way to manipulate mllib.linalg.{Vector, Matrix}
> from spark package side.
>
> Is there any reason why BLAS module is not set public?
> If we cannot use BLAS, what is the reasonable option to manipulate Vector
> and Matrix from spark package?
>
> Regards
> Kai Sasaki(@Lewuathe)
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org