You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Pulasthi Supun Wickramasinghe <pu...@gmail.com> on 2015/09/17 21:36:44 UTC
Creating BlockMatrix with java API
Hi All,
I am new to Spark and i am trying to do some BlockMatrix operations with
the Mllib API's. But i can't seem to create a BlockMatrix with the java
API. I tried the following
Matrix matrixa = Matrices.rand(4, 4, new Random(1000));
List<Tuple2<Tuple2<Integer,Integer>,Matrix>> list = new
ArrayList<Tuple2<Tuple2<Integer, Integer>, Matrix>>();
Tuple2<Integer,Integer> intTuple = new Tuple2<Integer, Integer>(0,0);
Tuple2<Tuple2<Integer,Integer>,Matrix> tuple2MatrixTuple2 = new
Tuple2<Tuple2<Integer, Integer>, Matrix>(intTuple,matrixa );
list.add(tuple2MatrixTuple2);
JavaRDD<Tuple2<Tuple2<Integer, Integer>, Matrix>> rdd = sc.parallelize(list);
BlockMatrix blockMatrix = new BlockMatrix(rdd,2,2);
but since BlockMatrix only
takes "RDD<scala.Tuple2<scala.Tuple2<java.lang.Object,java.lang.Object>,Matrix>>"
this code does not work. sc.parallelize() returns a JavaRDD so the two are
not compatible. I also couldn't find any code samples for this. Any help on
this would be highly appreciated.
Best Regards,
Pulasthi
--
Pulasthi S. Wickramasinghe
Graduate Student | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
cell: 224-386-9035
Re: Creating BlockMatrix with java API
Posted by Sabarish Sasidharan <sa...@manthan.com>.
What I meant is that something like this would work. Yes, it's less than
elegant but it works.
List<Tuple2<Tuple2<Object, Object>, Matrix>> blocks = new
ArrayList<Tuple2<Tuple2<Object,Object>,Matrix>>();
blocks.add(
new Tuple2<Tuple2<Object, Object>, Matrix>(
new Tuple2<Object, Object>(0, 0), Matrices.dense(2, 2, new double[] {0.0D,
1.1D, 2.0D, 3.1D})));
blocks.add(
new Tuple2<Tuple2<Object, Object>, Matrix>(
new Tuple2<Object, Object>(0, 1), Matrices.dense(2, 2, new double[] {0.0D,
1.1D, 2.0D, 3.1D})));
blocks.add(
new Tuple2<Tuple2<Object, Object>, Matrix>(
new Tuple2<Object, Object>(1, 0), Matrices.dense(2, 2, new double[] {0.0D,
1.1D, 2.0D, 3.1D})));
blocks.add(
new Tuple2<Tuple2<Object, Object>, Matrix>(
new Tuple2<Object, Object>(1, 1), Matrices.dense(2, 2, new double[] {0.0D,
1.1D, 2.0D, 3.1D})));
BlockMatrix bm = new
BlockMatrix(CurrentContext.getCurrentContext().parallelize(blocks).rdd(),
2, 2);
Regards
Sab
On Thu, Sep 24, 2015 at 2:35 AM, Pulasthi Supun Wickramasinghe <
pulasthi911@gmail.com> wrote:
> Hi YiZhi,
>
> Actually i was not able to try it out to see if it was working. I sent the
> previous reply assuming that Sabarish's solution would work :). Sorry if
> there was any confusion.
>
> Best Regards,
> Pulasthi
>
> On Wed, Sep 23, 2015 at 6:47 AM, YiZhi Liu <li...@mvad.com> wrote:
>
>> Hi Pulasthi,
>>
>> Are you sure this worked? When I applied rdd.rdd() to the constructor
>> of BlockMatrix, the complier complained
>>
>> [error]
>> spark/examples/src/main/java/org/apache/spark/examples/mllib/JavaSVD.java:38:
>> error: incompatible types: RDD<Tuple2<Tuple2<Integer,Integer>,Matrix>>
>> cannot be converted to RDD<Tuple2<Tuple2<Object,Object>,Matrix>>
>> [error] BlockMatrix blockMatrix = new BlockMatrix(rdd.rdd(), 2, 2);
>>
>> It must caused by the type elimination from scala to java. To make it
>> work, we have to define 'rdd' as JavaRDD<Tuple2<Tuple2<Object,
>> Object>, Matrix>>
>>
>> As Yanbo has mentioned, I think a Java friendly constructor is still in
>> demand.
>>
>> 2015-09-23 13:14 GMT+08:00 Pulasthi Supun Wickramasinghe
>> <pu...@gmail.com>:
>> > Hi Sabarish
>> >
>> > Thanks, that would indeed solve my problem
>> >
>> > Best Regards,
>> > Pulasthi
>> >
>> > On Wed, Sep 23, 2015 at 12:55 AM, Sabarish Sasidharan
>> > <sa...@manthan.com> wrote:
>> >>
>> >> Hi Pulasthi
>> >>
>> >> You can always use JavaRDD.rdd() to get the scala rdd. So in your case,
>> >>
>> >> new BlockMatrix(rdd.rdd(), 2, 2)
>> >>
>> >> should work.
>> >>
>> >> Regards
>> >> Sab
>> >>
>> >> On Tue, Sep 22, 2015 at 10:50 PM, Pulasthi Supun Wickramasinghe
>> >> <pu...@gmail.com> wrote:
>> >>>
>> >>> Hi Yanbo,
>> >>>
>> >>> Thanks for the reply. I thought i might be missing something. Anyway i
>> >>> moved to using scala since it is the complete API.
>> >>>
>> >>> Best Regards,
>> >>> Pulasthi
>> >>>
>> >>> On Tue, Sep 22, 2015 at 7:03 AM, Yanbo Liang <yb...@gmail.com>
>> wrote:
>> >>>>
>> >>>> This is due to the distributed matrices like
>> >>>> BlockMatrix/RowMatrix/IndexedRowMatrix/CoordinateMatrix do not
>> provide Java
>> >>>> friendly constructors. I have file a SPARK-10757 to track this issue.
>> >>>>
>> >>>> 2015-09-18 3:36 GMT+08:00 Pulasthi Supun Wickramasinghe
>> >>>> <pu...@gmail.com>:
>> >>>>>
>> >>>>> Hi All,
>> >>>>>
>> >>>>> I am new to Spark and i am trying to do some BlockMatrix operations
>> >>>>> with the Mllib API's. But i can't seem to create a BlockMatrix with
>> the java
>> >>>>> API. I tried the following
>> >>>>>
>> >>>>> Matrix matrixa = Matrices.rand(4, 4, new Random(1000));
>> >>>>> List<Tuple2<Tuple2<Integer,Integer>,Matrix>> list = new
>> >>>>> ArrayList<Tuple2<Tuple2<Integer, Integer>, Matrix>>();
>> >>>>> Tuple2<Integer,Integer> intTuple = new Tuple2<Integer,
>> Integer>(0,0);
>> >>>>> Tuple2<Tuple2<Integer,Integer>,Matrix> tuple2MatrixTuple2 = new
>> >>>>> Tuple2<Tuple2<Integer, Integer>, Matrix>(intTuple,matrixa );
>> >>>>> list.add(tuple2MatrixTuple2);
>> >>>>> JavaRDD<Tuple2<Tuple2<Integer, Integer>, Matrix>> rdd =
>> >>>>> sc.parallelize(list);
>> >>>>>
>> >>>>> BlockMatrix blockMatrix = new BlockMatrix(rdd,2,2);
>> >>>>>
>> >>>>>
>> >>>>> but since BlockMatrix only takes
>> >>>>>
>> "RDD<scala.Tuple2<scala.Tuple2<java.lang.Object,java.lang.Object>,Matrix>>"
>> >>>>> this code does not work. sc.parallelize() returns a JavaRDD so the
>> two are
>> >>>>> not compatible. I also couldn't find any code samples for this. Any
>> help on
>> >>>>> this would be highly appreciated.
>> >>>>>
>> >>>>> Best Regards,
>> >>>>> Pulasthi
>> >>>>> --
>> >>>>> Pulasthi S. Wickramasinghe
>> >>>>> Graduate Student | Research Assistant
>> >>>>> School of Informatics and Computing | Digital Science Center
>> >>>>> Indiana University, Bloomington
>> >>>>> cell: 224-386-9035
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Pulasthi S. Wickramasinghe
>> >>> Graduate Student | Research Assistant
>> >>> School of Informatics and Computing | Digital Science Center
>> >>> Indiana University, Bloomington
>> >>> cell: 224-386-9035
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> Architect - Big Data
>> >> Ph: +91 99805 99458
>> >>
>> >> Manthan Systems | Company of the year - Analytics (2014 Frost and
>> Sullivan
>> >> India ICT)
>> >> +++
>> >
>> >
>> >
>> >
>> > --
>> > Pulasthi S. Wickramasinghe
>> > Graduate Student | Research Assistant
>> > School of Informatics and Computing | Digital Science Center
>> > Indiana University, Bloomington
>> > cell: 224-386-9035
>>
>>
>>
>> --
>> 刘忆智
>> 广告质量部
>>
>> MV AD 聚效广告 上海 · 北京 · 广州 · 杭州
>> ______________________________________________________________
>> 上海市闸北区天目中路585号新梅大厦4楼 200070
>> MOB:15021072706
>> TEL:021-52559088
>> FAX:021-52559089
>> EMAIL:liuyz@mvad.com
>> HTTP:www.mvad.com
>> -------------------------------CONFIDENTIAL ------------------------------
>> 本邮件载有秘密信息,请您恪守保密义务,勿向第三人透露。谢谢合作。
>> This email communication is confidential. Recipient(s) named above
>> is(are) obligated to maintain secrecy and is(are) not permitted to
>> disclose the contents of this communication to others. Thank you.
>>
>>
>
>
> --
> Pulasthi S. Wickramasinghe
> Graduate Student | Research Assistant
> School of Informatics and Computing | Digital Science Center
> Indiana University, Bloomington
> cell: 224-386-9035
>
--
Architect - Big Data
Ph: +91 99805 99458
Manthan Systems | *Company of the year - Analytics (2014 Frost and Sullivan
India ICT)*
+++
Re: Creating BlockMatrix with java API
Posted by Pulasthi Supun Wickramasinghe <pu...@gmail.com>.
Hi YiZhi,
Actually i was not able to try it out to see if it was working. I sent the
previous reply assuming that Sabarish's solution would work :). Sorry if
there was any confusion.
Best Regards,
Pulasthi
On Wed, Sep 23, 2015 at 6:47 AM, YiZhi Liu <li...@mvad.com> wrote:
> Hi Pulasthi,
>
> Are you sure this worked? When I applied rdd.rdd() to the constructor
> of BlockMatrix, the complier complained
>
> [error]
> spark/examples/src/main/java/org/apache/spark/examples/mllib/JavaSVD.java:38:
> error: incompatible types: RDD<Tuple2<Tuple2<Integer,Integer>,Matrix>>
> cannot be converted to RDD<Tuple2<Tuple2<Object,Object>,Matrix>>
> [error] BlockMatrix blockMatrix = new BlockMatrix(rdd.rdd(), 2, 2);
>
> It must caused by the type elimination from scala to java. To make it
> work, we have to define 'rdd' as JavaRDD<Tuple2<Tuple2<Object,
> Object>, Matrix>>
>
> As Yanbo has mentioned, I think a Java friendly constructor is still in
> demand.
>
> 2015-09-23 13:14 GMT+08:00 Pulasthi Supun Wickramasinghe
> <pu...@gmail.com>:
> > Hi Sabarish
> >
> > Thanks, that would indeed solve my problem
> >
> > Best Regards,
> > Pulasthi
> >
> > On Wed, Sep 23, 2015 at 12:55 AM, Sabarish Sasidharan
> > <sa...@manthan.com> wrote:
> >>
> >> Hi Pulasthi
> >>
> >> You can always use JavaRDD.rdd() to get the scala rdd. So in your case,
> >>
> >> new BlockMatrix(rdd.rdd(), 2, 2)
> >>
> >> should work.
> >>
> >> Regards
> >> Sab
> >>
> >> On Tue, Sep 22, 2015 at 10:50 PM, Pulasthi Supun Wickramasinghe
> >> <pu...@gmail.com> wrote:
> >>>
> >>> Hi Yanbo,
> >>>
> >>> Thanks for the reply. I thought i might be missing something. Anyway i
> >>> moved to using scala since it is the complete API.
> >>>
> >>> Best Regards,
> >>> Pulasthi
> >>>
> >>> On Tue, Sep 22, 2015 at 7:03 AM, Yanbo Liang <yb...@gmail.com>
> wrote:
> >>>>
> >>>> This is due to the distributed matrices like
> >>>> BlockMatrix/RowMatrix/IndexedRowMatrix/CoordinateMatrix do not
> provide Java
> >>>> friendly constructors. I have file a SPARK-10757 to track this issue.
> >>>>
> >>>> 2015-09-18 3:36 GMT+08:00 Pulasthi Supun Wickramasinghe
> >>>> <pu...@gmail.com>:
> >>>>>
> >>>>> Hi All,
> >>>>>
> >>>>> I am new to Spark and i am trying to do some BlockMatrix operations
> >>>>> with the Mllib API's. But i can't seem to create a BlockMatrix with
> the java
> >>>>> API. I tried the following
> >>>>>
> >>>>> Matrix matrixa = Matrices.rand(4, 4, new Random(1000));
> >>>>> List<Tuple2<Tuple2<Integer,Integer>,Matrix>> list = new
> >>>>> ArrayList<Tuple2<Tuple2<Integer, Integer>, Matrix>>();
> >>>>> Tuple2<Integer,Integer> intTuple = new Tuple2<Integer, Integer>(0,0);
> >>>>> Tuple2<Tuple2<Integer,Integer>,Matrix> tuple2MatrixTuple2 = new
> >>>>> Tuple2<Tuple2<Integer, Integer>, Matrix>(intTuple,matrixa );
> >>>>> list.add(tuple2MatrixTuple2);
> >>>>> JavaRDD<Tuple2<Tuple2<Integer, Integer>, Matrix>> rdd =
> >>>>> sc.parallelize(list);
> >>>>>
> >>>>> BlockMatrix blockMatrix = new BlockMatrix(rdd,2,2);
> >>>>>
> >>>>>
> >>>>> but since BlockMatrix only takes
> >>>>>
> "RDD<scala.Tuple2<scala.Tuple2<java.lang.Object,java.lang.Object>,Matrix>>"
> >>>>> this code does not work. sc.parallelize() returns a JavaRDD so the
> two are
> >>>>> not compatible. I also couldn't find any code samples for this. Any
> help on
> >>>>> this would be highly appreciated.
> >>>>>
> >>>>> Best Regards,
> >>>>> Pulasthi
> >>>>> --
> >>>>> Pulasthi S. Wickramasinghe
> >>>>> Graduate Student | Research Assistant
> >>>>> School of Informatics and Computing | Digital Science Center
> >>>>> Indiana University, Bloomington
> >>>>> cell: 224-386-9035
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Pulasthi S. Wickramasinghe
> >>> Graduate Student | Research Assistant
> >>> School of Informatics and Computing | Digital Science Center
> >>> Indiana University, Bloomington
> >>> cell: 224-386-9035
> >>
> >>
> >>
> >>
> >> --
> >>
> >> Architect - Big Data
> >> Ph: +91 99805 99458
> >>
> >> Manthan Systems | Company of the year - Analytics (2014 Frost and
> Sullivan
> >> India ICT)
> >> +++
> >
> >
> >
> >
> > --
> > Pulasthi S. Wickramasinghe
> > Graduate Student | Research Assistant
> > School of Informatics and Computing | Digital Science Center
> > Indiana University, Bloomington
> > cell: 224-386-9035
>
>
>
> --
> 刘忆智
> 广告质量部
>
> MV AD 聚效广告 上海 · 北京 · 广州 · 杭州
> ______________________________________________________________
> 上海市闸北区天目中路585号新梅大厦4楼 200070
> MOB:15021072706
> TEL:021-52559088
> FAX:021-52559089
> EMAIL:liuyz@mvad.com
> HTTP:www.mvad.com
> -------------------------------CONFIDENTIAL ------------------------------
> 本邮件载有秘密信息,请您恪守保密义务,勿向第三人透露。谢谢合作。
> This email communication is confidential. Recipient(s) named above
> is(are) obligated to maintain secrecy and is(are) not permitted to
> disclose the contents of this communication to others. Thank you.
>
>
--
Pulasthi S. Wickramasinghe
Graduate Student | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
cell: 224-386-9035
Re: Creating BlockMatrix with java API
Posted by Pulasthi Supun Wickramasinghe <pu...@gmail.com>.
Hi Sabarish
Thanks, that would indeed solve my problem
Best Regards,
Pulasthi
On Wed, Sep 23, 2015 at 12:55 AM, Sabarish Sasidharan <
sabarish.sasidharan@manthan.com> wrote:
> Hi Pulasthi
>
> You can always use JavaRDD.rdd() to get the scala rdd. So in your case,
>
> new BlockMatrix(rdd.rdd(), 2, 2)
>
> should work.
>
> Regards
> Sab
>
> On Tue, Sep 22, 2015 at 10:50 PM, Pulasthi Supun Wickramasinghe <
> pulasthi911@gmail.com> wrote:
>
>> Hi Yanbo,
>>
>> Thanks for the reply. I thought i might be missing something. Anyway i
>> moved to using scala since it is the complete API.
>>
>> Best Regards,
>> Pulasthi
>>
>> On Tue, Sep 22, 2015 at 7:03 AM, Yanbo Liang <yb...@gmail.com> wrote:
>>
>>> This is due to the distributed matrices like BlockMatrix/RowMatrix/IndexedRowMatrix/CoordinateMatrix do
>>> not provide Java friendly constructors. I have file a SPARK-10757
>>> <https://issues.apache.org/jira/browse/SPARK-10757> to track this issue.
>>>
>>> 2015-09-18 3:36 GMT+08:00 Pulasthi Supun Wickramasinghe <
>>> pulasthi911@gmail.com>:
>>>
>>>> Hi All,
>>>>
>>>> I am new to Spark and i am trying to do some BlockMatrix operations
>>>> with the Mllib API's. But i can't seem to create a BlockMatrix with the
>>>> java API. I tried the following
>>>>
>>>> Matrix matrixa = Matrices.rand(4, 4, new Random(1000));
>>>> List<Tuple2<Tuple2<Integer,Integer>,Matrix>> list = new ArrayList<Tuple2<Tuple2<Integer, Integer>, Matrix>>();
>>>> Tuple2<Integer,Integer> intTuple = new Tuple2<Integer, Integer>(0,0);
>>>> Tuple2<Tuple2<Integer,Integer>,Matrix> tuple2MatrixTuple2 = new Tuple2<Tuple2<Integer, Integer>, Matrix>(intTuple,matrixa );
>>>> list.add(tuple2MatrixTuple2);
>>>> JavaRDD<Tuple2<Tuple2<Integer, Integer>, Matrix>> rdd = sc.parallelize(list);
>>>>
>>>> BlockMatrix blockMatrix = new BlockMatrix(rdd,2,2);
>>>>
>>>>
>>>> but since BlockMatrix only
>>>> takes "RDD<scala.Tuple2<scala.Tuple2<java.lang.Object,java.lang.Object>,Matrix>>"
>>>> this code does not work. sc.parallelize() returns a JavaRDD so the two
>>>> are not compatible. I also couldn't find any code samples for this. Any
>>>> help on this would be highly appreciated.
>>>>
>>>> Best Regards,
>>>> Pulasthi
>>>> --
>>>> Pulasthi S. Wickramasinghe
>>>> Graduate Student | Research Assistant
>>>> School of Informatics and Computing | Digital Science Center
>>>> Indiana University, Bloomington
>>>> cell: 224-386-9035
>>>>
>>>
>>>
>>
>>
>> --
>> Pulasthi S. Wickramasinghe
>> Graduate Student | Research Assistant
>> School of Informatics and Computing | Digital Science Center
>> Indiana University, Bloomington
>> cell: 224-386-9035
>>
>
>
>
> --
>
> Architect - Big Data
> Ph: +91 99805 99458
>
> Manthan Systems | *Company of the year - Analytics (2014 Frost and
> Sullivan India ICT)*
> +++
>
--
Pulasthi S. Wickramasinghe
Graduate Student | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
cell: 224-386-9035
Re: Creating BlockMatrix with java API
Posted by Sabarish Sasidharan <sa...@manthan.com>.
Hi Pulasthi
You can always use JavaRDD.rdd() to get the scala rdd. So in your case,
new BlockMatrix(rdd.rdd(), 2, 2)
should work.
Regards
Sab
On Tue, Sep 22, 2015 at 10:50 PM, Pulasthi Supun Wickramasinghe <
pulasthi911@gmail.com> wrote:
> Hi Yanbo,
>
> Thanks for the reply. I thought i might be missing something. Anyway i
> moved to using scala since it is the complete API.
>
> Best Regards,
> Pulasthi
>
> On Tue, Sep 22, 2015 at 7:03 AM, Yanbo Liang <yb...@gmail.com> wrote:
>
>> This is due to the distributed matrices like BlockMatrix/RowMatrix/IndexedRowMatrix/CoordinateMatrix do
>> not provide Java friendly constructors. I have file a SPARK-10757
>> <https://issues.apache.org/jira/browse/SPARK-10757> to track this issue.
>>
>> 2015-09-18 3:36 GMT+08:00 Pulasthi Supun Wickramasinghe <
>> pulasthi911@gmail.com>:
>>
>>> Hi All,
>>>
>>> I am new to Spark and i am trying to do some BlockMatrix operations with
>>> the Mllib API's. But i can't seem to create a BlockMatrix with the java
>>> API. I tried the following
>>>
>>> Matrix matrixa = Matrices.rand(4, 4, new Random(1000));
>>> List<Tuple2<Tuple2<Integer,Integer>,Matrix>> list = new ArrayList<Tuple2<Tuple2<Integer, Integer>, Matrix>>();
>>> Tuple2<Integer,Integer> intTuple = new Tuple2<Integer, Integer>(0,0);
>>> Tuple2<Tuple2<Integer,Integer>,Matrix> tuple2MatrixTuple2 = new Tuple2<Tuple2<Integer, Integer>, Matrix>(intTuple,matrixa );
>>> list.add(tuple2MatrixTuple2);
>>> JavaRDD<Tuple2<Tuple2<Integer, Integer>, Matrix>> rdd = sc.parallelize(list);
>>>
>>> BlockMatrix blockMatrix = new BlockMatrix(rdd,2,2);
>>>
>>>
>>> but since BlockMatrix only
>>> takes "RDD<scala.Tuple2<scala.Tuple2<java.lang.Object,java.lang.Object>,Matrix>>"
>>> this code does not work. sc.parallelize() returns a JavaRDD so the two
>>> are not compatible. I also couldn't find any code samples for this. Any
>>> help on this would be highly appreciated.
>>>
>>> Best Regards,
>>> Pulasthi
>>> --
>>> Pulasthi S. Wickramasinghe
>>> Graduate Student | Research Assistant
>>> School of Informatics and Computing | Digital Science Center
>>> Indiana University, Bloomington
>>> cell: 224-386-9035
>>>
>>
>>
>
>
> --
> Pulasthi S. Wickramasinghe
> Graduate Student | Research Assistant
> School of Informatics and Computing | Digital Science Center
> Indiana University, Bloomington
> cell: 224-386-9035
>
--
Architect - Big Data
Ph: +91 99805 99458
Manthan Systems | *Company of the year - Analytics (2014 Frost and Sullivan
India ICT)*
+++
Re: Creating BlockMatrix with java API
Posted by Pulasthi Supun Wickramasinghe <pu...@gmail.com>.
Hi Yanbo,
Thanks for the reply. I thought i might be missing something. Anyway i
moved to using scala since it is the complete API.
Best Regards,
Pulasthi
On Tue, Sep 22, 2015 at 7:03 AM, Yanbo Liang <yb...@gmail.com> wrote:
> This is due to the distributed matrices like BlockMatrix/RowMatrix/IndexedRowMatrix/CoordinateMatrix do
> not provide Java friendly constructors. I have file a SPARK-10757
> <https://issues.apache.org/jira/browse/SPARK-10757> to track this issue.
>
> 2015-09-18 3:36 GMT+08:00 Pulasthi Supun Wickramasinghe <
> pulasthi911@gmail.com>:
>
>> Hi All,
>>
>> I am new to Spark and i am trying to do some BlockMatrix operations with
>> the Mllib API's. But i can't seem to create a BlockMatrix with the java
>> API. I tried the following
>>
>> Matrix matrixa = Matrices.rand(4, 4, new Random(1000));
>> List<Tuple2<Tuple2<Integer,Integer>,Matrix>> list = new ArrayList<Tuple2<Tuple2<Integer, Integer>, Matrix>>();
>> Tuple2<Integer,Integer> intTuple = new Tuple2<Integer, Integer>(0,0);
>> Tuple2<Tuple2<Integer,Integer>,Matrix> tuple2MatrixTuple2 = new Tuple2<Tuple2<Integer, Integer>, Matrix>(intTuple,matrixa );
>> list.add(tuple2MatrixTuple2);
>> JavaRDD<Tuple2<Tuple2<Integer, Integer>, Matrix>> rdd = sc.parallelize(list);
>>
>> BlockMatrix blockMatrix = new BlockMatrix(rdd,2,2);
>>
>>
>> but since BlockMatrix only
>> takes "RDD<scala.Tuple2<scala.Tuple2<java.lang.Object,java.lang.Object>,Matrix>>"
>> this code does not work. sc.parallelize() returns a JavaRDD so the two
>> are not compatible. I also couldn't find any code samples for this. Any
>> help on this would be highly appreciated.
>>
>> Best Regards,
>> Pulasthi
>> --
>> Pulasthi S. Wickramasinghe
>> Graduate Student | Research Assistant
>> School of Informatics and Computing | Digital Science Center
>> Indiana University, Bloomington
>> cell: 224-386-9035
>>
>
>
--
Pulasthi S. Wickramasinghe
Graduate Student | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
cell: 224-386-9035
Re: Creating BlockMatrix with java API
Posted by Yanbo Liang <yb...@gmail.com>.
This is due to the distributed matrices like
BlockMatrix/RowMatrix/IndexedRowMatrix/CoordinateMatrix do
not provide Java friendly constructors. I have file a SPARK-10757
<https://issues.apache.org/jira/browse/SPARK-10757> to track this issue.
2015-09-18 3:36 GMT+08:00 Pulasthi Supun Wickramasinghe <
pulasthi911@gmail.com>:
> Hi All,
>
> I am new to Spark and i am trying to do some BlockMatrix operations with
> the Mllib API's. But i can't seem to create a BlockMatrix with the java
> API. I tried the following
>
> Matrix matrixa = Matrices.rand(4, 4, new Random(1000));
> List<Tuple2<Tuple2<Integer,Integer>,Matrix>> list = new ArrayList<Tuple2<Tuple2<Integer, Integer>, Matrix>>();
> Tuple2<Integer,Integer> intTuple = new Tuple2<Integer, Integer>(0,0);
> Tuple2<Tuple2<Integer,Integer>,Matrix> tuple2MatrixTuple2 = new Tuple2<Tuple2<Integer, Integer>, Matrix>(intTuple,matrixa );
> list.add(tuple2MatrixTuple2);
> JavaRDD<Tuple2<Tuple2<Integer, Integer>, Matrix>> rdd = sc.parallelize(list);
>
> BlockMatrix blockMatrix = new BlockMatrix(rdd,2,2);
>
>
> but since BlockMatrix only
> takes "RDD<scala.Tuple2<scala.Tuple2<java.lang.Object,java.lang.Object>,Matrix>>"
> this code does not work. sc.parallelize() returns a JavaRDD so the two
> are not compatible. I also couldn't find any code samples for this. Any
> help on this would be highly appreciated.
>
> Best Regards,
> Pulasthi
> --
> Pulasthi S. Wickramasinghe
> Graduate Student | Research Assistant
> School of Informatics and Computing | Digital Science Center
> Indiana University, Bloomington
> cell: 224-386-9035
>