You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Walter Gillett <wa...@yahoo.com> on 2010/12/23 20:06:37 UTC
Mahout code coverage
I'm interested in learning about and contributing to Mahout and seems to me that
creating some unit tests would be a good place to start. Looking at the code
coverage stats, appears that there's lots of work to do, e.g., there appears to
be no coverage for the Taste module. Should I just pick something
random/interesting to start with, or is there some high-priority part of the
code that you would recommend to work on?
Walter Gillett
Re: Mahout code coverage
Posted by Walter Gillett <wa...@yahoo.com>.
Sorry, my bad. I was looking at this:
https://hudson.apache.org/hudson/job/Mahout-Quality/ws/trunk/target/site/clover/pkg-summary.html
Packages 3-16 in that list are Taste packages that show zero coverage. But
realizing that I have misread this page, there are other Taste packages further
down the list that show significant coverage.
Dmitriy - happy to help, will follow up with you separately.
Thanks,
Walter
________________________________
From: Sean Owen <sr...@gmail.com>
To: dev@mahout.apache.org
Sent: Thu, December 23, 2010 2:20:46 PM
Subject: Re: Mahout code coverage
Hmm, that doesn't sound right -- are you looking at this?
https://hudson.apache.org/hudson/job/Mahout-Quality/clover/
This shows which packages are mostly un-tested and I don't believe it's any
in .cf.taste. I think you're welcome to jump in anywhere you fancy. Example
code does not need tests.
There are lots of bits of code in mahout-math imported from Colt which were
never further cleaned up or tested and are marked "@deprecated" as a result.
I think any of those would be a good place to start, even though some of the
bits are esoteric math and hard to perhaps understand.
Sebastian has done a good job establishing a pattern for unit testing
MapReduce jobs and that could be expanded to other MapReduce jobs. I don't
think the recommender jobs need it and maybe others can volunteer candidates
for testing.
@dev now is probably a good time to talk about whether more of the
deprecated, unused math code can be deleted. If nobody's touched it,
nobody's bothered to test it, nobody's using it (and recall it is not to be
used by external users as it is deprecated) -- would you want someone to
bother writing tests? I feel like this code is just going to be there in the
same state in another 12 months. It's not right to leave deprecated code
indefinitely. What should we do then?
On Thu, Dec 23, 2010 at 7:06 PM, Walter Gillett <wa...@yahoo.com>wrote:
> I'm interested in learning about and contributing to Mahout and seems to me
> that
> creating some unit tests would be a good place to start. Looking at the
> code
> coverage stats, appears that there's lots of work to do, e.g., there
> appears to
> be no coverage for the Taste module. Should I just pick something
> random/interesting to start with, or is there some high-priority part of
> the
> code that you would recommend to work on?
>
> Walter Gillett
>
>
>
>
Re: Mahout code coverage
Posted by Walter Gillett <wa...@yahoo.com>.
Lance,
The JobConf cleanup sounds like a good place to start. I'll be on vacation
through Dec 31 and will start on this at the beginning of January. Thanks,
Walter
________________________________
From: Lance Norskog <go...@gmail.com>
To: dev@mahout.apache.org
Sent: Thu, December 23, 2010 8:35:24 PM
Subject: Re: Mahout code coverage
Walter- in the middle up there is another very useful code sweep:
changing to the new Hadoop signatures.
For example: Hadoop JobConf is old and deprecated, Configuration is
new. JobConf is a subclass of Configuration to assist transition. The
task would be to find everywhere JobConf is used and figure out how to
switch to Configuration. I complained about this a few days ago and
Sean graciously did a sweep. Usually just unit tests have to change,
and of course user code.
Another great project is to fix code that uses 'instanceof' to use
object-oriented programming instead.
Lance
On 12/23/10, Ted Dunning <te...@gmail.com> wrote:
> Please, everybody else should help with this, of course. "TODO delete" will
> be the key string.
>
> On Thu, Dec 23, 2010 at 12:45 PM, Ted Dunning <te...@gmail.com> wrote:
>
>> OK. I don't know about 1.0 just yet, but I will start marking Colt
>> classes
>> for demolition and will post the current list roughly weekly.
>>
>> I will then start deleting classes that seem to have lazy consent.
>>
>>
>> On Thu, Dec 23, 2010 at 12:42 PM, Jeff Eastman <je...@narus.com> wrote:
>>
>>> +1 from me too
>>>
>>> -----Original Message-----
>>> From: Sean Owen [mailto:srowen@gmail.com]
>>> Sent: Thursday, December 23, 2010 12:33 PM
>>> To: dev@mahout.apache.org
>>> Subject: Re: Mahout code coverage
>>>
>>> +1 from me for sure.
>>>
>>> This and getting off Hadoop's deprecated APIs are good steps forward to a
>>> 1.0 release.
>>>
>>> On Thu, Dec 23, 2010 at 8:30 PM, Ted Dunning <te...@gmail.com>
>>> wrote:
>>>
>>> > Can we agree on some sort of way to annotate classes that might be
>>> deleted?
>>> >
>>> > I would suggest:
>>> >
>>> > // TODO remove before 0.5
>>> >
>>> > I am happy to add a bunch of these and summarize all of the annotations
>>> > that
>>> > I or anybody else makes. Sound good?
>>> >
>>> > I agree that now is a great time to drop a lot of unused code and I
>>> would
>>> > be
>>> > happy to cut deep rather than be cautious.
>>>
>>
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Mahout code coverage
Posted by Walter Gillett <wa...@yahoo.com>.
Sean,
Thanks for the guidance, I'll have another go at this-
Walter
________________________________
From: Sean Owen <sr...@gmail.com>
To: dev@mahout.apache.org
Sent: Thu, January 6, 2011 7:24:43 AM
Subject: Re: Mahout code coverage
The idea would be to move off of JobClient too, I believe. The running
theory is that we're trying to standardize on one approach, and about
half the code is using this thing inside Mahout called AbstractJob
that is the current way we're standardizing. It uses the 0.20.x APIs.
Updating anything to use that would be great, though it's a bit more
work. Just emulating the way it invokes jobs would be a step forward
if it moved off of the 0.19.x APIs.
You are right that some things that were deprecated are becoming
un-deprecated. My impression is that we're still "supposed" to be
migrating to 0.20.x APIs, even when using 0.21 and beyond. (We can't
use 0.21 just now, yes, in any event.)
I suggest it's still a worthy task since at the moment it's still
suboptimal to have different ways to run jobs.
On Thu, Jan 6, 2011 at 4:55 AM, Walter Gillett <wa...@yahoo.com> wrote:
> I looked into removing the use of JobConf, but looks like it's too early to do
> that. Removing JobConf from Mahout would require removing JobClient as well.
In
> this email thread I see:
>
Re: Mahout code coverage
Posted by Sean Owen <sr...@gmail.com>.
The idea would be to move off of JobClient too, I believe. The running
theory is that we're trying to standardize on one approach, and about
half the code is using this thing inside Mahout called AbstractJob
that is the current way we're standardizing. It uses the 0.20.x APIs.
Updating anything to use that would be great, though it's a bit more
work. Just emulating the way it invokes jobs would be a step forward
if it moved off of the 0.19.x APIs.
You are right that some things that were deprecated are becoming
un-deprecated. My impression is that we're still "supposed" to be
migrating to 0.20.x APIs, even when using 0.21 and beyond. (We can't
use 0.21 just now, yes, in any event.)
I suggest it's still a worthy task since at the moment it's still
suboptimal to have different ways to run jobs.
On Thu, Jan 6, 2011 at 4:55 AM, Walter Gillett <wa...@yahoo.com> wrote:
> I looked into removing the use of JobConf, but looks like it's too early to do
> that. Removing JobConf from Mahout would require removing JobClient as well. In
> this email thread I see:
>
Re: Mahout code coverage
Posted by Walter Gillett <wa...@yahoo.com>.
Drat, didn't realize that my email would be downgraded to plain text, losing all
the links. I'll put them in below in plain text style.
________________________________
From: Walter Gillett <wa...@yahoo.com>
To: dev@mahout.apache.org
Sent: Wed, January 5, 2011 11:55:07 PM
Subject: Re: Mahout code coverage
I looked into removing the use of JobConf, but looks like it's too early to do
that. Removing JobConf from Mahout would require removing JobClient as well. In
this email thread
(http://www.mail-archive.com/mapreduce-user@hadoop.apache.org/msg01102.html) I
see:
"In [Hadoop] 0.21, JobClient methods are available in
org.apache.hadoop.mapreduce.Job
and org.apache.hadoop.mapreduce.Cluster classes."
but in this email thread
(http://comments.gmane.org/gmane.comp.apache.mahout.user/5381) Sean Owen says:
"For the foreseeable future, we're on [Hadoop] 0.20.2. A major reason is that
Amazon EMR is on 0.20.2 and not 0.21.0 yet."
so it would be hard to eliminate the use of JobClient until we're able to
upgrade to Hadoop 0.21.
Also note here
(http://www.mail-archive.com/mapreduce-user@hadoop.apache.org/msg01107.html):
"Note that JobClient, along with the rest of the "old" API in
org.apache.hadoop.mapred, has been undeprecated in Hadoop 0.21.0 so
you can continue to use it without warnings."
Walter
________________________________
From: Lance Norskog <go...@gmail.com>
To: dev@mahout.apache.org
Sent: Thu, December 23, 2010 8:35:24 PM
Subject: Re: Mahout code coverage
Walter- in the middle up there is another very useful code sweep:
changing to the new Hadoop signatures.
For example: Hadoop JobConf is old and deprecated, Configuration is
new. JobConf is a subclass of Configuration to assist transition. The
task would be to find everywhere JobConf is used and figure out how to
switch to Configuration. I complained about this a few days ago and
Sean graciously did a sweep. Usually just unit tests have to change,
and of course user code.
Another great project is to fix code that uses 'instanceof' to use
object-oriented programming instead.
Lance
On 12/23/10, Ted Dunning <te...@gmail.com> wrote:
> Please, everybody else should help with this, of course. "TODO delete" will
> be the key string.
>
> On Thu, Dec 23, 2010 at 12:45 PM, Ted Dunning <te...@gmail.com> wrote:
>
>> OK. I don't know about 1.0 just yet, but I will start marking Colt
>> classes
>> for demolition and will post the current list roughly weekly.
>>
>> I will then start deleting classes that seem to have lazy consent.
>>
>>
>> On Thu, Dec 23, 2010 at 12:42 PM, Jeff Eastman <je...@narus.com> wrote:
>>
>>> +1 from me too
>>>
>>> -----Original Message-----
>>> From: Sean Owen [mailto:srowen@gmail.com]
>>> Sent: Thursday, December 23, 2010 12:33 PM
>>> To: dev@mahout.apache.org
>>> Subject: Re: Mahout code coverage
>>>
>>> +1 from me for sure.
>>>
>>> This and getting off Hadoop's deprecated APIs are good steps forward to a
>>> 1.0 release.
>>>
>>> On Thu, Dec 23, 2010 at 8:30 PM, Ted Dunning <te...@gmail.com>
>>> wrote:
>>>
>>> > Can we agree on some sort of way to annotate classes that might be
>>> deleted?
>>> >
>>> > I would suggest:
>>> >
>>> > // TODO remove before 0.5
>>> >
>>> > I am happy to add a bunch of these and summarize all of the annotations
>>> > that
>>> > I or anybody else makes. Sound good?
>>> >
>>> > I agree that now is a great time to drop a lot of unused code and I
>>> would
>>> > be
>>> > happy to cut deep rather than be cautious.
>>>
>>
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Mahout code coverage
Posted by Walter Gillett <wa...@yahoo.com>.
I looked into removing the use of JobConf, but looks like it's too early to do
that. Removing JobConf from Mahout would require removing JobClient as well. In
this email thread I see:
In [Hadoop] 0.21, JobClient methods are available in
>org.apache.hadoop.mapreduce.Job
>and org.apache.hadoop.mapreduce.Cluster classes.
but in this email thread Sean Owen says:
For the foreseeable future, we're on [Hadoop] 0.20.2. A major reason is that
>Amazon EMR is on 0.20.2 and not 0.21.0 yet.
so it would be hard to eliminate the use of JobClient until we're able to
upgrade to Hadoop 0.21.
Also note here:
Note that JobClient, along with the rest of the "old" API in
>org.apache.hadoop.mapred, has been undeprecated in Hadoop 0.21.0 so
>you can continue to use it without warnings.
Walter
________________________________
From: Lance Norskog <go...@gmail.com>
To: dev@mahout.apache.org
Sent: Thu, December 23, 2010 8:35:24 PM
Subject: Re: Mahout code coverage
Walter- in the middle up there is another very useful code sweep:
changing to the new Hadoop signatures.
For example: Hadoop JobConf is old and deprecated, Configuration is
new. JobConf is a subclass of Configuration to assist transition. The
task would be to find everywhere JobConf is used and figure out how to
switch to Configuration. I complained about this a few days ago and
Sean graciously did a sweep. Usually just unit tests have to change,
and of course user code.
Another great project is to fix code that uses 'instanceof' to use
object-oriented programming instead.
Lance
On 12/23/10, Ted Dunning <te...@gmail.com> wrote:
> Please, everybody else should help with this, of course. "TODO delete" will
> be the key string.
>
> On Thu, Dec 23, 2010 at 12:45 PM, Ted Dunning <te...@gmail.com> wrote:
>
>> OK. I don't know about 1.0 just yet, but I will start marking Colt
>> classes
>> for demolition and will post the current list roughly weekly.
>>
>> I will then start deleting classes that seem to have lazy consent.
>>
>>
>> On Thu, Dec 23, 2010 at 12:42 PM, Jeff Eastman <je...@narus.com> wrote:
>>
>>> +1 from me too
>>>
>>> -----Original Message-----
>>> From: Sean Owen [mailto:srowen@gmail.com]
>>> Sent: Thursday, December 23, 2010 12:33 PM
>>> To: dev@mahout.apache.org
>>> Subject: Re: Mahout code coverage
>>>
>>> +1 from me for sure.
>>>
>>> This and getting off Hadoop's deprecated APIs are good steps forward to a
>>> 1.0 release.
>>>
>>> On Thu, Dec 23, 2010 at 8:30 PM, Ted Dunning <te...@gmail.com>
>>> wrote:
>>>
>>> > Can we agree on some sort of way to annotate classes that might be
>>> deleted?
>>> >
>>> > I would suggest:
>>> >
>>> > // TODO remove before 0.5
>>> >
>>> > I am happy to add a bunch of these and summarize all of the annotations
>>> > that
>>> > I or anybody else makes. Sound good?
>>> >
>>> > I agree that now is a great time to drop a lot of unused code and I
>>> would
>>> > be
>>> > happy to cut deep rather than be cautious.
>>>
>>
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Mahout code coverage
Posted by Lance Norskog <go...@gmail.com>.
Walter- in the middle up there is another very useful code sweep:
changing to the new Hadoop signatures.
For example: Hadoop JobConf is old and deprecated, Configuration is
new. JobConf is a subclass of Configuration to assist transition. The
task would be to find everywhere JobConf is used and figure out how to
switch to Configuration. I complained about this a few days ago and
Sean graciously did a sweep. Usually just unit tests have to change,
and of course user code.
Another great project is to fix code that uses 'instanceof' to use
object-oriented programming instead.
Lance
On 12/23/10, Ted Dunning <te...@gmail.com> wrote:
> Please, everybody else should help with this, of course. "TODO delete" will
> be the key string.
>
> On Thu, Dec 23, 2010 at 12:45 PM, Ted Dunning <te...@gmail.com> wrote:
>
>> OK. I don't know about 1.0 just yet, but I will start marking Colt
>> classes
>> for demolition and will post the current list roughly weekly.
>>
>> I will then start deleting classes that seem to have lazy consent.
>>
>>
>> On Thu, Dec 23, 2010 at 12:42 PM, Jeff Eastman <je...@narus.com> wrote:
>>
>>> +1 from me too
>>>
>>> -----Original Message-----
>>> From: Sean Owen [mailto:srowen@gmail.com]
>>> Sent: Thursday, December 23, 2010 12:33 PM
>>> To: dev@mahout.apache.org
>>> Subject: Re: Mahout code coverage
>>>
>>> +1 from me for sure.
>>>
>>> This and getting off Hadoop's deprecated APIs are good steps forward to a
>>> 1.0 release.
>>>
>>> On Thu, Dec 23, 2010 at 8:30 PM, Ted Dunning <te...@gmail.com>
>>> wrote:
>>>
>>> > Can we agree on some sort of way to annotate classes that might be
>>> deleted?
>>> >
>>> > I would suggest:
>>> >
>>> > // TODO remove before 0.5
>>> >
>>> > I am happy to add a bunch of these and summarize all of the annotations
>>> > that
>>> > I or anybody else makes. Sound good?
>>> >
>>> > I agree that now is a great time to drop a lot of unused code and I
>>> would
>>> > be
>>> > happy to cut deep rather than be cautious.
>>>
>>
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Mahout code coverage
Posted by Ted Dunning <te...@gmail.com>.
Please, everybody else should help with this, of course. "TODO delete" will
be the key string.
On Thu, Dec 23, 2010 at 12:45 PM, Ted Dunning <te...@gmail.com> wrote:
> OK. I don't know about 1.0 just yet, but I will start marking Colt classes
> for demolition and will post the current list roughly weekly.
>
> I will then start deleting classes that seem to have lazy consent.
>
>
> On Thu, Dec 23, 2010 at 12:42 PM, Jeff Eastman <je...@narus.com> wrote:
>
>> +1 from me too
>>
>> -----Original Message-----
>> From: Sean Owen [mailto:srowen@gmail.com]
>> Sent: Thursday, December 23, 2010 12:33 PM
>> To: dev@mahout.apache.org
>> Subject: Re: Mahout code coverage
>>
>> +1 from me for sure.
>>
>> This and getting off Hadoop's deprecated APIs are good steps forward to a
>> 1.0 release.
>>
>> On Thu, Dec 23, 2010 at 8:30 PM, Ted Dunning <te...@gmail.com>
>> wrote:
>>
>> > Can we agree on some sort of way to annotate classes that might be
>> deleted?
>> >
>> > I would suggest:
>> >
>> > // TODO remove before 0.5
>> >
>> > I am happy to add a bunch of these and summarize all of the annotations
>> > that
>> > I or anybody else makes. Sound good?
>> >
>> > I agree that now is a great time to drop a lot of unused code and I
>> would
>> > be
>> > happy to cut deep rather than be cautious.
>>
>
>
Re: Mahout code coverage
Posted by Ted Dunning <te...@gmail.com>.
OK. I don't know about 1.0 just yet, but I will start marking Colt classes
for demolition and will post the current list roughly weekly.
I will then start deleting classes that seem to have lazy consent.
On Thu, Dec 23, 2010 at 12:42 PM, Jeff Eastman <je...@narus.com> wrote:
> +1 from me too
>
> -----Original Message-----
> From: Sean Owen [mailto:srowen@gmail.com]
> Sent: Thursday, December 23, 2010 12:33 PM
> To: dev@mahout.apache.org
> Subject: Re: Mahout code coverage
>
> +1 from me for sure.
>
> This and getting off Hadoop's deprecated APIs are good steps forward to a
> 1.0 release.
>
> On Thu, Dec 23, 2010 at 8:30 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > Can we agree on some sort of way to annotate classes that might be
> deleted?
> >
> > I would suggest:
> >
> > // TODO remove before 0.5
> >
> > I am happy to add a bunch of these and summarize all of the annotations
> > that
> > I or anybody else makes. Sound good?
> >
> > I agree that now is a great time to drop a lot of unused code and I would
> > be
> > happy to cut deep rather than be cautious.
>
RE: Mahout code coverage
Posted by Jeff Eastman <je...@Narus.com>.
+1 from me too
-----Original Message-----
From: Sean Owen [mailto:srowen@gmail.com]
Sent: Thursday, December 23, 2010 12:33 PM
To: dev@mahout.apache.org
Subject: Re: Mahout code coverage
+1 from me for sure.
This and getting off Hadoop's deprecated APIs are good steps forward to a
1.0 release.
On Thu, Dec 23, 2010 at 8:30 PM, Ted Dunning <te...@gmail.com> wrote:
> Can we agree on some sort of way to annotate classes that might be deleted?
>
> I would suggest:
>
> // TODO remove before 0.5
>
> I am happy to add a bunch of these and summarize all of the annotations
> that
> I or anybody else makes. Sound good?
>
> I agree that now is a great time to drop a lot of unused code and I would
> be
> happy to cut deep rather than be cautious.
Re: Mahout code coverage
Posted by Sean Owen <sr...@gmail.com>.
+1 from me for sure.
This and getting off Hadoop's deprecated APIs are good steps forward to a
1.0 release.
On Thu, Dec 23, 2010 at 8:30 PM, Ted Dunning <te...@gmail.com> wrote:
> Can we agree on some sort of way to annotate classes that might be deleted?
>
> I would suggest:
>
> // TODO remove before 0.5
>
> I am happy to add a bunch of these and summarize all of the annotations
> that
> I or anybody else makes. Sound good?
>
> I agree that now is a great time to drop a lot of unused code and I would
> be
> happy to cut deep rather than be cautious.
Re: Mahout code coverage
Posted by Ted Dunning <te...@gmail.com>.
Can we agree on some sort of way to annotate classes that might be deleted?
I would suggest:
// TODO remove before 0.5
I am happy to add a bunch of these and summarize all of the annotations that
I or anybody else makes. Sound good?
I agree that now is a great time to drop a lot of unused code and I would be
happy to cut deep rather than be cautious.
On Thu, Dec 23, 2010 at 11:20 AM, Sean Owen <sr...@gmail.com> wrote:
> @dev now is probably a good time to talk about whether more of the
> deprecated, unused math code can be deleted. If nobody's touched it,
> nobody's bothered to test it, nobody's using it (and recall it is not to be
> used by external users as it is deprecated) -- would you want someone to
> bother writing tests? I feel like this code is just going to be there in
> the
> same state in another 12 months. It's not right to leave deprecated code
> indefinitely. What should we do then?
>
Re: Mahout code coverage
Posted by Sean Owen <sr...@gmail.com>.
Hmm, that doesn't sound right -- are you looking at this?
https://hudson.apache.org/hudson/job/Mahout-Quality/clover/
This shows which packages are mostly un-tested and I don't believe it's any
in .cf.taste. I think you're welcome to jump in anywhere you fancy. Example
code does not need tests.
There are lots of bits of code in mahout-math imported from Colt which were
never further cleaned up or tested and are marked "@deprecated" as a result.
I think any of those would be a good place to start, even though some of the
bits are esoteric math and hard to perhaps understand.
Sebastian has done a good job establishing a pattern for unit testing
MapReduce jobs and that could be expanded to other MapReduce jobs. I don't
think the recommender jobs need it and maybe others can volunteer candidates
for testing.
@dev now is probably a good time to talk about whether more of the
deprecated, unused math code can be deleted. If nobody's touched it,
nobody's bothered to test it, nobody's using it (and recall it is not to be
used by external users as it is deprecated) -- would you want someone to
bother writing tests? I feel like this code is just going to be there in the
same state in another 12 months. It's not right to leave deprecated code
indefinitely. What should we do then?
On Thu, Dec 23, 2010 at 7:06 PM, Walter Gillett <wa...@yahoo.com>wrote:
> I'm interested in learning about and contributing to Mahout and seems to me
> that
> creating some unit tests would be a good place to start. Looking at the
> code
> coverage stats, appears that there's lots of work to do, e.g., there
> appears to
> be no coverage for the Taste module. Should I just pick something
> random/interesting to start with, or is there some high-priority part of
> the
> code that you would recommend to work on?
>
> Walter Gillett
>
>
>
>
Re: Mahout code coverage
Posted by Lance Norskog <go...@gmail.com>.
As an inexpert implementer, I would love to see a few datasets and
loose definitions of correctness for various operations. For example,
matrix decomposition. There are several algorithms and use cases, but
they all do the same thing: "derive a bunch of matrices and (perhaps)
vectors from a starter dataset. Multiply the derived matrices to form
the starter dataset". Except of course there is an epsilon. This test
would have several criteria for correctness, for example "the
Frobenius norms of the starter and output matrices will have a ratio
between 0.9 and 1.1".
These tests would give a great boost to Mahout's credibility as a
suite of finished tools.
On Fri, Dec 24, 2010 at 7:52 PM, Ted Dunning <te...@gmail.com> wrote:
> I started this some months ago and have been maintaining an lud branch on my
> github mahout mirror.
>
> https://github.com/tdunning/LatentFactorLogLinear/tree/lud
>
> My guess is that a substantial rewrite is in order. Keeping the framework
> is fine, but there is a lot of breaking of abstraction going on in that
> code.
>
> In the mean time, I did test QR decomposition. This is usually a better
> choice than LUD in any case for linear solutions since you can handle least
> squares solutions so easily that way.
>
> On Fri, Dec 24, 2010 at 12:54 PM, Sebastian Schelter <ss...@apache.org> wrote:
>
>> I'd be happy to see LUDecomposition be tested and undeprecated as I
>> don't trust my mathematical skills enough to do it myself.
>>
>> --sebastian
>>
>> Am 24.12.2010 21:37, schrieb Grant Ingersoll:
>> > I'd love to see some benchmarking done of the various algorithms, too,
>> just to add more to the wish list.
>> >
>> > On Dec 23, 2010, at 2:06 PM, Walter Gillett wrote:
>> >
>> >> I'm interested in learning about and contributing to Mahout and seems to
>> me that
>> >> creating some unit tests would be a good place to start. Looking at the
>> code
>> >> coverage stats, appears that there's lots of work to do, e.g., there
>> appears to
>> >> be no coverage for the Taste module. Should I just pick something
>> >> random/interesting to start with, or is there some high-priority part of
>> the
>> >> code that you would recommend to work on?
>> >>
>> >> Walter Gillett
>> >>
>> >>
>> >>
>> >
>> > --------------------------
>> > Grant Ingersoll
>> > http://www.lucidimagination.com
>> >
>>
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Mahout code coverage
Posted by Sebastian Schelter <ss...@apache.org>.
Am 25.12.2010 04:52, schrieb Ted Dunning:
> In the mean time, I did test QR decomposition. This is usually a better
> choice than LUD in any case for linear solutions since you can handle least
> squares solutions so easily that way.
>
QRDecomposition works fine for me. Thank you for the tip
--sebastian
Re: Mahout code coverage
Posted by Ted Dunning <te...@gmail.com>.
I started this some months ago and have been maintaining an lud branch on my
github mahout mirror.
https://github.com/tdunning/LatentFactorLogLinear/tree/lud
My guess is that a substantial rewrite is in order. Keeping the framework
is fine, but there is a lot of breaking of abstraction going on in that
code.
In the mean time, I did test QR decomposition. This is usually a better
choice than LUD in any case for linear solutions since you can handle least
squares solutions so easily that way.
On Fri, Dec 24, 2010 at 12:54 PM, Sebastian Schelter <ss...@apache.org> wrote:
> I'd be happy to see LUDecomposition be tested and undeprecated as I
> don't trust my mathematical skills enough to do it myself.
>
> --sebastian
>
> Am 24.12.2010 21:37, schrieb Grant Ingersoll:
> > I'd love to see some benchmarking done of the various algorithms, too,
> just to add more to the wish list.
> >
> > On Dec 23, 2010, at 2:06 PM, Walter Gillett wrote:
> >
> >> I'm interested in learning about and contributing to Mahout and seems to
> me that
> >> creating some unit tests would be a good place to start. Looking at the
> code
> >> coverage stats, appears that there's lots of work to do, e.g., there
> appears to
> >> be no coverage for the Taste module. Should I just pick something
> >> random/interesting to start with, or is there some high-priority part of
> the
> >> code that you would recommend to work on?
> >>
> >> Walter Gillett
> >>
> >>
> >>
> >
> > --------------------------
> > Grant Ingersoll
> > http://www.lucidimagination.com
> >
>
>
Re: Mahout code coverage
Posted by Sebastian Schelter <ss...@apache.org>.
I'd be happy to see LUDecomposition be tested and undeprecated as I
don't trust my mathematical skills enough to do it myself.
--sebastian
Am 24.12.2010 21:37, schrieb Grant Ingersoll:
> I'd love to see some benchmarking done of the various algorithms, too, just to add more to the wish list.
>
> On Dec 23, 2010, at 2:06 PM, Walter Gillett wrote:
>
>> I'm interested in learning about and contributing to Mahout and seems to me that
>> creating some unit tests would be a good place to start. Looking at the code
>> coverage stats, appears that there's lots of work to do, e.g., there appears to
>> be no coverage for the Taste module. Should I just pick something
>> random/interesting to start with, or is there some high-priority part of the
>> code that you would recommend to work on?
>>
>> Walter Gillett
>>
>>
>>
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
Re: Mahout code coverage
Posted by Grant Ingersoll <gs...@apache.org>.
I'd love to see some benchmarking done of the various algorithms, too, just to add more to the wish list.
On Dec 23, 2010, at 2:06 PM, Walter Gillett wrote:
> I'm interested in learning about and contributing to Mahout and seems to me that
> creating some unit tests would be a good place to start. Looking at the code
> coverage stats, appears that there's lots of work to do, e.g., there appears to
> be no coverage for the Taste module. Should I just pick something
> random/interesting to start with, or is there some high-priority part of the
> code that you would recommend to work on?
>
> Walter Gillett
>
>
>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com
Re: Mahout code coverage
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Walter,
I have a need for a benchmark and also overall compatibility between
seq2sparse vectorization products and the SVD method i am working on. If you
have an access to a good cluster that would certainly help me with
integration of mahout-376 with the rest of recommender/lsi framework there.
I am especially interested in integration testing with the framework to
build wikipedia-trained terms and bigram vectors using SSVD.
I also have a few patches (work in progress) improving dealign with wide an
tall matrices, that would need some integration verification and comparison
benchmarks too.
I would gladly employ somebody's interest in the area.
Thank you, sir.
-Dmitriy
On Thu, Dec 23, 2010 at 11:06 AM, Walter Gillett
<wa...@yahoo.com>wrote:
> I'm interested in learning about and contributing to Mahout and seems to me
> that
> creating some unit tests would be a good place to start. Looking at the
> code
> coverage stats, appears that there's lots of work to do, e.g., there
> appears to
> be no coverage for the Taste module. Should I just pick something
> random/interesting to start with, or is there some high-priority part of
> the
> code that you would recommend to work on?
>
> Walter Gillett
>
>
>
>