You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by "Burnett, Adam" <Ad...@idearc.com> on 2009/05/14 17:19:54 UTC

Getting started

Hello Mahout community.  I'm just getting started in exploring how to
integrate Mahout into my application. I've written a quick sample but
I'm getting the following exception when trying to use the
SlopeOneRecommender.

INFO: Building average diffs...
Exception in thread "main" java.lang.IllegalArgumentException
	at
org.apache.mahout.cf.taste.impl.common.RandomUtils.isNotPrime(RandomUtil
s.java:93)
	at
org.apache.mahout.cf.taste.impl.common.RandomUtils.nextPrime(RandomUtils
.java:81)
	at
org.apache.mahout.cf.taste.impl.common.RandomUtils.nextTwinPrime(RandomU
tils.java:67)
	at
org.apache.mahout.cf.taste.impl.common.FastMap.rehash(FastMap.java:285)
	at
org.apache.mahout.cf.taste.impl.recommender.slopeone.MemoryDiffStorage.p
runeInconsequentialDiffs(MemoryDiffStorage.java:254)
	at
org.apache.mahout.cf.taste.impl.recommender.slopeone.MemoryDiffStorage.b
uildAverageDiffs(MemoryDiffStorage.java:227)
	at
org.apache.mahout.cf.taste.impl.recommender.slopeone.MemoryDiffStorage.<
init>(MemoryDiffStorage.java:119)
	at
org.apache.mahout.cf.taste.impl.recommender.slopeone.SlopeOneRecommender
.<init>(SlopeOneRecommender.java:65)

I built Mahout from the 774566 revision of the trunk.  I have a
MySQLJDBCDataModel that reads a custom table from my database.  One
question I have is do I have to have the hadoop cluster running before
attempting to use Mahout in my app?  Could not having it running cause
the above error?  Any thoughts in how to proceed from here?  Thanks!

Adam Burnett



RE: Getting started

Posted by "Burnett, Adam" <Ad...@idearc.com>.
Ya you're right, TYPE_SCROLL_INSENSITIVE isn't correct to use for setFetchDirection. I'm not database expert either, even less so with Oracle. Anyway, I modified AbstractJDBCDataModel as follows and it looks like it's working now:

private final Statement statement;

private ResultSetUserIterator(DataSource dataSource, String getUsersSQL) throws TasteException {
      try {
        connection = dataSource.getConnection();
        statement = connection.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_READ_ONLY);
        statement.setFetchDirection(ResultSet.FETCH_UNKNOWN);
        log.debug("Executing SQL query: {}", getUsersSQL);
        resultSet = statement.executeQuery(getUsersSQL);
      } catch (SQLException sqle) {
        close();
        throw new TasteException(sqle);
      }
    }


-Adam

-----Original Message-----
From: Sean Owen [mailto:srowen@gmail.com] 
Sent: Friday, May 15, 2009 12:44 PM
To: Burnett, Adam
Cc: mahout-user@lucene.apache.org
Subject: Re: Getting started

Hmm, tough one. I do write the code in a way that I need to back up in
the ResultSet once in a while, so I need it to be bi-directional. I
had thought FETCH_UNKNOWN was the best way to communicate this, though
by its name, it seems to only say "fetch might be forward or reverse"
not "fetch can be both forward and reverse". I remember adding this
since, I think, MySQL would allow bidirectional access with this
value, but not of course with the default of FETCH_FORWARD.

Clearly this implementation either doesn't support bidirectional
access, or, is still thinking access is forward-only for some reason.

TYPE_SCROLL_INSENSITIVE... seems to be about whether the ResultSet is
worried about new rows appearing during scrolling or something? I
confess I don't know what this is or where it's used; you probably
know more.

So:
1) Can you actually pass TYPE_SCROLL_INSENSITIVE to
setFetchDirection()?? does it work?
2) What about calling setFetchDirection(FETCH_UNKNOWN) on the
ResultSet object itself -- could you try that change locally?
3) else, I will look at ways to avoid backing up in the result set. It
may not be too bad, and, in addition to obviating the problem, might
allow for more efficient DB access.

On Fri, May 15, 2009 at 5:06 PM, Burnett, Adam <Ad...@idearc.com> wrote:
> I'm running into another stack trace once I switched to using an OracleDataSource.  AbstractJDBCDataModel is setting the fetch direction to FETCH_UNKNOWN which causes the ResultSet.isLast() call to fail. I think using TYPE_SCROLL_INSENSITIVE as the direction should resolve this issue.  Do you think it makes sense to change this in the abstract data model class or is this a special case that requires a custom OracleJDBCDataModel?
>
> Here's the trace:
> java.sql.SQLException: Invalid operation for forward only resultset : isLast

Re: Getting started

Posted by Sean Owen <sr...@gmail.com>.
Hmm, tough one. I do write the code in a way that I need to back up in
the ResultSet once in a while, so I need it to be bi-directional. I
had thought FETCH_UNKNOWN was the best way to communicate this, though
by its name, it seems to only say "fetch might be forward or reverse"
not "fetch can be both forward and reverse". I remember adding this
since, I think, MySQL would allow bidirectional access with this
value, but not of course with the default of FETCH_FORWARD.

Clearly this implementation either doesn't support bidirectional
access, or, is still thinking access is forward-only for some reason.

TYPE_SCROLL_INSENSITIVE... seems to be about whether the ResultSet is
worried about new rows appearing during scrolling or something? I
confess I don't know what this is or where it's used; you probably
know more.

So:
1) Can you actually pass TYPE_SCROLL_INSENSITIVE to
setFetchDirection()?? does it work?
2) What about calling setFetchDirection(FETCH_UNKNOWN) on the
ResultSet object itself -- could you try that change locally?
3) else, I will look at ways to avoid backing up in the result set. It
may not be too bad, and, in addition to obviating the problem, might
allow for more efficient DB access.

On Fri, May 15, 2009 at 5:06 PM, Burnett, Adam <Ad...@idearc.com> wrote:
> I'm running into another stack trace once I switched to using an OracleDataSource.  AbstractJDBCDataModel is setting the fetch direction to FETCH_UNKNOWN which causes the ResultSet.isLast() call to fail. I think using TYPE_SCROLL_INSENSITIVE as the direction should resolve this issue.  Do you think it makes sense to change this in the abstract data model class or is this a special case that requires a custom OracleJDBCDataModel?
>
> Here's the trace:
> java.sql.SQLException: Invalid operation for forward only resultset : isLast

RE: Getting started

Posted by "Burnett, Adam" <Ad...@idearc.com>.
I'm running into another stack trace once I switched to using an OracleDataSource.  AbstractJDBCDataModel is setting the fetch direction to FETCH_UNKNOWN which causes the ResultSet.isLast() call to fail. I think using TYPE_SCROLL_INSENSITIVE as the direction should resolve this issue.  Do you think it makes sense to change this in the abstract data model class or is this a special case that requires a custom OracleJDBCDataModel?

Here's the trace:
java.sql.SQLException: Invalid operation for forward only resultset : isLast
	at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)
	at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:131)
	at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:197)
	at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:261)
	at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:269)
	at oracle.jdbc.driver.OracleResultSetImpl.isLast(OracleResultSetImpl.java:368)
	at org.apache.mahout.cf.taste.impl.model.jdbc.AbstractJDBCDataModel$ResultSetUserIterator.hasNext(AbstractJDBCDataModel.java:577)
	at org.apache.mahout.cf.taste.impl.recommender.TopItems.getTopUsers(TopItems.java:81)
	at org.apache.mahout.cf.taste.impl.neighborhood.NearestNUserNeighborhood.getUserNeighborhood(NearestNUserNeighborhood.java:102)
	at org.apache.mahout.cf.taste.impl.recommender.GenericUserBasedRecommender.recommend(GenericUserBasedRecommender.java:83)
	at org.apache.mahout.cf.taste.impl.recommender.AbstractRecommender.recommend(AbstractRecommender.java:53)
	at org.apache.mahout.cf.taste.impl.recommender.CachingRecommender$RecommendationRetriever.get(CachingRecommender.java:198)
	at org.apache.mahout.cf.taste.impl.recommender.CachingRecommender$RecommendationRetriever.get(CachingRecommender.java:185)
	at org.apache.mahout.cf.taste.impl.common.Cache.getAndCacheValue(Cache.java:103)
	at org.apache.mahout.cf.taste.impl.common.Cache.get(Cache.java:77)
	at org.apache.mahout.cf.taste.impl.recommender.CachingRecommender.recommend(CachingRecommender.java:123)
	at org.apache.mahout.cf.taste.impl.recommender.CachingRecommender.recommend(CachingRecommender.java:99)


-Adam 


-Adam

-----Original Message-----
From: Sean Owen [mailto:srowen@gmail.com] 
Sent: Thursday, May 14, 2009 11:29 AM
To: mahout-user@lucene.apache.org; Burnett, Adam
Subject: Re: Getting started

That's my bad. I just committed a fix that makes the result of these
methods in RandomUtils more reasonable for small values and should fix
this corner-ish case.

The org.apache.mahout.cf.taste stuff itself does not require Hadoop;
the Hadoop bindings are all in org.apache.mahout.cf.taste.hadoop.

2009/5/14 Burnett, Adam <Ad...@idearc.com>:
> Hello Mahout community.  I'm just getting started in exploring how to
> integrate Mahout into my application. I've written a quick sample but
> I'm getting the following exception when trying to use the
> SlopeOneRecommender.
>
> INFO: Building average diffs...
> Exception in thread "main" java.lang.IllegalArgumentException
>        at
> org.apache.mahout.cf.taste.impl.common.RandomUtils.isNotPrime(RandomUtil
> s.java:93)
>        at
> org.apache.mahout.cf.taste.impl.common.RandomUtils.nextPrime(RandomUtils
> .java:81)
>        at
> org.apache.mahout.cf.taste.impl.common.RandomUtils.nextTwinPrime(RandomU
> tils.java:67)
>        at
> org.apache.mahout.cf.taste.impl.common.FastMap.rehash(FastMap.java:285)
>        at
> org.apache.mahout.cf.taste.impl.recommender.slopeone.MemoryDiffStorage.p
> runeInconsequentialDiffs(MemoryDiffStorage.java:254)
>        at
> org.apache.mahout.cf.taste.impl.recommender.slopeone.MemoryDiffStorage.b
> uildAverageDiffs(MemoryDiffStorage.java:227)
>        at
> org.apache.mahout.cf.taste.impl.recommender.slopeone.MemoryDiffStorage.<
> init>(MemoryDiffStorage.java:119)
>        at
> org.apache.mahout.cf.taste.impl.recommender.slopeone.SlopeOneRecommender
> .<init>(SlopeOneRecommender.java:65)
>
> I built Mahout from the 774566 revision of the trunk.  I have a
> MySQLJDBCDataModel that reads a custom table from my database.  One
> question I have is do I have to have the hadoop cluster running before
> attempting to use Mahout in my app?  Could not having it running cause
> the above error?  Any thoughts in how to proceed from here?  Thanks!
>
> Adam Burnett
>
>
>

Re: Getting started

Posted by Sean Owen <sr...@gmail.com>.
Good, thanks for the report.

For such a small data set, you might get funky results with any
algorithm. Slope one is generally good when you have relatively few
items compared to users, or, need to update the data quickly at
runtime. Which algorithm gives the best-quality results is harder to
predict, and depends a lot on your domain and the quality of your
input data.

What you describe is the classic recommendation problem indeed. Any of
the algorithms in the package, in theory, solve this problem. I do
think slope one is a good place to start, in any event, because it is
so simple. You can try other algorithms next to see if they produce
better recommendations.

This is the forum to continue that discussion -- ready to consult here
as needed.

Sean

On Thu, May 14, 2009 at 5:01 PM, Burnett, Adam <Ad...@idearc.com> wrote:
> I updated and the exception is gone. Thanks for the quickness!
>
> The way my data is currently organized is I have 2 users that rated 10 different items. There is an 11th item they both rated highly.  Ratings are normalized to be between 0 and 1.  When I ask for recommendations for user A I want items other users who share similar tastes as A have rated highly, but that A has not yet rated. In other words, user A and B both like item 1 and 2, maybe A will also like item 3 since B likes it too.  Is slope one the correct algorithm to use in the case?

RE: Getting started

Posted by "Burnett, Adam" <Ad...@idearc.com>.
I updated and the exception is gone. Thanks for the quickness!

The way my data is currently organized is I have 2 users that rated 10 different items. There is an 11th item they both rated highly.  Ratings are normalized to be between 0 and 1.  When I ask for recommendations for user A I want items other users who share similar tastes as A have rated highly, but that A has not yet rated. In other words, user A and B both like item 1 and 2, maybe A will also like item 3 since B likes it too.  Is slope one the correct algorithm to use in the case?


-Adam

-----Original Message-----
From: Sean Owen [mailto:srowen@gmail.com] 
Sent: Thursday, May 14, 2009 11:29 AM
To: mahout-user@lucene.apache.org; Burnett, Adam
Subject: Re: Getting started

That's my bad. I just committed a fix that makes the result of these
methods in RandomUtils more reasonable for small values and should fix
this corner-ish case.

The org.apache.mahout.cf.taste stuff itself does not require Hadoop;
the Hadoop bindings are all in org.apache.mahout.cf.taste.hadoop.

2009/5/14 Burnett, Adam <Ad...@idearc.com>:
> Hello Mahout community.  I'm just getting started in exploring how to
> integrate Mahout into my application. I've written a quick sample but
> I'm getting the following exception when trying to use the
> SlopeOneRecommender.
>
> INFO: Building average diffs...
> Exception in thread "main" java.lang.IllegalArgumentException
>        at
> org.apache.mahout.cf.taste.impl.common.RandomUtils.isNotPrime(RandomUtil
> s.java:93)
>        at
> org.apache.mahout.cf.taste.impl.common.RandomUtils.nextPrime(RandomUtils
> .java:81)
>        at
> org.apache.mahout.cf.taste.impl.common.RandomUtils.nextTwinPrime(RandomU
> tils.java:67)
>        at
> org.apache.mahout.cf.taste.impl.common.FastMap.rehash(FastMap.java:285)
>        at
> org.apache.mahout.cf.taste.impl.recommender.slopeone.MemoryDiffStorage.p
> runeInconsequentialDiffs(MemoryDiffStorage.java:254)
>        at
> org.apache.mahout.cf.taste.impl.recommender.slopeone.MemoryDiffStorage.b
> uildAverageDiffs(MemoryDiffStorage.java:227)
>        at
> org.apache.mahout.cf.taste.impl.recommender.slopeone.MemoryDiffStorage.<
> init>(MemoryDiffStorage.java:119)
>        at
> org.apache.mahout.cf.taste.impl.recommender.slopeone.SlopeOneRecommender
> .<init>(SlopeOneRecommender.java:65)
>
> I built Mahout from the 774566 revision of the trunk.  I have a
> MySQLJDBCDataModel that reads a custom table from my database.  One
> question I have is do I have to have the hadoop cluster running before
> attempting to use Mahout in my app?  Could not having it running cause
> the above error?  Any thoughts in how to proceed from here?  Thanks!
>
> Adam Burnett
>
>
>

Re: Getting started

Posted by Sean Owen <sr...@gmail.com>.
That's my bad. I just committed a fix that makes the result of these
methods in RandomUtils more reasonable for small values and should fix
this corner-ish case.

The org.apache.mahout.cf.taste stuff itself does not require Hadoop;
the Hadoop bindings are all in org.apache.mahout.cf.taste.hadoop.

2009/5/14 Burnett, Adam <Ad...@idearc.com>:
> Hello Mahout community.  I'm just getting started in exploring how to
> integrate Mahout into my application. I've written a quick sample but
> I'm getting the following exception when trying to use the
> SlopeOneRecommender.
>
> INFO: Building average diffs...
> Exception in thread "main" java.lang.IllegalArgumentException
>        at
> org.apache.mahout.cf.taste.impl.common.RandomUtils.isNotPrime(RandomUtil
> s.java:93)
>        at
> org.apache.mahout.cf.taste.impl.common.RandomUtils.nextPrime(RandomUtils
> .java:81)
>        at
> org.apache.mahout.cf.taste.impl.common.RandomUtils.nextTwinPrime(RandomU
> tils.java:67)
>        at
> org.apache.mahout.cf.taste.impl.common.FastMap.rehash(FastMap.java:285)
>        at
> org.apache.mahout.cf.taste.impl.recommender.slopeone.MemoryDiffStorage.p
> runeInconsequentialDiffs(MemoryDiffStorage.java:254)
>        at
> org.apache.mahout.cf.taste.impl.recommender.slopeone.MemoryDiffStorage.b
> uildAverageDiffs(MemoryDiffStorage.java:227)
>        at
> org.apache.mahout.cf.taste.impl.recommender.slopeone.MemoryDiffStorage.<
> init>(MemoryDiffStorage.java:119)
>        at
> org.apache.mahout.cf.taste.impl.recommender.slopeone.SlopeOneRecommender
> .<init>(SlopeOneRecommender.java:65)
>
> I built Mahout from the 774566 revision of the trunk.  I have a
> MySQLJDBCDataModel that reads a custom table from my database.  One
> question I have is do I have to have the hadoop cluster running before
> attempting to use Mahout in my app?  Could not having it running cause
> the above error?  Any thoughts in how to proceed from here?  Thanks!
>
> Adam Burnett
>
>
>