You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Otis Gospodnetic <ot...@yahoo.com> on 2009/04/14 01:38:02 UTC
Shutting down RefreshHelper or Recommender
Hello,
Here is a quick Taste thing that looks like it could be a "bug". Quotes because it doesn't really break much, but...
So Taste has the handy RefreshHelper thing that monitors the input file and reloads it if it changes. I wanted to be able to explicitly reload my recommender, because I want to create a new one with new parameters. So I thought I'd just create a new recommender instance and let the other one get GCed. Then I started seeing a bunch of these:
INFO: Reading file info...
INFO: Reading file info...
INFO: Reading file info...
At which point I realized that the old recommender that I was "swapping", was still alive and its thread for reloading of the input file was still alive. Ouch. A quick scan of the code doesn't reveal a method that lets one "shut down" the recommender or at least this background thread, but perhaps it is in there somewhere and I'm just not seeing it?
Thanks,
Otis
Re: Shutting down RefreshHelper or Recommender
Posted by Sean Owen <sr...@gmail.com>.
Ah yes sorry I mean 'refresh(null)'. Seriously I am terrible at tracking my
own changes.
This should reload everything. If you want to change params - yeah rebuild
everything. Here again I am open to building better solutions to accommodate
your real use case should it prove painful to rebuild the recommender.
On Apr 14, 2009 9:59 PM, "Otis Gospodnetic" <ot...@yahoo.com>
wrote:
Hi,
> Should be even simpler: > > Recommender r = ...; > ... > r.refresh();
Is that really so? I don't see refresh() in any Recommenders in trunk:
otis@lesina:~/workspace/asf-mahout/core$ grep -ic refresh\(\)
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/*Recommender.java
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/AbstractRecommender.java:0
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/BooleanUserGenericUserBasedRecommender.java:0
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/CachingRecommender.java:0
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericItemBasedRecommender.java:0
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericUserBasedRecommender.java:0
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemAverageRecommender.java:0
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemUserAverageRecommender.java:0
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/TreeClusteringRecommender.java:0
I do see the alternative method that takes in a Collection of Refreshables,
as in the code snippet I sent:
otis@lesina:~/workspace/asf-mahout/core$ grep -i refresh\(Coll
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/*Recommender.java
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/BooleanUserGenericUserBasedRecommender.java:
public void refresh(Collection<Refreshable> alreadyRefreshed) {
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/CachingRecommender.java:
public void refresh(Collection<Refreshable> alreadyRefreshed) {
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericItemBasedRecommender.java:
public void refresh(Collection<Refreshable> alreadyRefreshed) {
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericUserBasedRecommender.java:
public void refresh(Collection<Refreshable> alreadyRefreshed) {
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemAverageRecommender.java:
public void refresh(Collection<Refreshable> alreadyRefreshed) {
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemUserAverageRecommender.java:
public void refresh(Collection<Refreshable> alreadyRefreshed) {
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/TreeClusteringRecommender.java:
public void refresh(Collection<Refreshable> alreadyRefreshed) {
Is refresh() really in there somewhere?
> The RefreshHelper business is supposed to take care of the details. >
Dependent components, all o...
Well, that's one part, yes. In particular, one thing that I want to do is
explicitly force the reload of the underlying input data (FileDataModel).
The other thing that I would like to be able to do is set new parameters for
the Recommender, or rather, classes that go into the recommender, such as
the neighbourhood size, minimal similarity, and a few parameters of my own
that I pass to my custom DataModel loader (it extends
BooleanPrefUserFileDataModel and takes several parameters in its ctor that
tell it which users/items not to load). I *think* that because of this and
for this piece I still need to make a completely new instance of the
Recommender... because I don't want to just do a refresh, I want to refresh,
but with a different set of parameters.
> As to the other question, I don't know of any reason it would behave >
differently on Java 5 vers...
I like this idea. As a matter of fact, I've thought along the same "diff
lines" while working on this recommender stuff.
> It is a good idea but the fact that other files in the same directory have
> an effect too has al...
I don't think that's it - I have only one "re-input.txt" file, which is my
(user,item,pref) file. I create it elsewhere in the system and them just
"mv" it to the right place. Before 0.1 Taste would see something changed
and reload the file (but, again, not on all systems) :)
I'll try explicit refresh now (with refresh(Collection<Refreshable)), which
I hope will really reload the DataModel.
Otis
> Otherwise, hmm, would have to dig in with you and debug. > > On Apr 14,
2009 4:20 PM, "Otis Gosp...
> wrote: > > > OK, cool, so with refresh() this can become a push that looks
like....: > > Rec...
> CollectionrefreshableDM = new ArrayList();
> refreshableDM.add(dm); > r.refresh(refreshableDM); // this seems to be
needed according...
> > To: mahout-dev@lucene.apache.org > > > Sent: Tuesday, April 14, 2009
9:57:34 AM > Subject: Re: ...
Re: Refreshable refresh not quite working
Posted by Sean Owen <sr...@gmail.com>.
On Thu, Apr 16, 2009 at 4:07 PM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
>> Seems like a bit of an oversight but I don't think it affects much, at
>> first glance. Just means a null might get passed through rather than
>> an empty set. I forget if there is any good reason for the difference
>> but I don't see one on looking at this briefly again.
(Actually the difference between the two style is one is just a
shortcut in the case where there is just one dependency. So yeah it
was on purpose on further review.)
> * My goal: re-read the underlying DataModel (the file), so the Recommender is aware of new data.
>
> * My observations: DataModel doesn't get reloaded when I call Recommender's refresh(....) method
> Plus, it is not clear to me what needs to be in that method's Collection<Refreshable> parameter. Which Refreshables should be in there, and should the caller construct this Collection and pass it in, or is the idea to be able to get that Collection from the Recommender itself, in which case we can probably have a simpler no-args refresh() method.
You should call refresh(null) since the process begins with nothing
refreshed. I eyeballed the code and it seemed to do the right thing.
Are you calling this way?
Re: Refreshable refresh not quite working
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi,
> On Wed, Apr 15, 2009 at 3:43 PM, Otis Gospodnetic
> wrote:
> > BooleanUserGenericUserBasedRecommender contains:
> > This never runs because refreshRunnable is always == null because of that ctor
> call with null. Is this a problem?
>
> I don't think so at first glance. It just says this class, itself, has
> nothing to do when refreshed. All it will do is refresh its
> dependencies then.
>
> > Next, in RefreshHelper I see maybeRefresh method that calls refresh, but only
> if a previously "unregistered" Refreshable is passed in. Otherwise is doesn't
> refresh this Refreshable. Is this by design?
>
> Yes the method will run a refresh on that object only if the list of
> things that have already been refreshed in this pass doesn't contain
> it. Say I have an object with two dependencies A and B, and both
> depend on C. Refreshing A means refreshing C, but, refreshing B also
> means refreshing C -- but that would be redundant. So we track that A
> and C are already done. By the time B refreshes, when C gets refreshed
> again, it will call this helper method and the method will recognize
> that it's already done.
Yeah, makes sense.
> > On the other hand, BooleanUserGenericUserBasedRecommender's refresh(...)
> method simply calls refreshHelper.refresh(alreadyRefreshed).
>
> Seems like a bit of an oversight but I don't think it affects much, at
> first glance. Just means a null might get passed through rather than
> an empty set. I forget if there is any good reason for the difference
> but I don't see one on looking at this briefly again.
>
>
> > I was able to get refresh to "work" by moving the refresh call *out* of that
> if block:
>
> So that unconditionally refreshes the target, even if it's been done
> before. Here in fact it gets done twice the first time. All that is
> redundant, though would imagine it works! But no I am not sure it
> should be changed like that.
Right, that's what I observed, too. This "worked" in a sense that this forced a refresh, but it really refreshed things multiple times because of the dependencies, as you described above. So I didn't touch any code here in the end.
> Can you describe the problem you're having -- desired versus actual behavior?
* My goal: re-read the underlying DataModel (the file), so the Recommender is aware of new data.
* My observations: DataModel doesn't get reloaded when I call Recommender's refresh(....) method
Plus, it is not clear to me what needs to be in that method's Collection<Refreshable> parameter. Which Refreshables should be in there, and should the caller construct this Collection and pass it in, or is the idea to be able to get that Collection from the Recommender itself, in which case we can probably have a simpler no-args refresh() method.
* More observations (through debugging): Calling refresh(...) method goes into RefreshHelper's refresh method, which goes into maybeRefresh, but refreshes the file only if it has never been refreshed before -- because it's done in that if block. Moving the actual refresh call out of the if block is bad because it causes multiple refreshes due to the dependency graph.
Finally, it's just not clear where the *actual* refresh call is made in RefreshHelper, other than in that if block.
Thanks,
Otis
P.S.
I have solved my immediate problem by calling refresh(null) directly on my DataModel: model.refresh(null).
This works, but is kind of a workaround because I couldn't get refresh to work through Recommender itself.
Re: Refreshable refresh not quite working
Posted by Sean Owen <sr...@gmail.com>.
Will look at this in detail later but a few initial answers:
On Wed, Apr 15, 2009 at 3:43 PM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
> BooleanUserGenericUserBasedRecommender contains:
> This never runs because refreshRunnable is always == null because of that ctor call with null. Is this a problem?
I don't think so at first glance. It just says this class, itself, has
nothing to do when refreshed. All it will do is refresh its
dependencies then.
> Next, in RefreshHelper I see maybeRefresh method that calls refresh, but only if a previously "unregistered" Refreshable is passed in. Otherwise is doesn't refresh this Refreshable. Is this by design?
Yes the method will run a refresh on that object only if the list of
things that have already been refreshed in this pass doesn't contain
it. Say I have an object with two dependencies A and B, and both
depend on C. Refreshing A means refreshing C, but, refreshing B also
means refreshing C -- but that would be redundant. So we track that A
and C are already done. By the time B refreshes, when C gets refreshed
again, it will call this helper method and the method will recognize
that it's already done.
> On the other hand, BooleanUserGenericUserBasedRecommender's refresh(...) method simply calls refreshHelper.refresh(alreadyRefreshed).
Seems like a bit of an oversight but I don't think it affects much, at
first glance. Just means a null might get passed through rather than
an empty set. I forget if there is any good reason for the difference
but I don't see one on looking at this briefly again.
> I was able to get refresh to "work" by moving the refresh call *out* of that if block:
So that unconditionally refreshes the target, even if it's been done
before. Here in fact it gets done twice the first time. All that is
redundant, though would imagine it works! But no I am not sure it
should be changed like that.
Can you describe the problem you're having -- desired versus actual behavior?
Refreshable refresh not quite working
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Back to this problem. I think the refresh is not quite working.
BooleanUserGenericUserBasedRecommender contains:
this.refreshHelper = new RefreshHelper(null); // note he null
This calls RefreshHelper's ctor:
public RefreshHelper(Callable<?> refreshRunnable) {
And RefreshHelper's refresh method (which, I think, is the key method) contains:
if (refreshRunnable != null) {
try {
System.out.println("Calling refresh runnable");
refreshRunnable.call();
} catch (Exception e) {
log.warn("Unexpected exception while refreshing", e);
}
}
This never runs because refreshRunnable is always == null because of that ctor call with null. Is this a problem?
Next, in RefreshHelper I see maybeRefresh method that calls refresh, but only if a previously "unregistered" Refreshable is passed in. Otherwise is doesn't refresh this Refreshable. Is this by design?
Next, I looked at a bunch of other Refreshables, e.g. CachingNeighborhood, and their refresh(...) looks like this:
public void refresh(Collection<Refreshable> alreadyRefreshed) {
alreadyRefreshed = RefreshHelper.buildRefreshed(alreadyRefreshed);
RefreshHelper.maybeRefresh(alreadyRefreshed, neighborhood);
}
On the other hand, BooleanUserGenericUserBasedRecommender's refresh(...) method simply calls refreshHelper.refresh(alreadyRefreshed).
So somehow it seems that the only actual call that causes a refresh is in that maybeRefresh method, but that refresh call is hidden in the if block that's true only the first time a specific Refreshable is passed in.
I was able to get refresh to "work" by moving the refresh call *out* of that if block:
public static void maybeRefresh(Collection<Refreshable> alreadyRefreshed, Refreshable refreshable) {
if (!alreadyRefreshed.contains(refreshable)) {
alreadyRefreshed.add(refreshable);
- refreshable.refresh(alreadyRefreshed);
+ log.info("Added refreshable: " + refreshable);
}
+ System.out.println("Refreshing: " + alreadyRefreshed);
+ refreshable.refresh(alreadyRefreshed);
}
This does reloads everything multiple times - perhaps because the call is to refreshable.refresh(*alreadyRefreshed*); -- that is, *all* refreshable instead of just the single refreshable passed into the method.
Otis
----- Original Message ----
> From: Otis Gospodnetic <ot...@yahoo.com>
> To: mahout-dev@lucene.apache.org
> Sent: Tuesday, April 14, 2009 4:58:46 PM
> Subject: Re: Shutting down RefreshHelper or Recommender
>
>
> Hi,
>
>
> > Should be even simpler:
> >
> > Recommender r = ...;
> > ...
> > r.refresh();
>
> Is that really so? I don't see refresh() in any Recommenders in trunk:
>
> otis@lesina:~/workspace/asf-mahout/core$ grep -ic refresh\(\)
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/*Recommender.java
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/AbstractRecommender.java:0
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/BooleanUserGenericUserBasedRecommender.java:0
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/CachingRecommender.java:0
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericItemBasedRecommender.java:0
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericUserBasedRecommender.java:0
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemAverageRecommender.java:0
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemUserAverageRecommender.java:0
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/TreeClusteringRecommender.java:0
>
>
> I do see the alternative method that takes in a Collection of Refreshables, as
> in the code snippet I sent:
>
> otis@lesina:~/workspace/asf-mahout/core$ grep -i refresh\(Coll
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/*Recommender.java
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/BooleanUserGenericUserBasedRecommender.java:
> public void refresh(CollectionalreadyRefreshed) {
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/CachingRecommender.java:
> public void refresh(CollectionalreadyRefreshed) {
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericItemBasedRecommender.java:
> public void refresh(CollectionalreadyRefreshed) {
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericUserBasedRecommender.java:
> public void refresh(CollectionalreadyRefreshed) {
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemAverageRecommender.java:
> public void refresh(CollectionalreadyRefreshed) {
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemUserAverageRecommender.java:
> public void refresh(CollectionalreadyRefreshed) {
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/TreeClusteringRecommender.java:
> public void refresh(CollectionalreadyRefreshed) {
>
> Is refresh() really in there somewhere?
>
> > The RefreshHelper business is supposed to take care of the details.
> > Dependent components, all of them, will be refreshed first, in order, and
> > exactly once. Or am I missing something you want to do?
>
> Well, that's one part, yes. In particular, one thing that I want to do is
> explicitly force the reload of the underlying input data (FileDataModel).
>
> The other thing that I would like to be able to do is set new parameters for the
> Recommender, or rather, classes that go into the recommender, such as the
> neighbourhood size, minimal similarity, and a few parameters of my own that I
> pass to my custom DataModel loader (it extends BooleanPrefUserFileDataModel and
> takes several parameters in its ctor that tell it which users/items not to
> load). I *think* that because of this and for this piece I still need to make a
> completely new instance of the Recommender... because I don't want to just do a
> refresh, I want to refresh, but with a different set of parameters.
>
> > As to the other question, I don't know of any reason it would behave
> > differently on Java 5 versus 6. It is supposed to completely reload all
> > data.
> >
> > I have a guess at the problem. I built in a 'feature' which lets you supply
> > 'update' files, named with a prefix like your main data file, which override
> > and update the main data file. The idea is it is a way to push updates to a
> > file without pushing the whole massive data file.
>
> I like this idea. As a matter of fact, I've thought along the same "diff lines"
> while working on this recommender stuff.
>
> > It is a good idea but the fact that other files in the same directory have
> > an effect too has already surprised even me - it created a test bug that a
> > kind user had to figure out for me. Is this possibly a factor? It might
> > explain why it behaves differently on a different machine.
>
> I don't think that's it - I have only one "re-input.txt" file, which is my
> (user,item,pref) file. I create it elsewhere in the system and them just "mv"
> it to the right place. Before 0.1 Taste would see something changed and reload
> the file (but, again, not on all systems) :)
>
> I'll try explicit refresh now (with refresh(Collection
> hope will really reload the DataModel.
>
> Otis
>
> > Otherwise, hmm, would have to dig in with you and debug.
> >
> > On Apr 14, 2009 4:20 PM, "Otis Gospodnetic"
> > wrote:
> >
> >
> > OK, cool, so with refresh() this can become a push that looks like....:
> >
> > Recommender r
> > ....
> > public void init() {
> > r = new SomeRecommender(...);
> > DataModel dm = recommender.getDataModel(); // make DataModel
> > refreshable
> > CollectionrefreshableDM = new ArrayList();
> > refreshableDM.add(dm);
> > r.refresh(refreshableDM); // this seems to be needed according to
> > Refreshable javadoc?
> > }
> >
> > // force DataModel refresh
> > public void forceRefresh() {
> > r.refresh(refreshableDM);
> > }
> > ?
> >
> > One more Q regarding refreshing:
> > Is this meant to completely reload the data (even if it has new users and
> > new items and new preferences)? The reason I'm asking is because I'm seeing
> > a mixed behaviour depending on which system I run my recommender app on. On
> > one system I can get the refresh to reload the data and "see" any new
> > users/items. This system runs with Java 1.6 and the data lives on the local
> > disk. On another system I can't get the changes to the data model file to
> > be visible to the recommender. This system runs Java 1.5 and the data file
> > lives on the mounted NAS disk. I *do* see "INFO: Reading file info...", but
> > the recommender still doesn't see, say, new users, and thus gives 0
> > recommendations for them. Both systems run slightly pre-0.1 code.
> >
> > Thanks, Otis ----- Original Message ---- > From: Sean Owen
> >
> >
> > > To: mahout-dev@lucene.apache.org
> >
> > > Sent: Tuesday, April 14, 2009 9:57:34 AM > Subject: Re: Shutting down
> > RefreshHelper or Recommender...
> >
> > > Yes the change is in 0.1. No you can always have everything
> > reload/recompute > with the refresh() ...
> >
> > > wrote: > > > Ah, I didn't even realize you took it out ... so it's not in
> > 0.1? (I'm > behind on ...
> > > ----- Original Message ---- > From: Sean Owen > To:
> >
> > > mahout-dev@lucene.apache.... > > > wrote: > > > > Hello, > > > > Here is a
> > quick Taste thing that...
Re: Shutting down RefreshHelper or Recommender
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Yes, pulled the trunk, recompiled, put in my local lib/, run the app with mvn as mentioned below.
Yes, 1 minute has passed and the file was modified, but refresh() was not explicitly called (this method doesn't even really exist in any Recommenders in trunk as far as I can tell).
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Sean Owen <sr...@gmail.com>
> To: mahout-user@lucene.apache.org
> Sent: Tuesday, April 14, 2009 5:37:06 PM
> Subject: Re: Shutting down RefreshHelper or Recommender
>
> Yeah I don't get this. You are using the latest code?
> It makes sense that a reload happens once, the first time anything is
> accessed. After that, it should only happen if something calls
> refresh(), *and*, it has been at least 1 minute since the last time
> the data was loaded.
> Does this all match your situation?
>
> On Tue, Apr 14, 2009 at 10:31 PM, Otis Gospodnetic
> wrote:
> >
> > I'm moving this to mahout-user.
> >
> > It looks like that Refresher background thread is still there:
> >
> > $ (mvn clean compile war:war jetty:run) &
> > ... the thole thing starts, I get recommendations for a user...
> > ... fake file change by touching the file...
> > $ touch /rt/apps/recommendation-engine/re-input.txt
> > ... wait a little...
> > Apr 14, 2009 5:28:53 PM org.slf4j.impl.JCLLoggerAdapter info
> > INFO: Reading file info...
> >
> >
> > Ha!
> > My understanding was that Sean removed this and this should no longer be
> happening on its own, no?
Re: Shutting down RefreshHelper or Recommender
Posted by Sean Owen <sr...@gmail.com>.
Yeah I don't get this. You are using the latest code?
It makes sense that a reload happens once, the first time anything is
accessed. After that, it should only happen if something calls
refresh(), *and*, it has been at least 1 minute since the last time
the data was loaded.
Does this all match your situation?
On Tue, Apr 14, 2009 at 10:31 PM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
>
> I'm moving this to mahout-user.
>
> It looks like that Refresher background thread is still there:
>
> $ (mvn clean compile war:war jetty:run) &
> ... the thole thing starts, I get recommendations for a user...
> ... fake file change by touching the file...
> $ touch /rt/apps/recommendation-engine/re-input.txt
> ... wait a little...
> Apr 14, 2009 5:28:53 PM org.slf4j.impl.JCLLoggerAdapter info
> INFO: Reading file info...
>
>
> Ha!
> My understanding was that Sean removed this and this should no longer be happening on its own, no?
Re: Estimating detailed preferences
Posted by Sean Owen <sr...@gmail.com>.
Yes it is up to you, or any caller, to extend existing DataModels to build
objects that carry more information. You can use this extra information in
customized alogrithms however you see fit. You do not just need to use
DetailedPreference. You are free to create other implementations with
whatever info you like.
Your specific question seems to be, rather than pass two 'ID' parameters, is
it better to pass them inside a Preference object? This seems more indirect
to me at first glance so I am inclined to say no.
If you mean you need information from the Preference that the user expresses
for the item - the method can look it up. From an API perspective, I do not
think the caller should be forced to do this. Does that still allow you to
get the behavior you want?
Sean
On Apr 15, 2009 3:03 PM, "Andre Panisson" <pa...@di.unito.it> wrote:
Hi Sean,
I have a doubt about estimating preferences with help of more information
than only user and item ids. What if I have other information like date for
example? I see that there is an implementation of DetailedPreference that
provide an object container for this information, and I think it is simple
to provide a dataModel with DetailedPreference as the base type of the
preferences, and a Recommender that uses it to learn. But the problem is
when I need to use the Recommender interface: it has a method
estimatePreference(Object userID, Object itemID) where you can provide only
user and item ids. I think it would be more straightforward to have a method
estimatePreference(Preference pref), where the given Preference argument
have only the preference value undefined. Then, in the method
implementation, one could make casts and take all provided information. What
you think?
André
Estimating detailed preferences
Posted by Andre Panisson <pa...@di.unito.it>.
Hi Sean,
I have a doubt about estimating preferences with help of more
information than only user and item ids. What if I have other
information like date for example? I see that there is an
implementation of DetailedPreference that provide an object container
for this information, and I think it is simple to provide a dataModel
with DetailedPreference as the base type of the preferences, and a
Recommender that uses it to learn. But the problem is when I need to
use the Recommender interface: it has a method
estimatePreference(Object userID, Object itemID) where you can provide
only user and item ids. I think it would be more straightforward to
have a method estimatePreference(Preference pref), where the given
Preference argument have only the preference value undefined. Then, in
the method implementation, one could make casts and take all provided
information. What you think?
André
Re: Shutting down RefreshHelper or Recommender
Posted by Sean Owen <sr...@gmail.com>.
Sure, that's OK. I have changed FileDataModel such that you should be
able to simply use it. Remind me later to delete
BooleanPrefFileDataModel and BooleanUserGenericDataModel -- or you are
welcome to do it yourself.
On Wed, Apr 15, 2009 at 12:11 PM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
>
> If the code can do that (and I thought I saw you mentioning automatic boolean case recognition to somebody a few days ago), that would be good.... but let's leave the class in there for another week ... after I'm done with a release (uses trunk). :) I'll add this to JIRA, so we don't forget.
Re: Shutting down RefreshHelper or Recommender
Posted by Otis Gospodnetic <ot...@yahoo.com>.
If the code can do that (and I thought I saw you mentioning automatic boolean case recognition to somebody a few days ago), that would be good.... but let's leave the class in there for another week ... after I'm done with a release (uses trunk). :) I'll add this to JIRA, so we don't forget.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Sean Owen <sr...@gmail.com>
> To: mahout-user@lucene.apache.org
> Sent: Tuesday, April 14, 2009 11:56:42 PM
> Subject: Re: Shutting down RefreshHelper or Recommender
>
> Ay of course, forgot you were using this version. Yes that should be
> corrected, will do in SVN. Ah the peril and copy and paste... actually
> I think we can perhaps merge the two classes.
>
> FileDataModel itself already, as of recently, will recognize lines
> like "123,ABC" (versus "123,ABC,1") and interpret them as
> "BooleanPreference" objects. This is half of what you need. You also
> need it to generate BooleanPrefUser objects too in this case. I think
> I can do that. I think then that the "Boolean" version of
> FileDataModel and GenericDataModel can simply be removed. Does that
> square with your needs / thoughts?
>
> Sean
>
> On Apr 14, 2009 11:11 PM, "Otis Gospodnetic" wrote:
>
>
> Ah, I think I found it in BooleanPrefUserFileDataModel, Sean:
>
> private final class RefreshTimerTask extends TimerTask {
>
> @Override
> public void run() {
> if (loaded) {
> long newModified = dataFile.lastModified();
> if (newModified > lastModified) {
> log.debug("File has changed; reloading...");
> lastModified = newModified;
> reload();
> }
> }
> }
> }
>
>
> ....and I just managed to comment it out and commit it accidentally :)
>
> It looks like that was the last remaining instance of this "relic":
>
> $ asf-mahout/core$ ffjg RefreshTimerTask
> ./src/main/java/org/apache/mahout/cf/taste/impl/model/file/BooleanPrefUserFileDataModel.java://
> timer.schedule(new RefreshTimerTask(), RELOAD_CHECK_INTERVAL_MS,
> RELOAD_CHECK_INTERVAL_MS);
> ./src/main/java/org/apache/mahout/cf/taste/impl/model/file/BooleanPrefUserFileDataModel.java://
> private final class RefreshTimerTask extends TimerTask {
>
> It looks like that indeed got rid of the background refresher thread
> and now I'm left to my own explicit refresh calls.
>
> Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> ----- Original Message ---- > ...
>
> > To: mahout-user@lucene.apache.org > Sent: Tuesday, April 14, 2009 5:31:34 PM >
> Subject: Re: Shutti...
>
> > I'm moving this to mahout-user. > > It looks like that Refresher background
> thread is still there...
>
> > > To: mahout-dev@lucene.apache.org > > Sent: Tuesday, April 14, 2009 4:58:46
> PM > > Subject: Re: S...
Re: Shutting down RefreshHelper or Recommender
Posted by Sean Owen <sr...@gmail.com>.
Ay of course, forgot you were using this version. Yes that should be
corrected, will do in SVN. Ah the peril and copy and paste... actually
I think we can perhaps merge the two classes.
FileDataModel itself already, as of recently, will recognize lines
like "123,ABC" (versus "123,ABC,1") and interpret them as
"BooleanPreference" objects. This is half of what you need. You also
need it to generate BooleanPrefUser objects too in this case. I think
I can do that. I think then that the "Boolean" version of
FileDataModel and GenericDataModel can simply be removed. Does that
square with your needs / thoughts?
Sean
On Apr 14, 2009 11:11 PM, "Otis Gospodnetic" <ot...@yahoo.com> wrote:
Ah, I think I found it in BooleanPrefUserFileDataModel, Sean:
private final class RefreshTimerTask extends TimerTask {
@Override
public void run() {
if (loaded) {
long newModified = dataFile.lastModified();
if (newModified > lastModified) {
log.debug("File has changed; reloading...");
lastModified = newModified;
reload();
}
}
}
}
....and I just managed to comment it out and commit it accidentally :)
It looks like that was the last remaining instance of this "relic":
$ asf-mahout/core$ ffjg RefreshTimerTask
./src/main/java/org/apache/mahout/cf/taste/impl/model/file/BooleanPrefUserFileDataModel.java://
timer.schedule(new RefreshTimerTask(), RELOAD_CHECK_INTERVAL_MS,
RELOAD_CHECK_INTERVAL_MS);
./src/main/java/org/apache/mahout/cf/taste/impl/model/file/BooleanPrefUserFileDataModel.java://
private final class RefreshTimerTask extends TimerTask {
It looks like that indeed got rid of the background refresher thread
and now I'm left to my own explicit refresh calls.
Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ---- > ...
> To: mahout-user@lucene.apache.org > Sent: Tuesday, April 14, 2009 5:31:34 PM > Subject: Re: Shutti...
> I'm moving this to mahout-user. > > It looks like that Refresher background thread is still there...
> > To: mahout-dev@lucene.apache.org > > Sent: Tuesday, April 14, 2009 4:58:46 PM > > Subject: Re: S...
Re: Shutting down RefreshHelper or Recommender
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Ah, I think I found it in BooleanPrefUserFileDataModel, Sean:
private final class RefreshTimerTask extends TimerTask {
@Override
public void run() {
if (loaded) {
long newModified = dataFile.lastModified();
if (newModified > lastModified) {
log.debug("File has changed; reloading...");
lastModified = newModified;
reload();
}
}
}
}
....and I just managed to comment it out and commit it accidentally :)
It looks like that was the last remaining instance of this "relic":
$ asf-mahout/core$ ffjg RefreshTimerTask
./src/main/java/org/apache/mahout/cf/taste/impl/model/file/BooleanPrefUserFileDataModel.java:// timer.schedule(new RefreshTimerTask(), RELOAD_CHECK_INTERVAL_MS, RELOAD_CHECK_INTERVAL_MS);
./src/main/java/org/apache/mahout/cf/taste/impl/model/file/BooleanPrefUserFileDataModel.java:// private final class RefreshTimerTask extends TimerTask {
It looks like that indeed got rid of the background refresher thread and now I'm left to my own explicit refresh calls.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Otis Gospodnetic <ot...@yahoo.com>
> To: mahout-user@lucene.apache.org
> Sent: Tuesday, April 14, 2009 5:31:34 PM
> Subject: Re: Shutting down RefreshHelper or Recommender
>
>
> I'm moving this to mahout-user.
>
> It looks like that Refresher background thread is still there:
>
> $ (mvn clean compile war:war jetty:run) &
> ... the thole thing starts, I get recommendations for a user...
> ... fake file change by touching the file...
> $ touch /rt/apps/recommendation-engine/re-input.txt
> ... wait a little...
> Apr 14, 2009 5:28:53 PM org.slf4j.impl.JCLLoggerAdapter info
> INFO: Reading file info...
>
>
> Ha!
> My understanding was that Sean removed this and this should no longer be
> happening on its own, no?
>
> Thanks,
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
> > From: Otis Gospodnetic
> > To: mahout-dev@lucene.apache.org
> > Sent: Tuesday, April 14, 2009 4:58:46 PM
> > Subject: Re: Shutting down RefreshHelper or Recommender
> >
> >
> > Hi,
> >
> >
> > > Should be even simpler:
> > >
> > > Recommender r = ...;
> > > ...
> > > r.refresh();
> >
> > Is that really so? I don't see refresh() in any Recommenders in trunk:
> >
> > otis@lesina:~/workspace/asf-mahout/core$ grep -ic refresh\(\)
> > ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/*Recommender.java
> >
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/AbstractRecommender.java:0
> >
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/BooleanUserGenericUserBasedRecommender.java:0
> >
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/CachingRecommender.java:0
> >
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericItemBasedRecommender.java:0
> >
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericUserBasedRecommender.java:0
> >
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemAverageRecommender.java:0
> >
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemUserAverageRecommender.java:0
> >
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/TreeClusteringRecommender.java:0
> >
> >
> > I do see the alternative method that takes in a Collection of Refreshables, as
>
> > in the code snippet I sent:
> >
> > otis@lesina:~/workspace/asf-mahout/core$ grep -i refresh\(Coll
> > ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/*Recommender.java
> >
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/BooleanUserGenericUserBasedRecommender.java:
>
> > public void refresh(CollectionalreadyRefreshed) {
> >
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/CachingRecommender.java:
>
> > public void refresh(CollectionalreadyRefreshed) {
> >
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericItemBasedRecommender.java:
>
> > public void refresh(CollectionalreadyRefreshed) {
> >
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericUserBasedRecommender.java:
>
> > public void refresh(CollectionalreadyRefreshed) {
> >
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemAverageRecommender.java:
>
> > public void refresh(CollectionalreadyRefreshed) {
> >
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemUserAverageRecommender.java:
>
> > public void refresh(CollectionalreadyRefreshed) {
> >
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/TreeClusteringRecommender.java:
>
> > public void refresh(CollectionalreadyRefreshed) {
> >
> > Is refresh() really in there somewhere?
> >
> > > The RefreshHelper business is supposed to take care of the details.
> > > Dependent components, all of them, will be refreshed first, in order, and
> > > exactly once. Or am I missing something you want to do?
> >
> > Well, that's one part, yes. In particular, one thing that I want to do is
> > explicitly force the reload of the underlying input data (FileDataModel).
> >
> > The other thing that I would like to be able to do is set new parameters for
> the
> > Recommender, or rather, classes that go into the recommender, such as the
> > neighbourhood size, minimal similarity, and a few parameters of my own that I
> > pass to my custom DataModel loader (it extends BooleanPrefUserFileDataModel
> and
> > takes several parameters in its ctor that tell it which users/items not to
> > load). I *think* that because of this and for this piece I still need to make
> a
> > completely new instance of the Recommender... because I don't want to just do
> a
> > refresh, I want to refresh, but with a different set of parameters.
> >
> > > As to the other question, I don't know of any reason it would behave
> > > differently on Java 5 versus 6. It is supposed to completely reload all
> > > data.
> > >
> > > I have a guess at the problem. I built in a 'feature' which lets you supply
> > > 'update' files, named with a prefix like your main data file, which override
> > > and update the main data file. The idea is it is a way to push updates to a
> > > file without pushing the whole massive data file.
> >
> > I like this idea. As a matter of fact, I've thought along the same "diff
> lines"
> > while working on this recommender stuff.
> >
> > > It is a good idea but the fact that other files in the same directory have
> > > an effect too has already surprised even me - it created a test bug that a
> > > kind user had to figure out for me. Is this possibly a factor? It might
> > > explain why it behaves differently on a different machine.
> >
> > I don't think that's it - I have only one "re-input.txt" file, which is my
> > (user,item,pref) file. I create it elsewhere in the system and them just "mv"
>
> > it to the right place. Before 0.1 Taste would see something changed and
> reload
> > the file (but, again, not on all systems) :)
> >
> > I'll try explicit refresh now (with refresh(Collection
> > hope will really reload the DataModel.
> >
> > Otis
> >
> > > Otherwise, hmm, would have to dig in with you and debug.
> > >
> > > On Apr 14, 2009 4:20 PM, "Otis Gospodnetic"
> > > wrote:
> > >
> > >
> > > OK, cool, so with refresh() this can become a push that looks like....:
> > >
> > > Recommender r
> > > ....
> > > public void init() {
> > > r = new SomeRecommender(...);
> > > DataModel dm = recommender.getDataModel(); // make DataModel
> > > refreshable
> > > CollectionrefreshableDM = new ArrayList();
> > > refreshableDM.add(dm);
> > > r.refresh(refreshableDM); // this seems to be needed according to
> > > Refreshable javadoc?
> > > }
> > >
> > > // force DataModel refresh
> > > public void forceRefresh() {
> > > r.refresh(refreshableDM);
> > > }
> > > ?
> > >
> > > One more Q regarding refreshing:
> > > Is this meant to completely reload the data (even if it has new users and
> > > new items and new preferences)? The reason I'm asking is because I'm seeing
> > > a mixed behaviour depending on which system I run my recommender app on. On
> > > one system I can get the refresh to reload the data and "see" any new
> > > users/items. This system runs with Java 1.6 and the data lives on the local
> > > disk. On another system I can't get the changes to the data model file to
> > > be visible to the recommender. This system runs Java 1.5 and the data file
> > > lives on the mounted NAS disk. I *do* see "INFO: Reading file info...", but
> > > the recommender still doesn't see, say, new users, and thus gives 0
> > > recommendations for them. Both systems run slightly pre-0.1 code.
> > >
> > > Thanks, Otis ----- Original Message ---- > From: Sean Owen
> > >
> > >
> > > > To: mahout-dev@lucene.apache.org
> > >
> > > > Sent: Tuesday, April 14, 2009 9:57:34 AM > Subject: Re: Shutting down
> > > RefreshHelper or Recommender...
> > >
> > > > Yes the change is in 0.1. No you can always have everything
> > > reload/recompute > with the refresh() ...
> > >
> > > > wrote: > > > Ah, I didn't even realize you took it out ... so it's not in
> > > 0.1? (I'm > behind on ...
> > > > ----- Original Message ---- > From: Sean Owen > To:
> > >
> > > > mahout-dev@lucene.apache.... > > > wrote: > > > > Hello, > > > > Here is a
> > > quick Taste thing that...
Re: Shutting down RefreshHelper or Recommender
Posted by Otis Gospodnetic <ot...@yahoo.com>.
I'm moving this to mahout-user.
It looks like that Refresher background thread is still there:
$ (mvn clean compile war:war jetty:run) &
... the thole thing starts, I get recommendations for a user...
... fake file change by touching the file...
$ touch /rt/apps/recommendation-engine/re-input.txt
... wait a little...
Apr 14, 2009 5:28:53 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Reading file info...
Ha!
My understanding was that Sean removed this and this should no longer be happening on its own, no?
Thanks,
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Otis Gospodnetic <ot...@yahoo.com>
> To: mahout-dev@lucene.apache.org
> Sent: Tuesday, April 14, 2009 4:58:46 PM
> Subject: Re: Shutting down RefreshHelper or Recommender
>
>
> Hi,
>
>
> > Should be even simpler:
> >
> > Recommender r = ...;
> > ...
> > r.refresh();
>
> Is that really so? I don't see refresh() in any Recommenders in trunk:
>
> otis@lesina:~/workspace/asf-mahout/core$ grep -ic refresh\(\)
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/*Recommender.java
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/AbstractRecommender.java:0
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/BooleanUserGenericUserBasedRecommender.java:0
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/CachingRecommender.java:0
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericItemBasedRecommender.java:0
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericUserBasedRecommender.java:0
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemAverageRecommender.java:0
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemUserAverageRecommender.java:0
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/TreeClusteringRecommender.java:0
>
>
> I do see the alternative method that takes in a Collection of Refreshables, as
> in the code snippet I sent:
>
> otis@lesina:~/workspace/asf-mahout/core$ grep -i refresh\(Coll
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/*Recommender.java
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/BooleanUserGenericUserBasedRecommender.java:
> public void refresh(CollectionalreadyRefreshed) {
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/CachingRecommender.java:
> public void refresh(CollectionalreadyRefreshed) {
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericItemBasedRecommender.java:
> public void refresh(CollectionalreadyRefreshed) {
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericUserBasedRecommender.java:
> public void refresh(CollectionalreadyRefreshed) {
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemAverageRecommender.java:
> public void refresh(CollectionalreadyRefreshed) {
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemUserAverageRecommender.java:
> public void refresh(CollectionalreadyRefreshed) {
> ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/TreeClusteringRecommender.java:
> public void refresh(CollectionalreadyRefreshed) {
>
> Is refresh() really in there somewhere?
>
> > The RefreshHelper business is supposed to take care of the details.
> > Dependent components, all of them, will be refreshed first, in order, and
> > exactly once. Or am I missing something you want to do?
>
> Well, that's one part, yes. In particular, one thing that I want to do is
> explicitly force the reload of the underlying input data (FileDataModel).
>
> The other thing that I would like to be able to do is set new parameters for the
> Recommender, or rather, classes that go into the recommender, such as the
> neighbourhood size, minimal similarity, and a few parameters of my own that I
> pass to my custom DataModel loader (it extends BooleanPrefUserFileDataModel and
> takes several parameters in its ctor that tell it which users/items not to
> load). I *think* that because of this and for this piece I still need to make a
> completely new instance of the Recommender... because I don't want to just do a
> refresh, I want to refresh, but with a different set of parameters.
>
> > As to the other question, I don't know of any reason it would behave
> > differently on Java 5 versus 6. It is supposed to completely reload all
> > data.
> >
> > I have a guess at the problem. I built in a 'feature' which lets you supply
> > 'update' files, named with a prefix like your main data file, which override
> > and update the main data file. The idea is it is a way to push updates to a
> > file without pushing the whole massive data file.
>
> I like this idea. As a matter of fact, I've thought along the same "diff lines"
> while working on this recommender stuff.
>
> > It is a good idea but the fact that other files in the same directory have
> > an effect too has already surprised even me - it created a test bug that a
> > kind user had to figure out for me. Is this possibly a factor? It might
> > explain why it behaves differently on a different machine.
>
> I don't think that's it - I have only one "re-input.txt" file, which is my
> (user,item,pref) file. I create it elsewhere in the system and them just "mv"
> it to the right place. Before 0.1 Taste would see something changed and reload
> the file (but, again, not on all systems) :)
>
> I'll try explicit refresh now (with refresh(Collection
> hope will really reload the DataModel.
>
> Otis
>
> > Otherwise, hmm, would have to dig in with you and debug.
> >
> > On Apr 14, 2009 4:20 PM, "Otis Gospodnetic"
> > wrote:
> >
> >
> > OK, cool, so with refresh() this can become a push that looks like....:
> >
> > Recommender r
> > ....
> > public void init() {
> > r = new SomeRecommender(...);
> > DataModel dm = recommender.getDataModel(); // make DataModel
> > refreshable
> > CollectionrefreshableDM = new ArrayList();
> > refreshableDM.add(dm);
> > r.refresh(refreshableDM); // this seems to be needed according to
> > Refreshable javadoc?
> > }
> >
> > // force DataModel refresh
> > public void forceRefresh() {
> > r.refresh(refreshableDM);
> > }
> > ?
> >
> > One more Q regarding refreshing:
> > Is this meant to completely reload the data (even if it has new users and
> > new items and new preferences)? The reason I'm asking is because I'm seeing
> > a mixed behaviour depending on which system I run my recommender app on. On
> > one system I can get the refresh to reload the data and "see" any new
> > users/items. This system runs with Java 1.6 and the data lives on the local
> > disk. On another system I can't get the changes to the data model file to
> > be visible to the recommender. This system runs Java 1.5 and the data file
> > lives on the mounted NAS disk. I *do* see "INFO: Reading file info...", but
> > the recommender still doesn't see, say, new users, and thus gives 0
> > recommendations for them. Both systems run slightly pre-0.1 code.
> >
> > Thanks, Otis ----- Original Message ---- > From: Sean Owen
> >
> >
> > > To: mahout-dev@lucene.apache.org
> >
> > > Sent: Tuesday, April 14, 2009 9:57:34 AM > Subject: Re: Shutting down
> > RefreshHelper or Recommender...
> >
> > > Yes the change is in 0.1. No you can always have everything
> > reload/recompute > with the refresh() ...
> >
> > > wrote: > > > Ah, I didn't even realize you took it out ... so it's not in
> > 0.1? (I'm > behind on ...
> > > ----- Original Message ---- > From: Sean Owen > To:
> >
> > > mahout-dev@lucene.apache.... > > > wrote: > > > > Hello, > > > > Here is a
> > quick Taste thing that...
Re: Shutting down RefreshHelper or Recommender
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi,
> Should be even simpler:
>
> Recommender r = ...;
> ...
> r.refresh();
Is that really so? I don't see refresh() in any Recommenders in trunk:
otis@lesina:~/workspace/asf-mahout/core$ grep -ic refresh\(\) ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/*Recommender.java
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/AbstractRecommender.java:0
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/BooleanUserGenericUserBasedRecommender.java:0
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/CachingRecommender.java:0
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericItemBasedRecommender.java:0
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericUserBasedRecommender.java:0
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemAverageRecommender.java:0
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemUserAverageRecommender.java:0
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/TreeClusteringRecommender.java:0
I do see the alternative method that takes in a Collection of Refreshables, as in the code snippet I sent:
otis@lesina:~/workspace/asf-mahout/core$ grep -i refresh\(Coll ./src/main/java/org/apache/mahout/cf/taste/impl/recommender/*Recommender.java
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/BooleanUserGenericUserBasedRecommender.java: public void refresh(Collection<Refreshable> alreadyRefreshed) {
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/CachingRecommender.java: public void refresh(Collection<Refreshable> alreadyRefreshed) {
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericItemBasedRecommender.java: public void refresh(Collection<Refreshable> alreadyRefreshed) {
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/GenericUserBasedRecommender.java: public void refresh(Collection<Refreshable> alreadyRefreshed) {
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemAverageRecommender.java: public void refresh(Collection<Refreshable> alreadyRefreshed) {
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/ItemUserAverageRecommender.java: public void refresh(Collection<Refreshable> alreadyRefreshed) {
./src/main/java/org/apache/mahout/cf/taste/impl/recommender/TreeClusteringRecommender.java: public void refresh(Collection<Refreshable> alreadyRefreshed) {
Is refresh() really in there somewhere?
> The RefreshHelper business is supposed to take care of the details.
> Dependent components, all of them, will be refreshed first, in order, and
> exactly once. Or am I missing something you want to do?
Well, that's one part, yes. In particular, one thing that I want to do is explicitly force the reload of the underlying input data (FileDataModel).
The other thing that I would like to be able to do is set new parameters for the Recommender, or rather, classes that go into the recommender, such as the neighbourhood size, minimal similarity, and a few parameters of my own that I pass to my custom DataModel loader (it extends BooleanPrefUserFileDataModel and takes several parameters in its ctor that tell it which users/items not to load). I *think* that because of this and for this piece I still need to make a completely new instance of the Recommender... because I don't want to just do a refresh, I want to refresh, but with a different set of parameters.
> As to the other question, I don't know of any reason it would behave
> differently on Java 5 versus 6. It is supposed to completely reload all
> data.
>
> I have a guess at the problem. I built in a 'feature' which lets you supply
> 'update' files, named with a prefix like your main data file, which override
> and update the main data file. The idea is it is a way to push updates to a
> file without pushing the whole massive data file.
I like this idea. As a matter of fact, I've thought along the same "diff lines" while working on this recommender stuff.
> It is a good idea but the fact that other files in the same directory have
> an effect too has already surprised even me - it created a test bug that a
> kind user had to figure out for me. Is this possibly a factor? It might
> explain why it behaves differently on a different machine.
I don't think that's it - I have only one "re-input.txt" file, which is my (user,item,pref) file. I create it elsewhere in the system and them just "mv" it to the right place. Before 0.1 Taste would see something changed and reload the file (but, again, not on all systems) :)
I'll try explicit refresh now (with refresh(Collection<Refreshable)), which I hope will really reload the DataModel.
Otis
> Otherwise, hmm, would have to dig in with you and debug.
>
> On Apr 14, 2009 4:20 PM, "Otis Gospodnetic"
> wrote:
>
>
> OK, cool, so with refresh() this can become a push that looks like....:
>
> Recommender r
> ....
> public void init() {
> r = new SomeRecommender(...);
> DataModel dm = recommender.getDataModel(); // make DataModel
> refreshable
> CollectionrefreshableDM = new ArrayList();
> refreshableDM.add(dm);
> r.refresh(refreshableDM); // this seems to be needed according to
> Refreshable javadoc?
> }
>
> // force DataModel refresh
> public void forceRefresh() {
> r.refresh(refreshableDM);
> }
> ?
>
> One more Q regarding refreshing:
> Is this meant to completely reload the data (even if it has new users and
> new items and new preferences)? The reason I'm asking is because I'm seeing
> a mixed behaviour depending on which system I run my recommender app on. On
> one system I can get the refresh to reload the data and "see" any new
> users/items. This system runs with Java 1.6 and the data lives on the local
> disk. On another system I can't get the changes to the data model file to
> be visible to the recommender. This system runs Java 1.5 and the data file
> lives on the mounted NAS disk. I *do* see "INFO: Reading file info...", but
> the recommender still doesn't see, say, new users, and thus gives 0
> recommendations for them. Both systems run slightly pre-0.1 code.
>
> Thanks, Otis ----- Original Message ---- > From: Sean Owen
>
>
> > To: mahout-dev@lucene.apache.org
>
> > Sent: Tuesday, April 14, 2009 9:57:34 AM > Subject: Re: Shutting down
> RefreshHelper or Recommender...
>
> > Yes the change is in 0.1. No you can always have everything
> reload/recompute > with the refresh() ...
>
> > wrote: > > > Ah, I didn't even realize you took it out ... so it's not in
> 0.1? (I'm > behind on ...
> > ----- Original Message ---- > From: Sean Owen > To:
>
> > mahout-dev@lucene.apache.... > > > wrote: > > > > Hello, > > > > Here is a
> quick Taste thing that...
Re: Shutting down RefreshHelper or Recommender
Posted by Sean Owen <sr...@gmail.com>.
Should be even simpler:
Recommender r = ...;
...
r.refresh();
The RefreshHelper business is supposed to take care of the details.
Dependent components, all of them, will be refreshed first, in order, and
exactly once. Or am I missing something you want to do?
As to the other question, I don't know of any reason it would behave
differently on Java 5 versus 6. It is supposed to completely reload all
data.
I have a guess at the problem. I built in a 'feature' which lets you supply
'update' files, named with a prefix like your main data file, which override
and update the main data file. The idea is it is a way to push updates to a
file without pushing the whole massive data file.
It is a good idea but the fact that other files in the same directory have
an effect too has already surprised even me - it created a test bug that a
kind user had to figure out for me. Is this possibly a factor? It might
explain why it behaves differently on a different machine.
Otherwise, hmm, would have to dig in with you and debug.
On Apr 14, 2009 4:20 PM, "Otis Gospodnetic" <ot...@yahoo.com>
wrote:
OK, cool, so with refresh() this can become a push that looks like....:
Recommender r
....
public void init() {
r = new SomeRecommender(...);
DataModel dm = recommender.getDataModel(); // make DataModel
refreshable
Collection<Refreshable> refreshableDM = new ArrayList<Refreshable>();
refreshableDM.add(dm);
r.refresh(refreshableDM); // this seems to be needed according to
Refreshable javadoc?
}
// force DataModel refresh
public void forceRefresh() {
r.refresh(refreshableDM);
}
?
One more Q regarding refreshing:
Is this meant to completely reload the data (even if it has new users and
new items and new preferences)? The reason I'm asking is because I'm seeing
a mixed behaviour depending on which system I run my recommender app on. On
one system I can get the refresh to reload the data and "see" any new
users/items. This system runs with Java 1.6 and the data lives on the local
disk. On another system I can't get the changes to the data model file to
be visible to the recommender. This system runs Java 1.5 and the data file
lives on the mounted NAS disk. I *do* see "INFO: Reading file info...", but
the recommender still doesn't see, say, new users, and thus gives 0
recommendations for them. Both systems run slightly pre-0.1 code.
Thanks, Otis ----- Original Message ---- > From: Sean Owen <sr...@gmail.com>
> To: mahout-dev@lucene.apache.org
> Sent: Tuesday, April 14, 2009 9:57:34 AM > Subject: Re: Shutting down
RefreshHelper or Recommender...
> Yes the change is in 0.1. No you can always have everything
reload/recompute > with the refresh() ...
> wrote: > > > Ah, I didn't even realize you took it out ... so it's not in
0.1? (I'm > behind on ...
> ----- Original Message ---- > From: Sean Owen > To:
> mahout-dev@lucene.apache.... > > > wrote: > > > > Hello, > > > > Here is a
quick Taste thing that...
Re: Shutting down RefreshHelper or Recommender
Posted by Otis Gospodnetic <ot...@yahoo.com>.
OK, cool, so with refresh() this can become a push that looks like....:
Recommender r
....
public void init() {
r = new SomeRecommender(...);
DataModel dm = recommender.getDataModel(); // make DataModel refreshable
Collection<Refreshable> refreshableDM = new ArrayList<Refreshable>();
refreshableDM.add(dm);
r.refresh(refreshableDM); // this seems to be needed according to Refreshable javadoc?
}
// force DataModel refresh
public void forceRefresh() {
r.refresh(refreshableDM);
}
?
One more Q regarding refreshing:
Is this meant to completely reload the data (even if it has new users and new items and new preferences)? The reason I'm asking is because I'm seeing a mixed behaviour depending on which system I run my recommender app on. On one system I can get the refresh to reload the data and "see" any new users/items. This system runs with Java 1.6 and the data lives on the local disk. On another system I can't get the changes to the data model file to be visible to the recommender. This system runs Java 1.5 and the data file lives on the mounted NAS disk. I *do* see "INFO: Reading file info...", but the recommender still doesn't see, say, new users, and thus gives 0 recommendations for them. Both systems run slightly pre-0.1 code.
Thanks,
Otis
----- Original Message ----
> From: Sean Owen <sr...@gmail.com>
> To: mahout-dev@lucene.apache.org
> Sent: Tuesday, April 14, 2009 9:57:34 AM
> Subject: Re: Shutting down RefreshHelper or Recommender
>
> Yes the change is in 0.1. No you can always have everything reload/recompute
> with the refresh() method. The idea is to call that on Recommender and let
> it manage the update of all dependencies, including the model.
>
> I think you rightly noted earlier that the refresh mechanism would be better
> if rewritten as a 'push' rather than 'pull' system. If this is causing
> trouble we can get on redesigning it.
>
> On Apr 14, 2009 2:15 PM, "Otis Gospodnetic"
> wrote:
>
>
> Ah, I didn't even realize you took it out ... so it's not in 0.1? (I'm
> behind on email, etc.).
> So reloading the data model now means creating a new instance of the
> recommender?
>
>
> Thanks,
> Otis
>
> ----- Original Message ---- > From: Sean Owen > To:
> mahout-dev@lucene.apache....
>
> > wrote: > > > > Hello, > > > > Here is a quick Taste thing that looks like
> it could be a "bug". Qu...
Re: Shutting down RefreshHelper or Recommender
Posted by Sean Owen <sr...@gmail.com>.
Yes the change is in 0.1. No you can always have everything reload/recompute
with the refresh() method. The idea is to call that on Recommender and let
it manage the update of all dependencies, including the model.
I think you rightly noted earlier that the refresh mechanism would be better
if rewritten as a 'push' rather than 'pull' system. If this is causing
trouble we can get on redesigning it.
On Apr 14, 2009 2:15 PM, "Otis Gospodnetic" <ot...@yahoo.com>
wrote:
Ah, I didn't even realize you took it out ... so it's not in 0.1? (I'm
behind on email, etc.).
So reloading the data model now means creating a new instance of the
recommender?
Thanks,
Otis
----- Original Message ---- > From: Sean Owen <sr...@gmail.com> > To:
mahout-dev@lucene.apache....
> wrote: > > > > Hello, > > > > Here is a quick Taste thing that looks like
it could be a "bug". Qu...
Re: Shutting down RefreshHelper or Recommender
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Ah, I didn't even realize you took it out ... so it's not in 0.1? (I'm behind on email, etc.).
So reloading the data model now means creating a new instance of the recommender?
Thanks,
Otis
----- Original Message ----
> From: Sean Owen <sr...@gmail.com>
> To: mahout-dev@lucene.apache.org
> Sent: Tuesday, April 14, 2009 12:46:37 AM
> Subject: Re: Shutting down RefreshHelper or Recommender
>
> Yeah that's part of the reason I just took this out -- there is no
> longer an automatic reload. Can you update and/or just remove this
> part of the code in your branch as well?
>
> I took a second look and didn't see anything else, like a static
> reference somewhere, that would prevent a Recommender from being
> GC'ed.
>
>
>
> On Tue, Apr 14, 2009 at 12:38 AM, Otis Gospodnetic
> wrote:
> >
> > Hello,
> >
> > Here is a quick Taste thing that looks like it could be a "bug". Quotes
> because it doesn't really break much, but...
> >
> > So Taste has the handy RefreshHelper thing that monitors the input file and
> reloads it if it changes. I wanted to be able to explicitly reload my
> recommender, because I want to create a new one with new parameters. So I
> thought I'd just create a new recommender instance and let the other one get
> GCed. Then I started seeing a bunch of these:
> > INFO: Reading file info...
> > INFO: Reading file info...
> > INFO: Reading file info...
> >
> > At which point I realized that the old recommender that I was "swapping", was
> still alive and its thread for reloading of the input file was still alive.
> Ouch. A quick scan of the code doesn't reveal a method that lets one "shut
> down" the recommender or at least this background thread, but perhaps it is in
> there somewhere and I'm just not seeing it?
> >
> > Thanks,
> > Otis
> >
Re: Shutting down RefreshHelper or Recommender
Posted by Sean Owen <sr...@gmail.com>.
Yeah that's part of the reason I just took this out -- there is no
longer an automatic reload. Can you update and/or just remove this
part of the code in your branch as well?
I took a second look and didn't see anything else, like a static
reference somewhere, that would prevent a Recommender from being
GC'ed.
On Tue, Apr 14, 2009 at 12:38 AM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
>
> Hello,
>
> Here is a quick Taste thing that looks like it could be a "bug". Quotes because it doesn't really break much, but...
>
> So Taste has the handy RefreshHelper thing that monitors the input file and reloads it if it changes. I wanted to be able to explicitly reload my recommender, because I want to create a new one with new parameters. So I thought I'd just create a new recommender instance and let the other one get GCed. Then I started seeing a bunch of these:
> INFO: Reading file info...
> INFO: Reading file info...
> INFO: Reading file info...
>
> At which point I realized that the old recommender that I was "swapping", was still alive and its thread for reloading of the input file was still alive. Ouch. A quick scan of the code doesn't reveal a method that lets one "shut down" the recommender or at least this background thread, but perhaps it is in there somewhere and I'm just not seeing it?
>
> Thanks,
> Otis
>