Re: Taste Recommenders, LuceneIterable Weights and loading custom code

[Benson Margulies <bi...@gmail.com>]

I'd rather integrate Spring. But you've heard me on this subject on Solr before.

On Wed, Jun 17, 2009 at 6:10 PM, Grant Ingersoll<gs...@apache.org> wrote:
> We are starting to get a couple of places where users can insert their own
> custom implementations.  Recommenders, Weight calculators (new today) and
> some other areas are all prime examples. Currently, it is likely the case
> that one has to do the whole un-jar/un-war dance in order to add their stuff
> in.
>
> Solr had this problem and solved it through a custom class loader and the
> Solr Home capability.  Essentially, Solr defines a HOME which has a lib dir
> in it.  When someone wants to put their own custom code into Solr they
> merely have to jar up their code and drop it in the lib directory and then
> tell Solr about it.
>
> Any thoughts on grabbing this functionality (SolrResourceLoader) and
> modifying it for Mahout for some of these things?  Not sure how it would
> play out for things running in Hadoop, but for both Taste and the utility
> functions it seems like a win.
>
> -Grant
>

Re: Taste Recommenders, LuceneIterable Weights and loading custom code

[Gérard Dupont <ge...@gmail.com>]

Hi all,

I'm not involved into mahout dev team, but I have my little experience on
the problem mentionned (ie loading custom implementations for certain
things).

We faced this problem in our team to manage RDF data. We defined an
"RDFHelperAPI" and then any of us compete to make to the usefull and fast
implementation. So to manage this and allow to load one implementation or
another, we simply made a Factory that load the implmeenation class name
from a property file and load it up. It's similar to Spring injection in a
way but really simple. So you can add any number of implementations in jars
and then dynamically change the one use by playing with the factory or let
use the configuration file to define the right one.

I'm pretty sure that this actually not the best solution (nor the brightest)
but that was just to mention it.

br,



On Thu, Jun 18, 2009 at 05:55, Ted Dunning <te...@gmail.com> wrote:

> +2
>
> On Wed, Jun 17, 2009 at 4:36 PM, Sean Owen <sr...@gmail.com> wrote:
>
> > I may be on a tangent now but I suppose my basic reaction is: skip this
> > complexity and build this as an extensible library. As always I am open
> to
> > being convinced otherwise.
> >
>



-- 
Gérard Dupont
Information Processing Control and Cognition (IPCC) - EADS DS
http://weblab-project.org

Document & Learning team - LITIS Laboratory

Re: Taste Recommenders, LuceneIterable Weights and loading custom code

[Ted Dunning <te...@gmail.com>]

+2

On Wed, Jun 17, 2009 at 4:36 PM, Sean Owen <sr...@gmail.com> wrote:

> I may be on a tangent now but I suppose my basic reaction is: skip this
> complexity and build this as an extensible library. As always I am open to
> being convinced otherwise.
>

Re: Taste Recommenders, LuceneIterable Weights and loading custom code

[Sean Owen <sr...@gmail.com>]

My general opinion is that it is easy to overthink and overengineer things
like this. I am open to new approaches as long as they can demonstrate real
value in real use cases. To state my biases: I am not a fan of Spring in
general. It makes 'calling a constructor with args' into something
difficult.

Grant starts to hit this on the head: there are many deployment scenarios
and supporting one tends to make others trickier. I would rather not favor a
deployment scenario over another, so have generally avoided trying to inject
things like custom classloaders or DI or config files.

My world view is that Mahout is foremost a library. It is a simple .jar you
can embed.

In my world I built one more layer around this, without disturbing the core
library, to support the most likely deployment scenario as a web app. That
seemed like a win. But it didn't make the code somehow web app-centric.

I may be on a tangent now but I suppose my basic reaction is: skip this
complexity and build this as an extensible library. As always I am open to
being convinced otherwise.

On Jun 18, 2009 12:26 AM, "Ted Dunning" <te...@gmail.com> wrote:

I have had very bad experiences with the complexity explosion that is common
with Spring, but I very much like the philosophical underpinnings that drive
it.

I really don't see the need for a custom class loader.  All we really have
to do is inject a few classes here and there and they should be in the class
path.  It isn't unreasonable to imagine that this injection be done in
spring style although the basic idea is so simple that it keeps getting
re-implemented by umpteen different people.  The real problem with spring is
that if you let it in the door, people start using it for everything and
there is a huge presumption that it is in charge of everything.  Suddenly
you have 5000 line XML configuration files and only 5% of your engineers can
change them without cascading failures that are only detected at run-time
and have *very* strange symptoms.

A really different approach would be to use something like a scripting
language as the glue.  That removes the problem by moving the center of
control and allowing the user to inject their implementations fairly
simply.

Similarly, if we just remove the idea that mahout is a command line program
that must load the user's classes, and have the user write the command line
side, then things are suddenly simple again because Mahout is just a
library.

On Wed, Jun 17, 2009 at 3:36 PM, Grant Ingersoll <gs...@apache.org>wrote:

> Ah, yeah, true. Not sure why that slipped my mind. Probably had Solr on >
the brain. > > > On J...

Re: Taste Recommenders, LuceneIterable Weights and loading custom code

[Ted Dunning <te...@gmail.com>]

I have had very bad experiences with the complexity explosion that is common
with Spring, but I very much like the philosophical underpinnings that drive
it.

I really don't see the need for a custom class loader.  All we really have
to do is inject a few classes here and there and they should be in the class
path.  It isn't unreasonable to imagine that this injection be done in
spring style although the basic idea is so simple that it keeps getting
re-implemented by umpteen different people.  The real problem with spring is
that if you let it in the door, people start using it for everything and
there is a huge presumption that it is in charge of everything.  Suddenly
you have 5000 line XML configuration files and only 5% of your engineers can
change them without cascading failures that are only detected at run-time
and have *very* strange symptoms.

A really different approach would be to use something like a scripting
language as the glue.  That removes the problem by moving the center of
control and allowing the user to inject their implementations fairly
simply.

Similarly, if we just remove the idea that mahout is a command line program
that must load the user's classes, and have the user write the command line
side, then things are suddenly simple again because Mahout is just a
library.

On Wed, Jun 17, 2009 at 3:36 PM, Grant Ingersoll <gs...@apache.org>wrote:

> Ah, yeah, true.  Not sure why that slipped my mind.  Probably had Solr on
> the brain.
>
>
> On Jun 17, 2009, at 6:22 PM, Benson Margulies wrote:
>
>  I'd rather integrate Spring. But you've heard me on this subject on Solr
>> before.
>>
>>

Re: Taste Recommenders, LuceneIterable Weights and loading custom code

[Grant Ingersoll <gs...@apache.org>]

Ah, yeah, true.  Not sure why that slipped my mind.  Probably had Solr  
on the brain.

On Jun 17, 2009, at 6:22 PM, Benson Margulies wrote:

> I'd rather integrate Spring. But you've heard me on this subject on  
> Solr before.
>
> On Wed, Jun 17, 2009 at 6:10 PM, Grant  
> Ingersoll<gs...@apache.org> wrote:
>> We are starting to get a couple of places where users can insert  
>> their own
>> custom implementations.  Recommenders, Weight calculators (new  
>> today) and
>> some other areas are all prime examples. Currently, it is likely  
>> the case
>> that one has to do the whole un-jar/un-war dance in order to add  
>> their stuff
>> in.
>>
>> Solr had this problem and solved it through a custom class loader  
>> and the
>> Solr Home capability.  Essentially, Solr defines a HOME which has a  
>> lib dir
>> in it.  When someone wants to put their own custom code into Solr  
>> they
>> merely have to jar up their code and drop it in the lib directory  
>> and then
>> tell Solr about it.
>>
>> Any thoughts on grabbing this functionality (SolrResourceLoader) and
>> modifying it for Mahout for some of these things?  Not sure how it  
>> would
>> play out for things running in Hadoop, but for both Taste and the  
>> utility
>> functions it seems like a win.
>>
>> -Grant
>>