You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Matthias Böhmer <ma...@googlemail.com> on 2010/07/15 16:38:20 UTC
Hadoop with MySQL-based data model and Spring integration
Dear mailing list,
I am just wondering, if it will be possible to run Mahout based on
distributed computing with Hadoop, if I have a MySQL-based
implementation of the data model. Namely, I am working with the Spring
Framework and I want to realize the data model and the recommender
system as Spring Beans to make them available to the other components
within my application. Any hints on this issue? Has anyone yet
integrated Mahout into an application based on the Spring Framework?
Best regards,
Matthias
Re: Hadoop with MySQL-based data model and Spring integration
Posted by Sean Owen <sr...@gmail.com>.
No, not really. A relational database is pretty at odds with the
nature of Hadoop, which is about distributing and computing only bits
of data at once, in ways that are independent of other data.
That's not a property of Mahout as much as Hadoop.
But that said, if you're doing recommendation, you might check out the
"pseudo distributed" recommender under .hadoop.pseudo. It's a way to
run n instances of a non-distributed recommender at once on Hadoop.
Those instances aren't distributed, so you have memory scalability
issues.
But it might be OK for you. And sure you can run whatever recommender
you like that way, including one that accesses MySQL.
There should be no issue with Spring per se, no.
2010/7/15 Matthias Böhmer <ma...@googlemail.com>:
> Dear mailing list,
>
> I am just wondering, if it will be possible to run Mahout based on
> distributed computing with Hadoop, if I have a MySQL-based
> implementation of the data model. Namely, I am working with the Spring
> Framework and I want to realize the data model and the recommender
> system as Spring Beans to make them available to the other components
> within my application. Any hints on this issue? Has anyone yet
> integrated Mahout into an application based on the Spring Framework?
>
>
> Best regards,
> Matthias
>
Re: Hadoop with MySQL-based data model and Spring integration
Posted by Sebastian Schelter <ss...@googlemail.com>.
Hi Matthias,
If you're using Spring 3 and don't want to configure constructor
injection via XML (which looks really messy when you have lots of classes),
you can also use Spring's Java Configuration capabilities.
I like it for it's compactness, here's a simple example:
@Configuration
public class ApplicationSetup {
public @Bean recommender() throws Exception() {
DataModel dataModel = ...
ItemSimilarity similarity = ...;
return new GenericItemBasedRecommender(dataModel, similarity);
}
...
}
--sebastian
Am 15.07.2010 16:49, schrieb Joe Spears:
> I use spring to manage mahout in the way you describe. The trick is to use
> Spring's constructor-args rather than property bean definitions. If you do
> this, you don't have to write a single line of code to get spring to run
> mahout.
>
> I can then inject my recommenders into my application classes.
>
> --
>
> As far as your data model is concerned, I had to write my own datamodel
> because I didn't want to code my table-name and column names (I use
> hibernate).... but the docs (and source comments) on how to write a
> datamodel are pretty good... so I created a datamodel that I injected my
> session into and that knew how to issue HQL statement.
>
> Hope this helps
>
> Joe
>
> 2010/7/15 Matthias Böhmer <ma...@googlemail.com>
>
>
>> Dear mailing list,
>>
>> I am just wondering, if it will be possible to run Mahout based on
>> distributed computing with Hadoop, if I have a MySQL-based
>> implementation of the data model. Namely, I am working with the Spring
>> Framework and I want to realize the data model and the recommender
>> system as Spring Beans to make them available to the other components
>> within my application. Any hints on this issue? Has anyone yet
>> integrated Mahout into an application based on the Spring Framework?
>>
>>
>> Best regards,
>> Matthias
>>
>>
>
Re: Hadoop with MySQL-based data model and Spring integration
Posted by Joe Spears <js...@indieplaya.com>.
I'm not using hadoop. My dataset isn't large enough to need distributed
(yet).
2010/7/15 Matthias Böhmer <ma...@m-boehmer.de>
> That sounds similar to the way I designed my system. However, I am
> using annotations instead of XML configurations. And my implementation
> of the data model uses the DAOs that already exist within my system.
>
> Now I am thinking about porting the implementation of my recommender
> system to Hadoop. Therefore I am just wondering, if this is possible
> generally. Do you have any clue on this? Are you running your system
> based on Hadoop?
>
> -- Matthias
>
>
>
> 2010/7/15 Joe Spears <js...@indieplaya.com>:
> > I use spring to manage mahout in the way you describe. The trick is to
> use
> > Spring's constructor-args rather than property bean definitions. If you
> do
> > this, you don't have to write a single line of code to get spring to run
> > mahout.
> >
> > I can then inject my recommenders into my application classes.
> >
> > --
> >
> > As far as your data model is concerned, I had to write my own datamodel
> > because I didn't want to code my table-name and column names (I use
> > hibernate).... but the docs (and source comments) on how to write a
> > datamodel are pretty good... so I created a datamodel that I injected my
> > session into and that knew how to issue HQL statement.
> >
> > Hope this helps
> >
> > Joe
> >
> > 2010/7/15 Matthias Böhmer <ma...@googlemail.com>
> >
> >> Dear mailing list,
> >>
> >> I am just wondering, if it will be possible to run Mahout based on
> >> distributed computing with Hadoop, if I have a MySQL-based
> >> implementation of the data model. Namely, I am working with the Spring
> >> Framework and I want to realize the data model and the recommender
> >> system as Spring Beans to make them available to the other components
> >> within my application. Any hints on this issue? Has anyone yet
> >> integrated Mahout into an application based on the Spring Framework?
> >>
> >>
> >> Best regards,
> >> Matthias
> >>
> >
>
>
>
> --
> --
>
Re: Hadoop with MySQL-based data model and Spring integration
Posted by Matthias Böhmer <ma...@m-boehmer.de>.
That sounds similar to the way I designed my system. However, I am
using annotations instead of XML configurations. And my implementation
of the data model uses the DAOs that already exist within my system.
Now I am thinking about porting the implementation of my recommender
system to Hadoop. Therefore I am just wondering, if this is possible
generally. Do you have any clue on this? Are you running your system
based on Hadoop?
-- Matthias
2010/7/15 Joe Spears <js...@indieplaya.com>:
> I use spring to manage mahout in the way you describe. The trick is to use
> Spring's constructor-args rather than property bean definitions. If you do
> this, you don't have to write a single line of code to get spring to run
> mahout.
>
> I can then inject my recommenders into my application classes.
>
> --
>
> As far as your data model is concerned, I had to write my own datamodel
> because I didn't want to code my table-name and column names (I use
> hibernate).... but the docs (and source comments) on how to write a
> datamodel are pretty good... so I created a datamodel that I injected my
> session into and that knew how to issue HQL statement.
>
> Hope this helps
>
> Joe
>
> 2010/7/15 Matthias Böhmer <ma...@googlemail.com>
>
>> Dear mailing list,
>>
>> I am just wondering, if it will be possible to run Mahout based on
>> distributed computing with Hadoop, if I have a MySQL-based
>> implementation of the data model. Namely, I am working with the Spring
>> Framework and I want to realize the data model and the recommender
>> system as Spring Beans to make them available to the other components
>> within my application. Any hints on this issue? Has anyone yet
>> integrated Mahout into an application based on the Spring Framework?
>>
>>
>> Best regards,
>> Matthias
>>
>
--
--
Re: Hadoop with MySQL-based data model and Spring integration
Posted by Joe Spears <js...@indieplaya.com>.
I use spring to manage mahout in the way you describe. The trick is to use
Spring's constructor-args rather than property bean definitions. If you do
this, you don't have to write a single line of code to get spring to run
mahout.
I can then inject my recommenders into my application classes.
--
As far as your data model is concerned, I had to write my own datamodel
because I didn't want to code my table-name and column names (I use
hibernate).... but the docs (and source comments) on how to write a
datamodel are pretty good... so I created a datamodel that I injected my
session into and that knew how to issue HQL statement.
Hope this helps
Joe
2010/7/15 Matthias Böhmer <ma...@googlemail.com>
> Dear mailing list,
>
> I am just wondering, if it will be possible to run Mahout based on
> distributed computing with Hadoop, if I have a MySQL-based
> implementation of the data model. Namely, I am working with the Spring
> Framework and I want to realize the data model and the recommender
> system as Spring Beans to make them available to the other components
> within my application. Any hints on this issue? Has anyone yet
> integrated Mahout into an application based on the Spring Framework?
>
>
> Best regards,
> Matthias
>