You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Grant Ingersoll <gs...@apache.org> on 2008/04/13 12:41:47 UTC
Various Ideas from Mahout BOF
Back from ApacheCon EU and can't sleep...
Had a fairly decent BOF the other night given the stage we are at.
Got to meet both Isabel and Karl. Had a good number of people in the
room and had some nice discussions on the challenges we are facing.
One of the main discussions was on obtaining data to be used by anyone
coming in to the project who may not have data. There were a couple
of good suggestions: Apache mail archives, Apache web logs (possibly,
after being dedacted), as well as the usual suspects like Wikipedia,
Reuters collection (David Lewis), etc. If someone wants to take up
creating a simple util for cleaning up mail archives for use, that
would be a great contribution.
Another one was about obtaining computing resources. Had some
possible contacts that may be able to help, but am not going to commit
anything at this point, until it is official. But hopefully we can
get something in place for committers. Also was the suggestion that
we work with Hadoop and Nutch to figure out ways to obtain computing
resources.
Also had some discussion on how Mahout will be accepted in the
community. Will academics be interested? Will companies be
interested? My take was that both are taking a wait and see approach,
probably, but we do have some early supporters which seems to be
positive. I think a lot of companies view ML as "secret sauce", so
we'll see. However, I think this will go away as we build out our
algorithms. As w/ search, ML will become more of a commodity and it
will be not that be so much that any company is using ML, but look at
all the cool things your application can do with ML.
On other notes, seemed to be fairly good general interest in Mahout.
Cheers,
Grant
Re: Various Ideas from Mahout BOF
Posted by Karl Wettin <ka...@gmail.com>.
Grant Ingersoll skrev:
> One of the main discussions was on obtaining data to be used by anyone
> coming in to the project who may not have data.
Rulebased instance generation and the possibillity to grow an existing
data set with morphed instances would be very nice contributions.
> There were a couple of good suggestions: Apache mail archives, Apache
> web logs (possibly, after being dedacted), as well as the usual
> suspects like Wikipedia, Reuters collection (David Lewis), etc.
I have been thinking about analyzing emails. We could try tagging
everything we send to the list:
"This is a problem"
"The problem is identical to problem #"
"This is a solution to a problem"
"This is confirmation that the solution works or does not work"
"This is an alternative solution to solution #"
"About classification"
"About clustering"
et c.
Perhaps treat tag names as unique per email address and map them to some
top level of tags. [sug]
> If someone wants to take up creating a simple util for cleaning up
> mail archives for use, that would be a great contribution.
Apache James does mbox archives and grammar to parse anything but
replies from Hossman is simple stuff. [sug] [prob]
It would be great if someone somehow could use that information to
implement an email thread de-jeopardy quotation formatter so I always
got to read emails the way I like it. [wish]
karl
Re: Various Ideas from Mahout BOF
Posted by Grant Ingersoll <gs...@apache.org>.
On Apr 13, 2008, at 10:16 AM, Toby DiPasquale wrote:
> On Sun, Apr 13, 2008 at 6:41 AM, Grant Ingersoll
> <gs...@apache.org> wrote:
>> Back from ApacheCon EU and can't sleep...
>>
>> Had a fairly decent BOF the other night given the stage we are at.
>> Got to
>> meet both Isabel and Karl. Had a good number of people in the room
>> and had
>> some nice discussions on the challenges we are facing.
>>
>> One of the main discussions was on obtaining data to be used by
>> anyone
>> coming in to the project who may not have data. There were a
>> couple of good
>> suggestions: Apache mail archives, Apache web logs (possibly,
>> after being
>> dedacted), as well as the usual suspects like Wikipedia, Reuters
>> collection
>> (David Lewis), etc. If someone wants to take up creating a simple
>> util for
>> cleaning up mail archives for use, that would be a great
>> contribution.
>
> What about http://infochimps.org/ ?
Cool, there is also http://theinfo.org/
>
>
>> Also had some discussion on how Mahout will be accepted in the
>> community.
>> Will academics be interested? Will companies be interested? My
>> take was
>> that both are taking a wait and see approach, probably, but we do
>> have some
>> early supporters which seems to be positive. I think a lot of
>> companies
>> view ML as "secret sauce", so we'll see. However, I think this
>> will go away
>> as we build out our algorithms. As w/ search, ML will become more
>> of a
>> commodity and it will be not that be so much that any company is
>> using ML,
>> but look at all the cool things your application can do with ML.
>
> Speaking from the business side of things, I am very interested in
> Mahout. A lot of places lack the internal resources to really build
> and QA a lot of these algorithms ourselves. Having a vibrant community
> around this is basically bring a little bit of R to the big-time, IMO.
> I'd like nothing better than for ML to become commoditized on top of
> Hadoop. And it seems as if there's plenty of interest from the
> academic side, given all the volunteering for the GSoC for Mahout ;-)
I agree. I'm sold on it (can't you tell?), but the question did come
up in the BOF. The key for us is translating this momentum into good
code, examples, docs and the more volunteers the better that all will
be.
-Grant
Re: Various Ideas from Mahout BOF
Posted by Toby DiPasquale <to...@cbcg.net>.
On Sun, Apr 13, 2008 at 6:41 AM, Grant Ingersoll <gs...@apache.org> wrote:
> Back from ApacheCon EU and can't sleep...
>
> Had a fairly decent BOF the other night given the stage we are at. Got to
> meet both Isabel and Karl. Had a good number of people in the room and had
> some nice discussions on the challenges we are facing.
>
> One of the main discussions was on obtaining data to be used by anyone
> coming in to the project who may not have data. There were a couple of good
> suggestions: Apache mail archives, Apache web logs (possibly, after being
> dedacted), as well as the usual suspects like Wikipedia, Reuters collection
> (David Lewis), etc. If someone wants to take up creating a simple util for
> cleaning up mail archives for use, that would be a great contribution.
What about http://infochimps.org/ ?
> Also had some discussion on how Mahout will be accepted in the community.
> Will academics be interested? Will companies be interested? My take was
> that both are taking a wait and see approach, probably, but we do have some
> early supporters which seems to be positive. I think a lot of companies
> view ML as "secret sauce", so we'll see. However, I think this will go away
> as we build out our algorithms. As w/ search, ML will become more of a
> commodity and it will be not that be so much that any company is using ML,
> but look at all the cool things your application can do with ML.
Speaking from the business side of things, I am very interested in
Mahout. A lot of places lack the internal resources to really build
and QA a lot of these algorithms ourselves. Having a vibrant community
around this is basically bring a little bit of R to the big-time, IMO.
I'd like nothing better than for ML to become commoditized on top of
Hadoop. And it seems as if there's plenty of interest from the
academic side, given all the volunteering for the GSoC for Mahout ;-)
--
Toby DiPasquale
Re: Various Ideas from Mahout BOF
Posted by Sean Owen <sr...@gmail.com>.
On Sun, Apr 13, 2008 at 6:41 AM, Grant Ingersoll <gs...@apache.org> wrote:
> One of the main discussions was on obtaining data to be used by anyone
> coming in to the project who may not have data. There were a couple of good
For what it's worth, the example that I included with my code was the
GroupLens data (http://grouplens.org/). This may be specific to the
algorithms I was implementing (it's movie recommendation data) but I
bet we can turn up some publicly-available data sets too. Even if we
can't redistribute them, as in the case of GroupLens, we can write
examples that let people plug in the data to them. Just a thought.
> Another one was about obtaining computing resources. Had some possible
> contacts that may be able to help, but am not going to commit anything at
> this point, until it is official. But hopefully we can get something in
> place for committers. Also was the suggestion that we work with Hadoop and
> Nutch to figure out ways to obtain computing resources.
Does Amazon donate resources to developers of open-source projects?
If I can ever get a Google App Engine account, and see if it fits our
needs, I could inquire about getting some modest resources for free
for Mahout. I think they'd be quite inclined in spirit to do this,
just a question of how ready GAE is.