Re: GSOC

[Ted Dunning <te...@gmail.com>]

Let's take this back to the mailing list so all can see.

If you are familiar with the stanford parser, then this seems like a
feasible project for you to accomplish.  I would expect that very similar
results could be achieved using simple word or phrase counts, possibly with
the addition of a chunker.  My guess is that the parser would add very
little.

Stefan Henß did some interesting and very simple work, for instance, for
automated FAQ generation that avoided parsing:

http://mail-archives.apache.org/mod_mbox/mahout-user/201102.mbox/%3C4D660038.2000807@gmail.com%3E

On Wed, Mar 23, 2011 at 3:24 AM, Harsh <ha...@vit.ac.in> wrote:

> I want to build over the Stanford parser (the one I am familiar with) and
> want to create a dependency graph for the sentences. The most occurring
> words in any paragraph generally depicts its theme. With the help of the
> dependency developed and word count, I want to guess the theme of the
> paragraph.
>
>

Re: GSOC

[Ted Dunning <te...@gmail.com>]

On Wed, Mar 23, 2011 at 11:18 AM, Harsh <ha...@vit.ac.in> wrote:

> And, to say in strict terms, it is more a computation theory work. It isn't
> totally Mahout-sh, as Ted said. The reason I wanted to do it with Mahout is
> this could be generalized and taken and implemented to cloud. This would
> improve the searching and would yield a better result.
>

Sounds like a reasonable thesis (that pronoun resolution would help the
theme extraction).  It also sounds like a reasonable use of Mahout and
Hadoop (if you have a whack of a lot of data).

The results will be interesting, but not particularly appropriate for
Mahout.


>
> If possible, we can do an IRC session, if you guys are comfortable with it.
>

I am comfortable with it, but I can't spare the time.

Re: GSOC

[Harsh <ha...@vit.ac.in>]

Well, how I saw it was, if we simply do normal word count, we wont get the
full picture as the pronouns present in the paragraph wont add up to
increase the importance of a particular word. Suppose, we are talking about
a person X and after the first sentence if X is referred to by "He" or
"She", then we will actually end up with something that has been given
secondary importance.

And, to say in strict terms, it is more a computation theory work. It isn't
totally Mahout-sh, as Ted said. The reason I wanted to do it with Mahout is
this could be generalized and taken and implemented to cloud. This would
improve the searching and would yield a better result.

If possible, we can do an IRC session, if you guys are comfortable with it.


On Wed, Mar 23, 2011 at 10:09 PM, Ted Dunning <te...@gmail.com> wrote:

> Another important question is whether this is something that is Mahout-ish.
>
> Mahout is a project that supports scalable data mining.  That currently
> includes a mature recommendation framework, less mature clustering and
> classification tools and a smattering of other tools.
>
> What you are proposing sounds a bit more like an application made up of
> different tools, possibly some from Mahout, and some from other sources.
>
> How do you see this?
>
>
> On Wed, Mar 23, 2011 at 9:37 AM, Ted Dunning <te...@gmail.com>wrote:
>
>> Let's take this back to the mailing list so all can see.
>>
>> If you are familiar with the stanford parser, then this seems like a
>> feasible project for you to accomplish.  I would expect that very similar
>> results could be achieved using simple word or phrase counts, possibly with
>> the addition of a chunker.  My guess is that the parser would add very
>> little.
>>
>> Stefan Henß did some interesting and very simple work, for instance, for
>> automated FAQ generation that avoided parsing:
>>
>>
>> http://mail-archives.apache.org/mod_mbox/mahout-user/201102.mbox/%3C4D660038.2000807@gmail.com%3E
>>
>> On Wed, Mar 23, 2011 at 3:24 AM, Harsh <ha...@vit.ac.in> wrote:
>>
>>> I want to build over the Stanford parser (the one I am familiar with) and
>>> want to create a dependency graph for the sentences. The most occurring
>>> words in any paragraph generally depicts its theme. With the help of the
>>> dependency developed and word count, I want to guess the theme of the
>>> paragraph.
>>>
>>>
>

Re: GSOC

[Ted Dunning <te...@gmail.com>]

Another important question is whether this is something that is Mahout-ish.

Mahout is a project that supports scalable data mining.  That currently
includes a mature recommendation framework, less mature clustering and
classification tools and a smattering of other tools.

What you are proposing sounds a bit more like an application made up of
different tools, possibly some from Mahout, and some from other sources.

How do you see this?

On Wed, Mar 23, 2011 at 9:37 AM, Ted Dunning <te...@gmail.com> wrote:

> Let's take this back to the mailing list so all can see.
>
> If you are familiar with the stanford parser, then this seems like a
> feasible project for you to accomplish.  I would expect that very similar
> results could be achieved using simple word or phrase counts, possibly with
> the addition of a chunker.  My guess is that the parser would add very
> little.
>
> Stefan Henß did some interesting and very simple work, for instance, for
> automated FAQ generation that avoided parsing:
>
>
> http://mail-archives.apache.org/mod_mbox/mahout-user/201102.mbox/%3C4D660038.2000807@gmail.com%3E
>
> On Wed, Mar 23, 2011 at 3:24 AM, Harsh <ha...@vit.ac.in> wrote:
>
>> I want to build over the Stanford parser (the one I am familiar with) and
>> want to create a dependency graph for the sentences. The most occurring
>> words in any paragraph generally depicts its theme. With the help of the
>> dependency developed and word count, I want to guess the theme of the
>> paragraph.
>>
>>