You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Ken Krugler <kk...@transpac.com> on 2009/11/16 20:14:40 UTC
Feedback on "big data" analysis talk
Hi all,
I'm going to be giving a talk at the Bay Area ACM data mining SIG in
December, and I need to finalize my topic today :)
I was going to expand on my "Web mining for SEO keywords" talk from
the ACM unconference a few weeks back.
But the fact that I'll have 10s to 100s of millions of web page data
to work with, from my public terabyte dataset crawl, makes me want to
apply Mahout to the data.
So I'm going to toss out one idea...
- I'd like to automatically generate a timeline of events.
- I can extract dates and 2-to-4 word "terms" from web pages.
- Could I use LDA to create clusters of common terms for dates?
I don't want to get into named entity extraction, or anything that
involves more than simple data extraction and then the application of
a scalable algorithm currently supported by Mahout.
Looking for feedback on the above, in terms of feasibility, level of
effort, interest by others to help, etc.
Or if somebody else has a suggestion for something simple that could
provide interesting/obvious results, I'm all ears.
Thanks!
-- Ken
PS - Some additional thoughts on the problem...
* I would skip any of my multi-word terms that start/end with
stopwords.
* I would skip any pages that don't have associated dates.
* I wouldn't worry about trying to figure out which date "the
previous Tuesday" referred to.
* I could skip terms that have too many different associated dates
- though handling dates for a term that cluster (are close to one
another) would be important.
--------------------------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g
Re: Feedback on "big data" analysis talk
Posted by Ted Dunning <te...@gmail.com>.
http://nlp.stanford.edu/pubs/hall-emnlp08.pdf
See also:
http://www.stanford.edu/class/cs229/proj2007/HaleyHallRodgers-TopicCorrelationsOverTime.pdf
http://www.aclweb.org/anthology/D/D09/D09-1109.pdf
On Mon, Nov 16, 2009 at 1:52 PM, Grant Ingersoll <gs...@apache.org>wrote:
> I recall David Hall saying something about using LDA for Topics over time
> which would be interesting too. Last we talked, he said he had an
> implementation as well, but that was this summer and I haven't seen a patch.
>
--
Ted Dunning, CTO
DeepDyve
Re: Feedback on "big data" analysis talk
Posted by Miles Osborne <mi...@inf.ed.ac.uk>.
see this paper:
http://www.cs.umass.edu/~mimno/papers/fast-topic-model.pdf
--efficient LDA over streams. looks like what you need here
Miles
2009/11/16 Grant Ingersoll <gs...@apache.org>:
>
> On Nov 16, 2009, at 2:14 PM, Ken Krugler wrote:
>
>> Hi all,
>>
>> I'm going to be giving a talk at the Bay Area ACM data mining SIG in December, and I need to finalize my topic today :)
>>
>> I was going to expand on my "Web mining for SEO keywords" talk from the ACM unconference a few weeks back.
>>
>> But the fact that I'll have 10s to 100s of millions of web page data to work with, from my public terabyte dataset crawl, makes me want to apply Mahout to the data.
>>
>> So I'm going to toss out one idea...
>>
>> - I'd like to automatically generate a timeline of events.
>>
>> - I can extract dates and 2-to-4 word "terms" from web pages.
>>
>> - Could I use LDA to create clusters of common terms for dates?
>
> I recall David Hall saying something about using LDA for Topics over time which would be interesting too. Last we talked, he said he had an implementation as well, but that was this summer and I haven't seen a patch.
>
>>
>> I don't want to get into named entity extraction, or anything that involves more than simple data extraction and then the application of a scalable algorithm currently supported by Mahout.
>>
>> Looking for feedback on the above, in terms of feasibility, level of effort, interest by others to help, etc.
>>
>> Or if somebody else has a suggestion for something simple that could provide interesting/obvious results, I'm all ears.
>
>
>
>
>
>
--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
Re: Feedback on "big data" analysis talk
Posted by Grant Ingersoll <gs...@apache.org>.
On Nov 16, 2009, at 2:14 PM, Ken Krugler wrote:
> Hi all,
>
> I'm going to be giving a talk at the Bay Area ACM data mining SIG in December, and I need to finalize my topic today :)
>
> I was going to expand on my "Web mining for SEO keywords" talk from the ACM unconference a few weeks back.
>
> But the fact that I'll have 10s to 100s of millions of web page data to work with, from my public terabyte dataset crawl, makes me want to apply Mahout to the data.
>
> So I'm going to toss out one idea...
>
> - I'd like to automatically generate a timeline of events.
>
> - I can extract dates and 2-to-4 word "terms" from web pages.
>
> - Could I use LDA to create clusters of common terms for dates?
I recall David Hall saying something about using LDA for Topics over time which would be interesting too. Last we talked, he said he had an implementation as well, but that was this summer and I haven't seen a patch.
>
> I don't want to get into named entity extraction, or anything that involves more than simple data extraction and then the application of a scalable algorithm currently supported by Mahout.
>
> Looking for feedback on the above, in terms of feasibility, level of effort, interest by others to help, etc.
>
> Or if somebody else has a suggestion for something simple that could provide interesting/obvious results, I'm all ears.
Re: Feedback on "big data" analysis talk
Posted by Isabel Drost <is...@apache.org>.
On Monday 16 November 2009 20:14:40 Ken Krugler wrote:
> I'm going to be giving a talk at the Bay Area ACM data mining SIG in
> December, and I need to finalize my topic today :)
Would you please share some information on topics discussed, the feedback you
got and the like after the event with the Mahout people? I'd be really
interested on the discussions and ideas the data mining SIG are interessted
in.
Cheers,
Isabel
--
|\ _,,,---,,_ Web: <http://www.isabel-drost.de>
/,`.-'`' -. ;-;;,_
|,4- ) )-,_..;\ ( `'-'
'---''(_/--' `-'\_) (fL) IM: <xm...@spaceboyz.net>
Re: Feedback on "big data" analysis talk
Posted by Ted Dunning <te...@gmail.com>.
Ken,
This sounds like (another) good idea.
What I would suggest is that you have three or four kinds of dates:
- the date the page itself happened (if you can get that from page headers)
- any dates that appear textually near a phrase or on the same page as a
phrase
- the date that a page linking to a page was created.
- any dates that appear textually near a link to a page.
For clustering, what I would do first is to try to find significant
association between dates and phrases. Since you want association due to
proximity in time as well as proximity in text, you probably should encode
the dates at several levels of resolution. My stab at that would be to use
semi-overlapping intervals that vary in size from a single day to about a
year.
The analysis I am suggesting would proceed roughly as follows:
- find interesting phrases by simple cooccurrence analysis or even just by
frequency
- find interesting cooccurrence of words and phrases with any particular
time period from any of the four date sources above.
You could probably abuse the LDA program into giving you something
interesting about topic spikes by rearranging all of your text into
"documents" that are slices of time. The topic distribution for each period
would vary when something of interest for a short period of time turns up.
My guess is that the cooccurrence based stuff would give you better results
in the short-run because it would have an easier time finding specific
examples of interest.
On Mon, Nov 16, 2009 at 11:14 AM, Ken Krugler
<kk...@transpac.com>wrote:
> So I'm going to toss out one idea...
>
> - I'd like to automatically generate a timeline of events.
>
> - I can extract dates and 2-to-4 word "terms" from web pages.
>
> - Could I use LDA to create clusters of common terms for dates?
>
> I don't want to get into named entity extraction, or anything that involves
> more than simple data extraction and then the application of a scalable
> algorithm currently supported by Mahout.
>
> Looking for feedback on the above, in terms of feasibility, level of
> effort, interest by others to help, etc.
>
> Or if somebody else has a suggestion for something simple that could
> provide interesting/obvious results, I'm all ears.
>
--
Ted Dunning, CTO
DeepDyve