You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Cuong Hoang <cl...@gmail.com> on 2005/07/26 10:04:13 UTC
Information extraction
Hi all,
Does anyone have experience with designing web information extraction such
as shopbots/pricebots? I'm currently doing research on this topic and want
to integrate Nutch. A few guidelines from anyone who has designed this type
of systems will really be helpful to me.
Regards,
Cuong Hoang
Re: Information extraction
Posted by Chris Lu <ch...@gmail.com>.
My approach to tackle structured information is to use DBSight, which
create Lecene indexes on retrieved data from any database.
As Erik mentioned, scraping is highly fragile. By going directly to
database, we can get more reliable/up-to-date/flexible with the data.
On the other hand, you will need database access, and this approach is
quite different from Nutch.
Or Nutch/Lucene can provide a simple XML analyzer, consuming a
specific format of XML data filtered by any plug-in XSL from any XML
structure.
--
Chris Lu
---------------------
Full-Text Search on Any Database
http://www.dbsight.net
On 7/26/05, Erik Hatcher <er...@ehatchersolutions.com> wrote:
> Further on the information extraction idea, consider what the SIMILE
> team at MIT are doing... http://simile.mit.edu
>
> The lower-case semantic web is gaining a lot of momentum these days,
> and I'm a strong proponent and student of it at the moment. Scraping
> rich information from a site is certainly reasonably pragmatic, but
> it is also highly fragile. SIMILE's Piggy Bank has a scraper
> facility. In an more ideal world, computer shops, book stores,
> libraries, and anyone with data to share would publish it in a
> reusable and structured way (RDF seems to me to be the best way to do
> this). Merging a full-text search engine with structured
> information, though, is yet another tricky thing that I am myself
> working with at the moment.
>
> I'd love to have more discussions along these lines.
>
> Erik
>
>
> On Jul 26, 2005, at 5:50 AM, Cuong Hoang wrote:
>
> > Hi Jack,
> >
> > I've been doing research the last few days and I think that once
> > successfully implemented, an information extraction system should
> > be able to
> > extract information from various sources. I've started reading
> > pattern/context free grammar/ontology which I think will be the
> > core of such
> > a system. I intend to index computer shops.
> >
> > Regards,
> >
> > Cuong Hoang
> >
> > -----Original Message-----
> > From: Jack Tang [mailto:himars@gmail.com]
> > Sent: Tuesday, 26 July 2005 6:16 PM
> > To: nutch-user@lucene.apache.org; nutch-dev@lucene.apache.org
> > Subject: Re: Information extraction
> >
> > Hi Cuong.
> >
> > I am going to build private book search engine. And I am face the same
> > problem.
> > Could you describe more about the information you want to extract and
> > the website?
> >
> > Regards
> > /Jack
> >
> > On 7/26/05, Cuong Hoang <cl...@gmail.com> wrote:
> >
> >> Hi all,
> >>
> >>
> >>
> >> Does anyone have experience with designing web information
> >> extraction such
> >> as shopbots/pricebots? I'm currently doing research on this topic
> >> and want
> >> to integrate Nutch. A few guidelines from anyone who has designed
> >> this
> >>
> > type
> >
> >> of systems will really be helpful to me.
> >>
> >>
> >>
> >> Regards,
> >>
> >>
> >>
> >> Cuong Hoang
> >>
> >>
> >>
> >>
> >
> >
> > --
> > Keep Discovering ... ...
> > http://www.jroller.com/page/jmars
> >
>
>
Re: Information extraction
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
Further on the information extraction idea, consider what the SIMILE
team at MIT are doing... http://simile.mit.edu
The lower-case semantic web is gaining a lot of momentum these days,
and I'm a strong proponent and student of it at the moment. Scraping
rich information from a site is certainly reasonably pragmatic, but
it is also highly fragile. SIMILE's Piggy Bank has a scraper
facility. In an more ideal world, computer shops, book stores,
libraries, and anyone with data to share would publish it in a
reusable and structured way (RDF seems to me to be the best way to do
this). Merging a full-text search engine with structured
information, though, is yet another tricky thing that I am myself
working with at the moment.
I'd love to have more discussions along these lines.
Erik
On Jul 26, 2005, at 5:50 AM, Cuong Hoang wrote:
> Hi Jack,
>
> I've been doing research the last few days and I think that once
> successfully implemented, an information extraction system should
> be able to
> extract information from various sources. I've started reading
> pattern/context free grammar/ontology which I think will be the
> core of such
> a system. I intend to index computer shops.
>
> Regards,
>
> Cuong Hoang
>
> -----Original Message-----
> From: Jack Tang [mailto:himars@gmail.com]
> Sent: Tuesday, 26 July 2005 6:16 PM
> To: nutch-user@lucene.apache.org; nutch-dev@lucene.apache.org
> Subject: Re: Information extraction
>
> Hi Cuong.
>
> I am going to build private book search engine. And I am face the same
> problem.
> Could you describe more about the information you want to extract and
> the website?
>
> Regards
> /Jack
>
> On 7/26/05, Cuong Hoang <cl...@gmail.com> wrote:
>
>> Hi all,
>>
>>
>>
>> Does anyone have experience with designing web information
>> extraction such
>> as shopbots/pricebots? I'm currently doing research on this topic
>> and want
>> to integrate Nutch. A few guidelines from anyone who has designed
>> this
>>
> type
>
>> of systems will really be helpful to me.
>>
>>
>>
>> Regards,
>>
>>
>>
>> Cuong Hoang
>>
>>
>>
>>
>
>
> --
> Keep Discovering ... ...
> http://www.jroller.com/page/jmars
>
RE: Information extraction
Posted by Cuong Hoang <cl...@gmail.com>.
Hi Jack,
I've been doing research the last few days and I think that once
successfully implemented, an information extraction system should be able to
extract information from various sources. I've started reading
pattern/context free grammar/ontology which I think will be the core of such
a system. I intend to index computer shops.
Regards,
Cuong Hoang
-----Original Message-----
From: Jack Tang [mailto:himars@gmail.com]
Sent: Tuesday, 26 July 2005 6:16 PM
To: nutch-user@lucene.apache.org; nutch-dev@lucene.apache.org
Subject: Re: Information extraction
Hi Cuong.
I am going to build private book search engine. And I am face the same
problem.
Could you describe more about the information you want to extract and
the website?
Regards
/Jack
On 7/26/05, Cuong Hoang <cl...@gmail.com> wrote:
> Hi all,
>
>
>
> Does anyone have experience with designing web information extraction such
> as shopbots/pricebots? I'm currently doing research on this topic and want
> to integrate Nutch. A few guidelines from anyone who has designed this
type
> of systems will really be helpful to me.
>
>
>
> Regards,
>
>
>
> Cuong Hoang
>
>
>
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
Re: Information extraction
Posted by Jack Tang <hi...@gmail.com>.
Hi Cuong.
I am going to build private book search engine. And I am face the same problem.
Could you describe more about the information you want to extract and
the website?
Regards
/Jack
On 7/26/05, Cuong Hoang <cl...@gmail.com> wrote:
> Hi all,
>
>
>
> Does anyone have experience with designing web information extraction such
> as shopbots/pricebots? I'm currently doing research on this topic and want
> to integrate Nutch. A few guidelines from anyone who has designed this type
> of systems will really be helpful to me.
>
>
>
> Regards,
>
>
>
> Cuong Hoang
>
>
>
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
Re: Information extraction
Posted by Jack Tang <hi...@gmail.com>.
Hi Cuong.
I am going to build private book search engine. And I am face the same problem.
Could you describe more about the information you want to extract and
the website?
Regards
/Jack
On 7/26/05, Cuong Hoang <cl...@gmail.com> wrote:
> Hi all,
>
>
>
> Does anyone have experience with designing web information extraction such
> as shopbots/pricebots? I'm currently doing research on this topic and want
> to integrate Nutch. A few guidelines from anyone who has designed this type
> of systems will really be helpful to me.
>
>
>
> Regards,
>
>
>
> Cuong Hoang
>
>
>
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
server option in nutch
Posted by Jay Pound <we...@poundwebhosting.com>.
what is the best pratice for using the search option in nutch, is it for
multiple machines each with a chunk of the index's and every request is
forwarded to each machine? or is it something else?
Thanks,
-Jay
RE: Information extraction
Posted by Cuong Hoang <cl...@gmail.com>.
This link is also very useful:
ftp://ftp.umiacs.umd.edu/pub/louiqa/BAA9709/PUB98/sJCS98.pdf
-----Original Message-----
From: Jack Tang [mailto:himars@gmail.com]
Sent: Wednesday, 27 July 2005 2:15 PM
To: nutch-user@lucene.apache.org; nutch-dev@incubator.apache.org
Subject: Re: Information extraction
Hi Cuong
Thanks for the demo.
I agree with that
Infomation Extraction = Segmenation + Classfication + Clustering +
Association.
I am going to extends HtmlParseFilter and do text mining on
parse.getText(). Is that a good way?
Thoughts?
And I'd like to share some resources I am reading now
http://www.ke.informatik.tu-darmstadt.de/lehre/ss05/web-mining/wm-ie.pdf
Regards
/Jack
On 7/26/05, Cuong Hoang <cl...@gmail.com> wrote:
> Jack,
>
> So far, I found two demos online:
>
> http://eso.vse.cz/~labsky/cgi-bin/client/
> http://iit.demokritos.gr/skel/crossmarc/
>
> On these websites, there are several documents that maybe useful. I don't
> think they will release the source code.
>
>
> Regards,
>
> Cuong Hoang
> -----Original Message-----
> From: Jack Tang [mailto:himars@gmail.com]
> Sent: Tuesday, 26 July 2005 8:29 PM
> To: nutch-user@lucene.apache.org
> Subject: Re: Information extraction
>
> Hi Matthias.
>
> The website is interesting but any document about the implementation
> avaiable?
>
> Cuong.
> I notice a lot paper mentioned HMM is great for information
> extraction. But I cannot find one demo in opensource way:(
> What's your thoughts?
>
>
> Regards
> /Jack
>
>
> On 7/26/05, Matthias Jaekle <ja...@eventax.de> wrote:
> > In the list of public nutch servers you find the following, which might
> > be interesting:
> > http://www.betherebesquare.com/
> >
> > Matthias
> >
>
>
> --
> Keep Discovering ... ...
> http://www.jroller.com/page/jmars
>
>
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
RE: Information extraction
Posted by Cuong Hoang <cl...@gmail.com>.
Hi Jack,
I think you will find this project somewhat useful:
http://groups.csail.mit.edu/graphics/lapis/
Regards
Cuong hoang
-----Original Message-----
From: Jack Tang [mailto:himars@gmail.com]
Sent: Wednesday, 27 July 2005 2:15 PM
To: nutch-user@lucene.apache.org; nutch-dev@incubator.apache.org
Subject: Re: Information extraction
Hi Cuong
Thanks for the demo.
I agree with that
Infomation Extraction = Segmenation + Classfication + Clustering +
Association.
I am going to extends HtmlParseFilter and do text mining on
parse.getText(). Is that a good way?
Thoughts?
And I'd like to share some resources I am reading now
http://www.ke.informatik.tu-darmstadt.de/lehre/ss05/web-mining/wm-ie.pdf
Regards
/Jack
On 7/26/05, Cuong Hoang <cl...@gmail.com> wrote:
> Jack,
>
> So far, I found two demos online:
>
> http://eso.vse.cz/~labsky/cgi-bin/client/
> http://iit.demokritos.gr/skel/crossmarc/
>
> On these websites, there are several documents that maybe useful. I don't
> think they will release the source code.
>
>
> Regards,
>
> Cuong Hoang
> -----Original Message-----
> From: Jack Tang [mailto:himars@gmail.com]
> Sent: Tuesday, 26 July 2005 8:29 PM
> To: nutch-user@lucene.apache.org
> Subject: Re: Information extraction
>
> Hi Matthias.
>
> The website is interesting but any document about the implementation
> avaiable?
>
> Cuong.
> I notice a lot paper mentioned HMM is great for information
> extraction. But I cannot find one demo in opensource way:(
> What's your thoughts?
>
>
> Regards
> /Jack
>
>
> On 7/26/05, Matthias Jaekle <ja...@eventax.de> wrote:
> > In the list of public nutch servers you find the following, which might
> > be interesting:
> > http://www.betherebesquare.com/
> >
> > Matthias
> >
>
>
> --
> Keep Discovering ... ...
> http://www.jroller.com/page/jmars
>
>
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
Re: Information extraction
Posted by Jack Tang <hi...@gmail.com>.
Hi Cuong
Thanks for the demo.
I agree with that
Infomation Extraction = Segmenation + Classfication + Clustering +
Association.
I am going to extends HtmlParseFilter and do text mining on
parse.getText(). Is that a good way?
Thoughts?
And I'd like to share some resources I am reading now
http://www.ke.informatik.tu-darmstadt.de/lehre/ss05/web-mining/wm-ie.pdf
Regards
/Jack
On 7/26/05, Cuong Hoang <cl...@gmail.com> wrote:
> Jack,
>
> So far, I found two demos online:
>
> http://eso.vse.cz/~labsky/cgi-bin/client/
> http://iit.demokritos.gr/skel/crossmarc/
>
> On these websites, there are several documents that maybe useful. I don't
> think they will release the source code.
>
>
> Regards,
>
> Cuong Hoang
> -----Original Message-----
> From: Jack Tang [mailto:himars@gmail.com]
> Sent: Tuesday, 26 July 2005 8:29 PM
> To: nutch-user@lucene.apache.org
> Subject: Re: Information extraction
>
> Hi Matthias.
>
> The website is interesting but any document about the implementation
> avaiable?
>
> Cuong.
> I notice a lot paper mentioned HMM is great for information
> extraction. But I cannot find one demo in opensource way:(
> What's your thoughts?
>
>
> Regards
> /Jack
>
>
> On 7/26/05, Matthias Jaekle <ja...@eventax.de> wrote:
> > In the list of public nutch servers you find the following, which might
> > be interesting:
> > http://www.betherebesquare.com/
> >
> > Matthias
> >
>
>
> --
> Keep Discovering ... ...
> http://www.jroller.com/page/jmars
>
>
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
Re: Information extraction
Posted by Jack Tang <hi...@gmail.com>.
Hi Cuong
Thanks for the demo.
I agree with that
Infomation Extraction = Segmenation + Classfication + Clustering +
Association.
I am going to extends HtmlParseFilter and do text mining on
parse.getText(). Is that a good way?
Thoughts?
And I'd like to share some resources I am reading now
http://www.ke.informatik.tu-darmstadt.de/lehre/ss05/web-mining/wm-ie.pdf
Regards
/Jack
On 7/26/05, Cuong Hoang <cl...@gmail.com> wrote:
> Jack,
>
> So far, I found two demos online:
>
> http://eso.vse.cz/~labsky/cgi-bin/client/
> http://iit.demokritos.gr/skel/crossmarc/
>
> On these websites, there are several documents that maybe useful. I don't
> think they will release the source code.
>
>
> Regards,
>
> Cuong Hoang
> -----Original Message-----
> From: Jack Tang [mailto:himars@gmail.com]
> Sent: Tuesday, 26 July 2005 8:29 PM
> To: nutch-user@lucene.apache.org
> Subject: Re: Information extraction
>
> Hi Matthias.
>
> The website is interesting but any document about the implementation
> avaiable?
>
> Cuong.
> I notice a lot paper mentioned HMM is great for information
> extraction. But I cannot find one demo in opensource way:(
> What's your thoughts?
>
>
> Regards
> /Jack
>
>
> On 7/26/05, Matthias Jaekle <ja...@eventax.de> wrote:
> > In the list of public nutch servers you find the following, which might
> > be interesting:
> > http://www.betherebesquare.com/
> >
> > Matthias
> >
>
>
> --
> Keep Discovering ... ...
> http://www.jroller.com/page/jmars
>
>
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
RE: Information extraction
Posted by Cuong Hoang <cl...@gmail.com>.
Thanks for very useful information Matthias. I just wrote the an email.
Regards,
Cuong Hoang
-----Original Message-----
From: Matthias Jaekle [mailto:jaekle@eventax.de]
Sent: Tuesday, 26 July 2005 8:34 PM
To: nutch-user@lucene.apache.org
Subject: Re: Information extraction
Hi,
the author of this system announced he would like to contribute some of
his modifications. Here is his post to list from 2005-06-10:
> Hello,
>
> I'd like to announce the launch of a new search engine that uses the
> Nutch engine.
> http://betherebesquare.com is an Event Search Engine for the San
> Francisco Bay Area that allows users to specify keywords, date, city,
> address, and category and get details about events in 4 different
> views.
>
> I believe this is the first search engine to parse events out of free
> text. I am very thankful to the Nutch developers for making such a
> high quality engine. I will try to contribute back some parts I've
> written that might be of general use. I'd love to hear any comments or
> suggestions you might have.
>
> Thank you,
> Tim Martin
> timmartin@alumni.cmu.edu
Maybe you could ask him.
Matthias
Re: Information extraction
Posted by Matthias Jaekle <ja...@eventax.de>.
Hi,
the author of this system announced he would like to contribute some of
his modifications. Here is his post to list from 2005-06-10:
> Hello,
>
> I'd like to announce the launch of a new search engine that uses the
> Nutch engine.
> http://betherebesquare.com is an Event Search Engine for the San
> Francisco Bay Area that allows users to specify keywords, date, city,
> address, and category and get details about events in 4 different
> views.
>
> I believe this is the first search engine to parse events out of free
> text. I am very thankful to the Nutch developers for making such a
> high quality engine. I will try to contribute back some parts I've
> written that might be of general use. I'd love to hear any comments or
> suggestions you might have.
>
> Thank you,
> Tim Martin
> timmartin@alumni.cmu.edu
Maybe you could ask him.
Matthias
RE: Information extraction
Posted by Cuong Hoang <cl...@gmail.com>.
Jack,
So far, I found two demos online:
http://eso.vse.cz/~labsky/cgi-bin/client/
http://iit.demokritos.gr/skel/crossmarc/
On these websites, there are several documents that maybe useful. I don't
think they will release the source code.
Regards,
Cuong Hoang
-----Original Message-----
From: Jack Tang [mailto:himars@gmail.com]
Sent: Tuesday, 26 July 2005 8:29 PM
To: nutch-user@lucene.apache.org
Subject: Re: Information extraction
Hi Matthias.
The website is interesting but any document about the implementation
avaiable?
Cuong.
I notice a lot paper mentioned HMM is great for information
extraction. But I cannot find one demo in opensource way:(
What's your thoughts?
Regards
/Jack
On 7/26/05, Matthias Jaekle <ja...@eventax.de> wrote:
> In the list of public nutch servers you find the following, which might
> be interesting:
> http://www.betherebesquare.com/
>
> Matthias
>
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
Re: Information extraction
Posted by Jack Tang <hi...@gmail.com>.
Hi Matthias.
The website is interesting but any document about the implementation avaiable?
Cuong.
I notice a lot paper mentioned HMM is great for information
extraction. But I cannot find one demo in opensource way:(
What's your thoughts?
Regards
/Jack
On 7/26/05, Matthias Jaekle <ja...@eventax.de> wrote:
> In the list of public nutch servers you find the following, which might
> be interesting:
> http://www.betherebesquare.com/
>
> Matthias
>
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
Re: Information extraction
Posted by Matthias Jaekle <ja...@eventax.de>.
In the list of public nutch servers you find the following, which might
be interesting:
http://www.betherebesquare.com/
Matthias