You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by d e <cr...@gmail.com> on 2007/03/10 00:48:43 UTC
Indexing the Interesting Part Only...
If I'm indexing a news article, I want to avoid getting the junk (other than
the title, auther and article) into the index. I want to avoid getting the
advertizments, etc. How do I do that sort of thing?
What parts of what manual should I be reading so I will know how to do this
sort of thing.
RE: Indexing the Interesting Part Only...
Posted by Steve Severance <st...@ivirtuoso.com>.
This is a Natural Language Processing problem, although you can certainly
take hints from URL graph structures and host block lists. Nutch does not
support this natively (that I know of) but you can certainly extend Nutch to
be able to recognize and filter ads. Start by looking at how to develop
plugins and also look at the indexing plugin.
Regards,
Steve
> -----Original Message-----
> From: d e [mailto:crawler29@gmail.com]
> Sent: Friday, March 09, 2007 6:49 PM
> To: nutch-dev@lucene.apache.org
> Subject: Indexing the Interesting Part Only...
>
> If I'm indexing a news article, I want to avoid getting the junk (other
> than
> the title, auther and article) into the index. I want to avoid getting
> the
> advertizments, etc. How do I do that sort of thing?
>
> What parts of what manual should I be reading so I will know how to do
> this
> sort of thing.
Re: Indexing the Interesting Part Only...
Posted by Michael Wechner <mi...@wyona.com>.
Michael Wechner wrote:
> d e wrote:
>
>> I'm sorry! I guess I was REALLY not clear. I mean my problem is to
>> drop the
>> junk *on each page*. I am indexing news sites. I want to harvest news
>> STORIES, not the advertisements and other junk text around the
>> outside of
>> each page. Got suggestions for THAT problem?
>
>
>
> I guess you are controlling the sites you are refering to, right? If
> so, then you might want to add something like
>
> <div id="index">... resp. <div id="no-index">...
>
> whereas I am not sure if "HTML parser" of Nutch is supporting these
> kind of tags.
it just comes to my mind that I still think the best would be something like
<html>
<head>
<link rel="search" href="search-foo.xml" type="application/search+xml"/>
whereas the search-foo.xml would contain all the data which should be
indexed by whatever search engine.
AFAIK no such standard exists (whereas one might want to consider RDF).
Cheers
Michael
>
> HTH
>
> Michael
>
>>
>> Thanks!
>>
>>
>> On 3/10/07, Björn Wilmsmann <bj...@wilmsmann.de> wrote:
>>
>>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> There are quite a few ways to do this. In fact, Google's PageRank is
>>> one such approach. Text classification (as done in spam filters, for
>>> example) is another. It just depends on what you are going to do.
>>>
>>> d e wrote:
>>>
>>> > We plan to index many websites. Got any suggestions on how to drop
>>> > the junk
>>> > without having to do too much work for each such site? Know anyone
>>> > who has a
>>> > background on doing this sort of thing? What sorts of approaches
>>> > would you
>>> > recommend?
>>>
>>> - --
>>> Best regards,
>>> Bjoern Wilmsmann
>>>
>>>
>>>
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG v1.4.7 (Darwin)
>>>
>>> iD8DBQFF812mgz0R1bg11MERAqXCAKCVTfLN7KXJYdAqLGWMI57ChKaM8QCfdQBc
>>> 1CyrQfD+5vCzSBvYbviX17o=
>>> =+TK/
>>> -----END PGP SIGNATURE-----
>>>
>>
>>
>
>
--
Michael Wechner
Wyona - Open Source Content Management - Apache Lenya
http://www.wyona.com http://lenya.apache.org
michael.wechner@wyona.com michi@apache.org
+41 44 272 91 61
Re: Indexing the Interesting Part Only...
Posted by Michael Wechner <mi...@wyona.com>.
d e wrote:
> I'm sorry! I guess I was REALLY not clear. I mean my problem is to
> drop the
> junk *on each page*. I am indexing news sites. I want to harvest news
> STORIES, not the advertisements and other junk text around the outside of
> each page. Got suggestions for THAT problem?
I guess you are controlling the sites you are refering to, right? If so,
then you might want to add something like
<div id="index">... resp. <div id="no-index">...
whereas I am not sure if "HTML parser" of Nutch is supporting these kind
of tags.
HTH
Michael
>
> Thanks!
>
>
> On 3/10/07, Björn Wilmsmann <bj...@wilmsmann.de> wrote:
>
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> There are quite a few ways to do this. In fact, Google's PageRank is
>> one such approach. Text classification (as done in spam filters, for
>> example) is another. It just depends on what you are going to do.
>>
>> d e wrote:
>>
>> > We plan to index many websites. Got any suggestions on how to drop
>> > the junk
>> > without having to do too much work for each such site? Know anyone
>> > who has a
>> > background on doing this sort of thing? What sorts of approaches
>> > would you
>> > recommend?
>>
>> - --
>> Best regards,
>> Bjoern Wilmsmann
>>
>>
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.7 (Darwin)
>>
>> iD8DBQFF812mgz0R1bg11MERAqXCAKCVTfLN7KXJYdAqLGWMI57ChKaM8QCfdQBc
>> 1CyrQfD+5vCzSBvYbviX17o=
>> =+TK/
>> -----END PGP SIGNATURE-----
>>
>
>
--
Michael Wechner
Wyona - Open Source Content Management - Apache Lenya
http://www.wyona.com http://lenya.apache.org
michael.wechner@wyona.com michi@apache.org
+41 44 272 91 61
Re: Indexing the Interesting Part Only...
Posted by Björn Wilmsmann <bj...@wilmsmann.de>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
d e wrote:
> Good thinking, Bjoern. Still, does the HTML Parser have a hook so
> it can
> break the text up into elements that will be indexed as discrete
> documents?
> This may be a dumb question but we are just getting our feet wet with
> spidering and really need some pointers!
I think splitting up documents and indexing them as separate
documents cannot be done easily. What I have in mind rather is a
filtering approach that discards stuff like teasers etc.
However, admittedly this does not work if you have more than one
complete article per document. In order to address this issue one
would have to change the way Nutch handles documents.
- --
Best regards,
Bjoern Wilmsmann
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
iD8DBQFF9CVNgz0R1bg11MERAsTPAJ9ZpMOQspUF8Ai//wXb4j/cLH4QNQCg/GFU
tvodou2/ROHF7wc1iRRAKRM=
=EeFM
-----END PGP SIGNATURE-----
Re: Indexing the Interesting Part Only...
Posted by d e <cr...@gmail.com>.
Good thinking, Bjoern. Still, does the HTML Parser have a hook so it can
break the text up into elements that will be indexed as discrete documents?
This may be a dumb question but we are just getting our feet wet with
spidering and really need some pointers!
Exactly how would the parser plug in express that it was seeing two
different articles on the same page, for example? Or do I need to do this in
a different plug in? Or is this something that can't be done from plug ins
alone?
Thanks!
On 3/11/07, Björn Wilmsmann <bj...@wilmsmann.de> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> d e wrote:
>
> > - Is it possible that the web crawling might slow to a crawl if I do
> > it in the middle of the Nutch process (or does that not matter
> > because Nutch
> > is doing stuff in multiple threads anyway so I have little to be
> > concerned
> > about)? Would it be better to havest pages and do the article
> > extraction
> > from pages once they are harvested?
>
> Text classification certainly requires more computing time than
> simply extracting and storing plain text, but given that - as you
> mentioned - Nutch runs in multi-threaded environment I would
> definitely give online text classification a try, as it is the most
> straightforward and easiest to implement way. Moreover, the lengthy
> process involved in text classification is the training part,
> classification itself usually runs pretty fast.
>
> > - Does the HTML Parser have the ability (assuming that it figures
> > out
> > how to do this) to break a page up into several elements,some of
> > of which
> > should be indexed but not indexed as one unit, indexed as several
> > different
> > units? In the end, for example, the end user in my application is
> > looking
> > for an article, say, where the term A and B appear. They are
> > *not*looking for a page where in one article A appears and another
> > article in
> > which B appears. We are really trying to index articles, *not
> > web pages*. Is that practical in Nutch? *If so, how do we do it*?
>
> This one is probably going to be really tricky. Text classification
> could be used for this, too. However, given that in this case, you
> would not decide between two discreet categories ('article-like' and
> 'not article-like), but subtle variations within the same category,
> you would probably end up with either crippled articles or many false
> positives.
> Maybe markup code could help distinguishing between genuine article
> text and mere teasers, but this depends on the news sites you are
> crawling and probably needs a lot of fine tuning for each site.
> This is actually the reason why the Semantic Web is such an awesome
> concept. If the pages contained a markup clearly telling the crawler
> that the following section is merely a teaser, your problem would
> already be solved.
>
> - --
> Best regards,
> Bjoern Wilmsmann
>
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.7 (Darwin)
>
> iD8DBQFF9Bvrgz0R1bg11MERAmzaAJ4oQPSAWYfUFqvXvb5ZlxnEWrC5zQCgwnL2
> nrMVbxv3ttPUn4mDYwACiBk=
> =sg/d
> -----END PGP SIGNATURE-----
>
Re: Indexing the Interesting Part Only...
Posted by Björn Wilmsmann <bj...@wilmsmann.de>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
d e wrote:
> - Is it possible that the web crawling might slow to a crawl if I do
> it in the middle of the Nutch process (or does that not matter
> because Nutch
> is doing stuff in multiple threads anyway so I have little to be
> concerned
> about)? Would it be better to havest pages and do the article
> extraction
> from pages once they are harvested?
Text classification certainly requires more computing time than
simply extracting and storing plain text, but given that - as you
mentioned - Nutch runs in multi-threaded environment I would
definitely give online text classification a try, as it is the most
straightforward and easiest to implement way. Moreover, the lengthy
process involved in text classification is the training part,
classification itself usually runs pretty fast.
> - Does the HTML Parser have the ability (assuming that it figures
> out
> how to do this) to break a page up into several elements,some of
> of which
> should be indexed but not indexed as one unit, indexed as several
> different
> units? In the end, for example, the end user in my application is
> looking
> for an article, say, where the term A and B appear. They are
> *not*looking for a page where in one article A appears and another
> article in
> which B appears. We are really trying to index articles, *not
> web pages*. Is that practical in Nutch? *If so, how do we do it*?
This one is probably going to be really tricky. Text classification
could be used for this, too. However, given that in this case, you
would not decide between two discreet categories ('article-like' and
'not article-like), but subtle variations within the same category,
you would probably end up with either crippled articles or many false
positives.
Maybe markup code could help distinguishing between genuine article
text and mere teasers, but this depends on the news sites you are
crawling and probably needs a lot of fine tuning for each site.
This is actually the reason why the Semantic Web is such an awesome
concept. If the pages contained a markup clearly telling the crawler
that the following section is merely a teaser, your problem would
already be solved.
- --
Best regards,
Bjoern Wilmsmann
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
iD8DBQFF9Bvrgz0R1bg11MERAmzaAJ4oQPSAWYfUFqvXvb5ZlxnEWrC5zQCgwnL2
nrMVbxv3ttPUn4mDYwACiBk=
=sg/d
-----END PGP SIGNATURE-----
Re: Indexing the Interesting Part Only...
Posted by d e <cr...@gmail.com>.
Bjorn - now THAT is a cool idea! I love it. *Very* clever. The indexed
website could change layout and my program would not care even a little bit!
My immediate questions are:
- Is it possible that the web crawling might slow to a crawl if I do
it in the middle of the Nutch process (or does that not matter because Nutch
is doing stuff in multiple threads anyway so I have little to be concerned
about)? Would it be better to havest pages and do the article extraction
from pages once they are harvested?
- If there are two articles on the same web page (or a teaser for the
second one on the page for the first) the statistics might show that they
are both article text but would it figure out that they are two different
articles, both of which should be index and indexed as separate documents?
*How do I figure out that they are two different articles?*
- Does the HTML Parser have the ability (assuming that it figures out
how to do this) to break a page up into several elements,some of of which
should be indexed but not indexed as one unit, indexed as several different
units? In the end, for example, the end user in my application is looking
for an article, say, where the term A and B appear. They are
*not*looking for a page where in one article A appears and another
article in
which B appears. We are really trying to index articles, *not
web pages*. Is that practical in Nutch? *If so, how do we do it*?
Anyway, I'm totally jazzed about your concept and very grateful for the
help!
On 3/11/07, Björn Wilmsmann <bj...@wilmsmann.de> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I think text classification could be used for this purpose. You would
> have to extract text blocks from HTML code (for example enclosed in
> <td></td> or <div></div>), then compare each block against a
> previously trained model and discard those blocks whose value is
> below a certain threshold (to start you off, see the 'techniques'
> section under
> http://en.wikipedia.org/wiki/Text_classification for instance).
> The best place to implement such a feature in Nutch probably is the
> HtmlParser class.
>
> d e wrote:
>
> > I'm sorry! I guess I was REALLY not clear. I mean my problem is to
> > drop the
> > junk *on each page*. I am indexing news sites. I want to harvest news
> > STORIES, not the advertisements and other junk text around the
> > outside of
> > each page. Got suggestions for THAT problem?
>
> - --
> Best regards,
> Bjoern Wilmsmann
>
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.7 (Darwin)
>
> iD8DBQFF8/fjgz0R1bg11MERArYfAJ40GxGX4dwJwNyu5NssELly5StCgACgzUT6
> qTIan/FUmCHd1RW0XlI4HNQ=
> =Paad
> -----END PGP SIGNATURE-----
>
Re: Indexing the Interesting Part Only...
Posted by Björn Wilmsmann <bj...@wilmsmann.de>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I think text classification could be used for this purpose. You would
have to extract text blocks from HTML code (for example enclosed in
<td></td> or <div></div>), then compare each block against a
previously trained model and discard those blocks whose value is
below a certain threshold (to start you off, see the 'techniques'
section under
http://en.wikipedia.org/wiki/Text_classification for instance).
The best place to implement such a feature in Nutch probably is the
HtmlParser class.
d e wrote:
> I'm sorry! I guess I was REALLY not clear. I mean my problem is to
> drop the
> junk *on each page*. I am indexing news sites. I want to harvest news
> STORIES, not the advertisements and other junk text around the
> outside of
> each page. Got suggestions for THAT problem?
- --
Best regards,
Bjoern Wilmsmann
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
iD8DBQFF8/fjgz0R1bg11MERArYfAJ40GxGX4dwJwNyu5NssELly5StCgACgzUT6
qTIan/FUmCHd1RW0XlI4HNQ=
=Paad
-----END PGP SIGNATURE-----
Re: Indexing the Interesting Part Only...
Posted by d e <cr...@gmail.com>.
I'm sorry! I guess I was REALLY not clear. I mean my problem is to drop the
junk *on each page*. I am indexing news sites. I want to harvest news
STORIES, not the advertisements and other junk text around the outside of
each page. Got suggestions for THAT problem?
Thanks!
On 3/10/07, Björn Wilmsmann <bj...@wilmsmann.de> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> There are quite a few ways to do this. In fact, Google's PageRank is
> one such approach. Text classification (as done in spam filters, for
> example) is another. It just depends on what you are going to do.
>
> d e wrote:
>
> > We plan to index many websites. Got any suggestions on how to drop
> > the junk
> > without having to do too much work for each such site? Know anyone
> > who has a
> > background on doing this sort of thing? What sorts of approaches
> > would you
> > recommend?
>
> - --
> Best regards,
> Bjoern Wilmsmann
>
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.7 (Darwin)
>
> iD8DBQFF812mgz0R1bg11MERAqXCAKCVTfLN7KXJYdAqLGWMI57ChKaM8QCfdQBc
> 1CyrQfD+5vCzSBvYbviX17o=
> =+TK/
> -----END PGP SIGNATURE-----
>
Re: Indexing the Interesting Part Only...
Posted by Björn Wilmsmann <bj...@wilmsmann.de>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
There are quite a few ways to do this. In fact, Google's PageRank is
one such approach. Text classification (as done in spam filters, for
example) is another. It just depends on what you are going to do.
d e wrote:
> We plan to index many websites. Got any suggestions on how to drop
> the junk
> without having to do too much work for each such site? Know anyone
> who has a
> background on doing this sort of thing? What sorts of approaches
> would you
> recommend?
- --
Best regards,
Bjoern Wilmsmann
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
iD8DBQFF812mgz0R1bg11MERAqXCAKCVTfLN7KXJYdAqLGWMI57ChKaM8QCfdQBc
1CyrQfD+5vCzSBvYbviX17o=
=+TK/
-----END PGP SIGNATURE-----
RE: Indexing the Interesting Part Only...
Posted by Steve Severance <st...@ivirtuoso.com>.
I think if anyone here had the perfect answer for that one they would have
sold it Google, Microsoft or Yahoo for a ton of money. You will need an
algorithm that can detect ads. I have not written ad filters since my search
engine is currently using a domain whitelist. I can tell you that a whole
web crawl will definetly need it since it can cut down on pages in the index
by 10-20%. If you do a whole web crawl you will also need spam detection.
I would recommend looking for some academic papers on the topic. Maybe use
CiteSeer or something like that.
Steve
-----Original Message-----
From: d e [mailto:crawler29@gmail.com]
Sent: Saturday, March 10, 2007 3:07 PM
To: nutch-dev@lucene.apache.org
Subject: Re: Indexing the Interesting Part Only...
We plan to index many websites. Got any suggestions on how to drop the junk
without having to do too much work for each such site? Know anyone who has a
background on doing this sort of thing? What sorts of approaches would you
recommend?
Are there existing plug ins I should consider using?
On 3/9/07, J. Delgado <jd...@lendingclub.com> wrote:
>
> You have to build a special HTML Junk parser.
>
> 2007/3/9, d e <cr...@gmail.com>:
> >
> > If I'm indexing a news article, I want to avoid getting the junk (other
> > than
> > the title, auther and article) into the index. I want to avoid getting
> the
> > advertizments, etc. How do I do that sort of thing?
> >
> > What parts of what manual should I be reading so I will know how to do
> > this
> > sort of thing.
> >
>
Re: Indexing the Interesting Part Only...
Posted by d e <cr...@gmail.com>.
We plan to index many websites. Got any suggestions on how to drop the junk
without having to do too much work for each such site? Know anyone who has a
background on doing this sort of thing? What sorts of approaches would you
recommend?
Are there existing plug ins I should consider using?
On 3/9/07, J. Delgado <jd...@lendingclub.com> wrote:
>
> You have to build a special HTML Junk parser.
>
> 2007/3/9, d e <cr...@gmail.com>:
> >
> > If I'm indexing a news article, I want to avoid getting the junk (other
> > than
> > the title, auther and article) into the index. I want to avoid getting
> the
> > advertizments, etc. How do I do that sort of thing?
> >
> > What parts of what manual should I be reading so I will know how to do
> > this
> > sort of thing.
> >
>
Re: Indexing the Interesting Part Only...
Posted by "J. Delgado" <jd...@lendingclub.com>.
You have to build a special HTML Junk parser.
2007/3/9, d e <cr...@gmail.com>:
>
> If I'm indexing a news article, I want to avoid getting the junk (other
> than
> the title, auther and article) into the index. I want to avoid getting the
> advertizments, etc. How do I do that sort of thing?
>
> What parts of what manual should I be reading so I will know how to do
> this
> sort of thing.
>