You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Find Me <fi...@gmail.com> on 2006/10/17 17:54:07 UTC

near duplicates

How to eliminate near duplicates from the index? Someone suggested that I
could look at the TermVectors and do a comparision to remove the duplicates.
One major problem with this is the structure of the document is no longer
important. Are there any obvious pitfalls? For example: Document A being a
subset of Document B but in no particular order.

Nutch's DeleteDuplicates class is useful only when the documents are
identical with respect to either URL or the content.

Re: near duplicates

Posted by Andrzej Bialecki <ab...@getopt.org>.
NG-Marketing, M.Schneider wrote:
>> DeleteDuplicates removes documents having the same digest or the same
>> url. If you use the TextProfileSigniture instead of MD5Signiture, it
>> will remove near similar documents. The MD5Signiture class set digest as
>> the md5 of all the content, whereas textProfileSigniture sets digest as
>> the md5 of significant terms. You should check the class for
>> implementation details.  also look at signitureFactory for how to change
>> the configuration.
>>     
>
> DeleteDuplicates does NOT delete same URLs, it compares only the digest. See
> Nutch 371
> http://www.mail-archive.com/nutch-dev@lucene.apache.org/msg04635.html
>   

Erhm, please see the followup here: 
http://issues.apache.org/jira/browse/NUTCH-371 . This issue is fixed now.

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com



RE: near duplicates

Posted by "NG-Marketing, M.Schneider" <sc...@ng-marketing.com>.
> 
> DeleteDuplicates removes documents having the same digest or the same
> url. If you use the TextProfileSigniture instead of MD5Signiture, it
> will remove near similar documents. The MD5Signiture class set digest as
> the md5 of all the content, whereas textProfileSigniture sets digest as
> the md5 of significant terms. You should check the class for
> implementation details.  also look at signitureFactory for how to change
> the configuration.

DeleteDuplicates does NOT delete same URLs, it compares only the digest. See
Nutch 371
http://www.mail-archive.com/nutch-dev@lucene.apache.org/msg04635.html

In fact I have some important URLs in every single segment (although this
should not happen because I generate with the topN option. Maybe topN
doesn't look in the crawldb or so.)


Re: near duplicates

Posted by Enis Soztutar <en...@gmail.com>.
John Casey wrote:
> On 10/18/06, Isabel Drost <id...@htwm.de> wrote:
>>
>> Find Me wrote:
>> > How to eliminate near duplicates from the index? Someone suggested 
>> that
>> I
>> > could look at the TermVectors and do a comparision to remove the
>> > duplicates.
>>
>> As an alternative you could also have a look at the paper "Detecting
>> Phrase-Level Duplication on the World Wide Web" by Dennis Fetterly, Mark
>> Manasse, Marc Najork.
>
>
> Another good reference would be Soumen Chakrabarti's reference book, 
> "Mining
> the Web - Discovering Knowledge from Hypertext Data",2003 and the 
> section on
> shingling and the elimination of near duplicates. Of course I think this
> works at the document level rather than at the term vector level but it
> might be useful to prevent duplicate documents from being indexed
> altogether.
>
>> One major problem with this is the structure of the document is
>> > no longer important. Are there any obvious pitfalls? For example:
>> Document
>> > A being a subset of Document B but in no particular order.
>>
>> I think this case is pretty unlikely. But I am not sure whether you can
>> detect
>> near duplicates by only comparing term-document vectors. There might be
>> problems with documents with slightly changed words, words that were
>> replaced
>> with synonyms...
>>
>> However, if you want to keep some information on the word order, you 
>> might
>> consider comparing n-gram document vectors. That is, each dimension 
>> in the
>>
>> vector does not only represent one word but a sequence of 2, 3, 4, 5...
>> words.
>
>
>
> would this involve something like a window of 2-5 words around a 
> particular
> term in a document?
>
> Cheers,
>> Isabel
>>
>

DeleteDuplicates removes documents having the same digest or the same 
url. If you use the TextProfileSigniture instead of MD5Signiture, it 
will remove near similar documents. The MD5Signiture class set digest as 
the md5 of all the content, whereas textProfileSigniture sets digest as 
the md5 of significant terms. You should check the class for 
implementation details.  also look at signitureFactory for how to change 
the configuration.


Re: near duplicates

Posted by John Casey <jo...@gmail.com>.
On 10/18/06, Isabel Drost <id...@htwm.de> wrote:
>
> Find Me wrote:
> > How to eliminate near duplicates from the index? Someone suggested that
> I
> > could look at the TermVectors and do a comparision to remove the
> > duplicates.
>
> As an alternative you could also have a look at the paper "Detecting
> Phrase-Level Duplication on the World Wide Web" by Dennis Fetterly, Mark
> Manasse, Marc Najork.


Another good reference would be Soumen Chakrabarti's reference book, "Mining
the Web - Discovering Knowledge from Hypertext Data",2003 and the section on
shingling and the elimination of near duplicates. Of course I think this
works at the document level rather than at the term vector level but it
might be useful to prevent duplicate documents from being indexed
altogether.

> One major problem with this is the structure of the document is
> > no longer important. Are there any obvious pitfalls? For example:
> Document
> > A being a subset of Document B but in no particular order.
>
> I think this case is pretty unlikely. But I am not sure whether you can
> detect
> near duplicates by only comparing term-document vectors. There might be
> problems with documents with slightly changed words, words that were
> replaced
> with synonyms...
>
> However, if you want to keep some information on the word order, you might
> consider comparing n-gram document vectors. That is, each dimension in the
>
> vector does not only represent one word but a sequence of 2, 3, 4, 5...
> words.



would this involve something like a window of 2-5 words around a particular
term in a document?

Cheers,
> Isabel
>

Re: near duplicates

Posted by John Casey <jo...@gmail.com>.
On 10/18/06, Isabel Drost <id...@htwm.de> wrote:
>
> Find Me wrote:
> > How to eliminate near duplicates from the index? Someone suggested that
> I
> > could look at the TermVectors and do a comparision to remove the
> > duplicates.
>
> As an alternative you could also have a look at the paper "Detecting
> Phrase-Level Duplication on the World Wide Web" by Dennis Fetterly, Mark
> Manasse, Marc Najork.


Another good reference would be Soumen Chakrabarti's reference book, "Mining
the Web - Discovering Knowledge from Hypertext Data",2003 and the section on
shingling and the elimination of near duplicates. Of course I think this
works at the document level rather than at the term vector level but it
might be useful to prevent duplicate documents from being indexed
altogether.

> One major problem with this is the structure of the document is
> > no longer important. Are there any obvious pitfalls? For example:
> Document
> > A being a subset of Document B but in no particular order.
>
> I think this case is pretty unlikely. But I am not sure whether you can
> detect
> near duplicates by only comparing term-document vectors. There might be
> problems with documents with slightly changed words, words that were
> replaced
> with synonyms...
>
> However, if you want to keep some information on the word order, you might
> consider comparing n-gram document vectors. That is, each dimension in the
>
> vector does not only represent one word but a sequence of 2, 3, 4, 5...
> words.



would this involve something like a window of 2-5 words around a particular
term in a document?

Cheers,
> Isabel
>

Re: near duplicates

Posted by Isabel Drost <id...@HTWM.De>.
Find Me wrote:
> How to eliminate near duplicates from the index? Someone suggested that I
> could look at the TermVectors and do a comparision to remove the
> duplicates.

As an alternative you could also have a look at the paper "Detecting 
Phrase-Level Duplication on the World Wide Web" by Dennis Fetterly, Mark 
Manasse, Marc Najork. 


> One major problem with this is the structure of the document is 
> no longer important. Are there any obvious pitfalls? For example: Document
> A being a subset of Document B but in no particular order.

I think this case is pretty unlikely. But I am not sure whether you can detect 
near duplicates by only comparing term-document vectors. There might be 
problems with documents with slightly changed words, words that were replaced 
with synonyms...

However, if you want to keep some information on the word order, you might 
consider comparing n-gram document vectors. That is, each dimension in the 
vector does not only represent one word but a sequence of 2, 3, 4, 5... 
words.

Cheers,
Isabel

-- 
QOTD: Knucklehead: "Knock, knock" Pee Wee: "Who's there?" Knucklehead: "Little 
ol' lady." Pee Wee: "Liddle ol' lady who?" Knucklehead: "I didn't know you 
could yodel" 
  |\      _,,,---,,_       Web:   <http://www.isabel-drost.de>
  /,`.-'`'    -.  ;-;;,_   VoIP:    <si...@sipgate.de>
 |,4-  ) )-,_..;\ (  `'-'  Jabber: <xm...@jabber.org>
'---''(_/--'  `-'\_) (fL)  Kein ToFu:  <http://learn.to/quote>

Re: near duplicates

Posted by karl wettin <ka...@gmail.com>.
17 okt 2006 kl. 18.55 skrev Andrzej Bialecki:
> You need to create a fuzzy signature of the document, based on term  
> histogram or shingles - take a look a the Signature framework in  
> Nutch.
>
> There is a substantial literature on this subject - go to Citeseer  
> and run a search for "near duplicate detection".

Interesting. I'll have to check this out a bit more some day(tm).

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: near duplicates

Posted by Andrzej Bialecki <ab...@getopt.org>.
Beto Siless wrote:
> Hi Andrej!
>
> I'm taking a look to fuzzy signatures for near duplicate detection and 
> and I have seen your TextProfileSignature. The question is: If I index 
> the documents with their text signature, is there a way to filter near 
> duplicates at search time without comparing each document with all other?


Well, Nutch uses DeleteDuplicates tool for this, which basically sorts 
and compares these signatures and removes all duplicates from the index. 
Then, when searching, there are simply no duplicates anymore because 
they have been already deleted.

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: near duplicates

Posted by Find Me <fi...@gmail.com>.
It doesn't make sense to eliminate near duplicates during search time. But
if you are trying to cluster duplicates together then probably you want to
look at Carrot.

On 10/24/06, Beto Siless <be...@tera-code.com.ar> wrote:
>
> Hi Andrej!
>
> I'm taking a look to fuzzy signatures for near duplicate detection and
> and I have seen your TextProfileSignature. The question is: If I index
> the documents with their text signature, is there a way to filter near
> duplicates at search time without comparing each document with all other?
>
> Thanks
> Beto
>
> Andrzej Bialecki wrote:
> > karl wettin wrote:
> >>
> >> 17 okt 2006 kl. 17.54 skrev Find Me:
> >>
> >>> How to eliminate near duplicates from the index?
> >>
> >> I would probably try to measure the Ecludian distance between all
> >> documents, computed on terms and their positions. Or perhaps use
> >> standard deviation to find the distribution of terms in a document.
> >> One would based on the output from that try to find a threashold.
> >> Either way it will consume lots of CPU.
> >
> >
> > There are better ways to achieve this. You need to create a fuzzy
> > signature of the document, based on term histogram or shingles - take a
> > look a the Signature framework in Nutch.
> >
> > There is a substantial literature on this subject - go to Citeseer and
> > run a search for "near duplicate detection".
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: near duplicates

Posted by Beto Siless <be...@tera-code.com.ar>.
Hi Andrej!

I'm taking a look to fuzzy signatures for near duplicate detection and 
and I have seen your TextProfileSignature. The question is: If I index 
the documents with their text signature, is there a way to filter near 
duplicates at search time without comparing each document with all other?

Thanks
Beto

Andrzej Bialecki wrote:
> karl wettin wrote:
>>
>> 17 okt 2006 kl. 17.54 skrev Find Me:
>>
>>> How to eliminate near duplicates from the index?
>>
>> I would probably try to measure the Ecludian distance between all 
>> documents, computed on terms and their positions. Or perhaps use 
>> standard deviation to find the distribution of terms in a document. 
>> One would based on the output from that try to find a threashold. 
>> Either way it will consume lots of CPU.
> 
> 
> There are better ways to achieve this. You need to create a fuzzy 
> signature of the document, based on term histogram or shingles - take a 
> look a the Signature framework in Nutch.
> 
> There is a substantial literature on this subject - go to Citeseer and 
> run a search for "near duplicate detection".
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: near duplicates

Posted by Andrzej Bialecki <ab...@getopt.org>.
karl wettin wrote:
>
> 17 okt 2006 kl. 17.54 skrev Find Me:
>
>> How to eliminate near duplicates from the index?
>
> I would probably try to measure the Ecludian distance between all 
> documents, computed on terms and their positions. Or perhaps use 
> standard deviation to find the distribution of terms in a document. 
> One would based on the output from that try to find a threashold. 
> Either way it will consume lots of CPU.


There are better ways to achieve this. You need to create a fuzzy 
signature of the document, based on term histogram or shingles - take a 
look a the Signature framework in Nutch.

There is a substantial literature on this subject - go to Citeseer and 
run a search for "near duplicate detection".

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: near duplicates

Posted by karl wettin <ka...@gmail.com>.
17 okt 2006 kl. 17.54 skrev Find Me:

> How to eliminate near duplicates from the index?

I would probably try to measure the Ecludian distance between all  
documents, computed on terms and their positions. Or perhaps use  
standard deviation to find the distribution of terms in a document.  
One would based on the output from that try to find a threashold.  
Either way it will consume lots of CPU.



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: near duplicates

Posted by Beto Siless <be...@tera-code.com.ar>.
Hi Karl!
I'm interested in near duplicate detection based on termFreqVectos. Now 
I'm comparing all documents with each other (calculating the angle)... 
Is there a way to avoid that?

Thanks!
Beto

karl wettin wrote:
> 
> 17 okt 2006 kl. 17.54 skrev Find Me:
> 
>> How to eliminate near duplicates from the index?
> 
> Oh, one more thing. You should probably look at the norms in order to 
> avoid comparing all documents to each other.
> 
> 
> 
> ------------------------------------------------------------------------
> 
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.1.408 / Virus Database: 268.13.4/477 - Release Date: 10/16/2006

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: near duplicates

Posted by karl wettin <ka...@gmail.com>.
17 okt 2006 kl. 17.54 skrev Find Me:

> How to eliminate near duplicates from the index?

Oh, one more thing. You should probably look at the norms in order to  
avoid comparing all documents to each other.


crystal

Posted by TCXO <fy...@worldzgc.com>.
Guangzhou zgc electronic co.ltd
ZGC Electroric co.,ltd. is an incustrial  entity that is  involved in
developing and manufacturing quartz crystal resonator of various models and
specifications.
tel:86-20-32068136
fax:86-20-32068201
contact:manager guo
http://www.worldzgc.com http://www.worldzgc.com 
-- 
View this message in context: http://www.nabble.com/near-duplicates-tf2460501.html#a10239874
Sent from the Nutch - User mailing list archive at Nabble.com.


Re: Fetching outside the domain ?

Posted by Andrzej Bialecki <ab...@getopt.org>.
Tomi N/A wrote:
> 2007/4/19, qi wu <ch...@gmail.com>:
>> I am using Fetcher2, pls. refer the code fragment below:
>>
>> if (newUrl != null && !newUrl.equals(fit.url.toString())) {
>> ...
> 
> I'd have to take a look at the surrounding code to really understand
> what the code does. I'm sure it's perfectly obvious to the authors and
> they can't understand what's not to understand, though. :)

:) That's a good one ... In plain English: the page is trying to 
redirect us to a new url. If the new url is present, and passed the 
filters/normalizers, and it's different from the current url, consider 
fetching it.

And I agree that the current code is ignorant of the setting of 
db.ignore.external.links. The question however is whether we should 
pollute this already complex code with additional checks for every 
possible option ... This smells like a code that requires refactoring, 
IMHO the decisions about such things should be fatored out and 
encapsulated in a utility class.

This is more work than just adding a single line check, which may 
suggest why it hasn't been done yet. Patches are welcome ;)


-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com


Re: Fetching outside the domain ?

Posted by Tomi N/A <he...@gmail.com>.
2007/4/19, qi wu <ch...@gmail.com>:
> I am using Fetcher2, pls. refer the code fragment below:
>
> if (newUrl != null && !newUrl.equals(fit.url.toString())) {
> ...

I'd have to take a look at the surrounding code to really understand
what the code does. I'm sure it's perfectly obvious to the authors and
they can't understand what's not to understand, though. :)

>           if (LOG.isDebugEnabled()) {
>            LOG.debug(" - protocol redirect to "+ redirUrl + " (fetching now)");
>      }

This might be improved like so:

           if (LOG.isDebugEnabled()) {
            LOG.debug(" - protocol redirect to
".concat(redirUrl).concat(" (fetching now)"));
      }


> add statement  to check whether redirUrl and fit.url are from the same host.I'll provide a patch through JIRA soon,but it need some time for me to learn how to provide patch.
> If you are too lazy to change code, just set "http.max.redirect" in your nutch site configure to 0 also should work.

No, not lazy, but sorely short on time: http.max.redirect is a great
hint, thanks.

Thank you for the help and the patch. Cheers,
t.n.a.

Re: Fetching outside the domain ?

Posted by qi wu <ch...@gmail.com>.
I am using Fetcher2, pls. refer the code fragment below:

if (newUrl != null && !newUrl.equals(fit.url.toString())) {
     UTF8 redirUrl = new UTF8(newUrl);
     if (maxRedirect > 0) {
          redirecting = true;
          redirectCount++;
          fit = FetchItem.create(redirUrl,new CrawlDatum(), byIP);
          FetchItemQueue fiq = fetchQueues.getFetchItemQueue(fit.queueID);
          fiq.addInProgressFetchItem(fit);
          if (LOG.isDebugEnabled()) {
           LOG.debug(" - protocol redirect to "+ redirUrl + " (fetching now)");
     }

add statement  to check whether redirUrl and fit.url are from the same host.I'll provide a patch through JIRA soon,but it need some time for me to learn how to provide patch.
If you are too lazy to change code, just set "http.max.redirect" in your nutch site configure to 0 also should work.


----- Original Message ----- 
From: "Tomi N/A" <he...@gmail.com>
To: <nu...@lucene.apache.org>
Sent: Thursday, April 19, 2007 10:07 PM
Subject: Re: Fetching outside the domain ?


> 2007/4/19, qi wu <ch...@gmail.com>:
>> I find there is a bug in Fetcher,which  cause the problem you reported...
>> Now,Nutch only take external link check during the parsing process,which can make sure all the outlinks generated are in the same host  as the from-URL.But for the links which  will be redirected during fetch,this is not enough.we also need to make sure the redirected url is are in the same host with in the source URL.
>> Just take the link below as an example:
>> http://www.nxtravel.net/?feed=AS&template=Lander_Hybrid&rank=4&keyword=Loans&d=unsecured-direct-loan.com&rid=http%3A%2F%2Fwww.google.com%2Furl%3Fsa%3DL%26ai%3DBLo7nXConRq6MG5_IhQS6xtEClJquHNzjjKMGrOuW0wTAuAIQBBgEIInKzAcoBzABOAFQ0PfZ2vj_____AWCdudCBkAWYAeeHAZgBhogBqgEFMDI1MTSyAQxueHRyYXZlbC5uZXTIAQHaAQxueHRyYXZlbC5uZXTIApS06QHZAzr5xMjNnhl44AMC%26num%3D4%26q%3Dhttp%3A%2F%2Funsecured-direct-loan.com%2Funsecured-loans-online.html%26usg%3DAFrqEzct1VSZnZ48RrXOwHNyxS8qzm9O_w
>> it will be redirected to
>> http://unsecured-direct-loan.com/unsecured-loans-online.html
> 
> Nice to know I haven't lost it completely: finally someone else
> acknowledged the problem exists. :)
> Could you please clarify what you ment by "So just add external link
> check for moved and temp_moved urls should fix this problem"?
> 
> TIA,
> t.n.a.
>

Re: Fetching outside the domain ?

Posted by Tomi N/A <he...@gmail.com>.
2007/4/19, qi wu <ch...@gmail.com>:
> I find there is a bug in Fetcher,which  cause the problem you reported...
> Now,Nutch only take external link check during the parsing process,which can make sure all the outlinks generated are in the same host  as the from-URL.But for the links which  will be redirected during fetch,this is not enough.we also need to make sure the redirected url is are in the same host with in the source URL.
> Just take the link below as an example:
> http://www.nxtravel.net/?feed=AS&template=Lander_Hybrid&rank=4&keyword=Loans&d=unsecured-direct-loan.com&rid=http%3A%2F%2Fwww.google.com%2Furl%3Fsa%3DL%26ai%3DBLo7nXConRq6MG5_IhQS6xtEClJquHNzjjKMGrOuW0wTAuAIQBBgEIInKzAcoBzABOAFQ0PfZ2vj_____AWCdudCBkAWYAeeHAZgBhogBqgEFMDI1MTSyAQxueHRyYXZlbC5uZXTIAQHaAQxueHRyYXZlbC5uZXTIApS06QHZAzr5xMjNnhl44AMC%26num%3D4%26q%3Dhttp%3A%2F%2Funsecured-direct-loan.com%2Funsecured-loans-online.html%26usg%3DAFrqEzct1VSZnZ48RrXOwHNyxS8qzm9O_w
> it will be redirected to
> http://unsecured-direct-loan.com/unsecured-loans-online.html

Nice to know I haven't lost it completely: finally someone else
acknowledged the problem exists. :)
Could you please clarify what you ment by "So just add external link
check for moved and temp_moved urls should fix this problem"?

TIA,
t.n.a.

Re: Fetching outside the domain ?

Posted by qi wu <ch...@gmail.com>.
I find there is a bug in Fetcher,which  cause the problem you reported...
Now,Nutch only take external link check during the parsing process,which can make sure all the outlinks generated are in the same host  as the from-URL.But for the links which  will be redirected during fetch,this is not enough.we also need to make sure the redirected url is are in the same host with in the source URL.
Just take the link below as an example: 
http://www.nxtravel.net/?feed=AS&template=Lander_Hybrid&rank=4&keyword=Loans&d=unsecured-direct-loan.com&rid=http%3A%2F%2Fwww.google.com%2Furl%3Fsa%3DL%26ai%3DBLo7nXConRq6MG5_IhQS6xtEClJquHNzjjKMGrOuW0wTAuAIQBBgEIInKzAcoBzABOAFQ0PfZ2vj_____AWCdudCBkAWYAeeHAZgBhogBqgEFMDI1MTSyAQxueHRyYXZlbC5uZXTIAQHaAQxueHRyYXZlbC5uZXTIApS06QHZAzr5xMjNnhl44AMC%26num%3D4%26q%3Dhttp%3A%2F%2Funsecured-direct-loan.com%2Funsecured-loans-online.html%26usg%3DAFrqEzct1VSZnZ48RrXOwHNyxS8qzm9O_w 
it will be redirected to 
http://unsecured-direct-loan.com/unsecured-loans-online.html

and this new redirected URL, will be added  fetch queue....
So just add external link check for moved and temp_moved urls should fix this problem.

----- Original Message ----- 
From: "Tomi N/A" <he...@gmail.com>
To: <nu...@lucene.apache.org>
Sent: Wednesday, April 18, 2007 6:40 PM
Subject: Re: Fetching outside the domain ?


> Continuing the noble tradition of replying to my own messages, I have
> a small update on the topic of the crawler crawling outside of the
> given list of hosts in spite of db.ignore.external.links=true...
> 
> 2006/10/25, Tomi NA <he...@gmail.com>:
> 
>> > Could you give an example of a root URL, which leads to this symptom
>> > (i.e. leaks outside the original site)?
>>
>> I'll try to find out exactly where the crawler starts to run loose as
>> I have several web sites in my initial URL list.
> 
> I'm using nutch 0.9 now and have run into the problem again. It's a
> bit hard to reproduce as I have dozens of hosts in my initial URL list
> and the crawler leaves them days after I start the crawl: it's very
> difficult to pinpoint how or why the crawler steps outside it's
> bounds.
> 
> Did anyone else run into such a problem?
> Is there anything else I needed to do set up except
> db.ignore.external.links=true?
> 
> TIA,
> t.n.a.
>

Re: Fetching outside the domain ?

Posted by Tomi N/A <he...@gmail.com>.
Continuing the noble tradition of replying to my own messages, I have
a small update on the topic of the crawler crawling outside of the
given list of hosts in spite of db.ignore.external.links=true...

2006/10/25, Tomi NA <he...@gmail.com>:

> > Could you give an example of a root URL, which leads to this symptom
> > (i.e. leaks outside the original site)?
>
> I'll try to find out exactly where the crawler starts to run loose as
> I have several web sites in my initial URL list.

I'm using nutch 0.9 now and have run into the problem again. It's a
bit hard to reproduce as I have dozens of hosts in my initial URL list
and the crawler leaves them days after I start the crawl: it's very
difficult to pinpoint how or why the crawler steps outside it's
bounds.

Did anyone else run into such a problem?
Is there anything else I needed to do set up except
db.ignore.external.links=true?

TIA,
t.n.a.

Re: Fetching outside the domain ?

Posted by Tomi NA <he...@gmail.com>.
2006/10/23, Andrzej Bialecki <ab...@getopt.org>:
> Tomi NA wrote:
> > 2006/10/18, goudal@enseirb.fr <go...@enseirb.fr>:
> >
> >> Btw we have some virtual local hosts, hoz does the
> >> db.ignore.external.links
> >> deal with that ?
> >
> > Update:
> > setting db.ignore.external.links to true in nutch-site (and later also
> > in nutch-default as a sanity check) *doesn't work*: I feed the crawl
> > process a handfull of URLs and can only helplessly watch as the crawl
> > spreads to dozens of other sites.
>
> Could you give an example of a root URL, which leads to this symptom
> (i.e. leaks outside the original site)?

I'll try to find out exactly where the crawler starts to run loose as
I have several web sites in my initial URL list.

> > In answer to your question, it seems pointless to talk about virtual
> > host handling if the elementary filtering logic doesn't seem to
> > work... :-\
>
> Well, if this logic doesn't work it needs to be fixed, that's all.

Won't argue with you there.

t.n.a.

Re: Fetching outside the domain ?

Posted by Tomi NA <he...@gmail.com>.
2006/10/18, Frederic Goudal <go...@enseirb.fr>:
>
> Hello,
>
> I'm begining to play with nutch to index our own web site.
> I have done a first crawl and I have trid the recrawl script.
> While fetching I have lines like that :
>
> fetching http://www.yourdictionary.com/grammars.html
> fetching http://www.cours.polymtl.ca/if540/hiv_00.htm
> fetching http://www.maxim-ic.com/quick_view2.cfm/qv_pk/</font></a>
>
> but by crawl-urlfilter.txt is :
>
> # skip file:, ftp:, & mailto: urls
> -^(file|ftp|mailto):
>
> # skip image and other suffixes we can't yet parse
> -\.(gif|GIF|jpg|JPG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|
> exe|png)$
>
> # skip URLs containing certain characters as probable queries, etc.
> -[?*!@=]
>
> # skip URLs with slash-delimited segment that repeats 3+ times, to break loops
> -.*(/.+?)/.*?\1/.*?\1/
>
> # accept hosts in MY.DOMAIN.NAME
> #+^http://([a-z0-9]*\.)*enseirb.fr/
> +^http://www.enseirb.fr/
>
> # skip everything else
> -.
>
> So... I think I miss some point.

Frederic, what exactly is the problem? You'd like the recrawl not to
leave your web site? You can do that very easily: set the
"db.ignore.external.links" property in nutch-site.xml to "true" (you
can copy the xml property from nutch-default and then change the value
to "true");

> Btw as a beginner, totally ignorant of java, and timeless system ingeneer in
> charge of too many things, is there any doc that really explain the behaviour
> of nutch ?

A good place to read about nutch is the nutch wiki:
http://wiki.apache.org/nutch/

Cheers,
t.n.a.