You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Matthew Holt <mh...@redhat.com> on 2006/08/09 15:51:19 UTC
Error in 0.8 regex-urlfilter.txt
I was doing a search and noticed that a 'png' file was indexed. I
checked the crawl-urlfilter.txt and it had the following line preventing
the index of a png file:
-\.(gif|GIF|jpg|JPG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|png)$
I then looked at regex-urlfilter.txt, and the line was similar, but
lacked the 'png' definition. So apparently the recrawl was indexing the
png files. The original regex-urlfilter.txt line is below:
-\.(gif|GIF|jpg|JPG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe)$
It needs to be modified in trunk to match the line from crawl-urlfilter.txt.
Matt
Re: Error in 0.8 regex-urlfilter.txt
Posted by og...@yahoo.com.
Thanks, committed.
Otis
----- Original Message ----
From: Matthew Holt <mh...@redhat.com>
To: nutch-user@lucene.apache.org; nutch-dev@lucene.apache.org
Sent: Wednesday, August 9, 2006 9:51:19 AM
Subject: Error in 0.8 regex-urlfilter.txt
I was doing a search and noticed that a 'png' file was indexed. I
checked the crawl-urlfilter.txt and it had the following line preventing
the index of a png file:
-\.(gif|GIF|jpg|JPG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|png)$
I then looked at regex-urlfilter.txt, and the line was similar, but
lacked the 'png' definition. So apparently the recrawl was indexing the
png files. The original regex-urlfilter.txt line is below:
-\.(gif|GIF|jpg|JPG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe)$
It needs to be modified in trunk to match the line from crawl-urlfilter.txt.
Matt