You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by nsnyder <na...@saic.com> on 2008/05/08 16:56:42 UTC
How to skip dot files on drive crawl
I have been trying to add stuff to the regex-urlfileter.txt file to skip file
that start with a period
such as .settings or .svn when doing a local drive crawl. However I have
been unsuccessful.
Everything I try skips everything. Any suggestions??
--
View this message in context: http://www.nabble.com/How-to-skip-dot-files-on-drive-crawl-tp17127288p17127288.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: How to skip dot files on drive crawl
Posted by ivrokv <iv...@gmail.com>.
In your regex-urlfilter.txt, can you try this:
# skip URLs containing certain characters as probable queries, etc.
-[.?*!@=#%^&()+={}?;+]
Notice that I included the period "." as the first character.
nsnyder wrote:
>
> I have been trying to add stuff to the regex-urlfileter.txt file to skip
> file that start with a period
> such as .settings or .svn when doing a local drive crawl. However I have
> been unsuccessful.
> Everything I try skips everything. Any suggestions??
>
--
View this message in context: http://www.nabble.com/How-to-skip-dot-files-on-drive-crawl-tp17127288p17141336.html
Sent from the Nutch - User mailing list archive at Nabble.com.