You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by costas0811 <co...@hotmail.com> on 2012/02/03 05:06:59 UTC
Crawling Local Files within Cygwin
Currently trying to simply crawl a folder and index the contents (Word Docs,
HTML Files, etc..) but no matter how it is configured I get the error:
fetch of file://C:/TestFiles/ failed with:
org.apache.nutch.protocol.ProtocolNotFound: protocol not found for url=file
Here is the current set up with notes below:
-- nutch-site.xml --
<configuration>
<property>
<name>http.agent.name</name>
<value>My Nutch Spider</value>
</property>
<property>
<name>http.robots.agents</name>
<value>My Nutch Spider</value>
</property>
<property>
<name>plugin.auto-activation</name>
<value>true</value>
</property>
<property>
<name>plugin.includes</name>
<value>protocol-file|protocol-http|urlfilter-regex|parse-(text|html|js|msexcel|mspowerpoint|msword|oo|pdf|swf|zip)|index-basic|query-(basic|site|url)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
</property>
<property>
<name>file.content.limit</name>
<value>-1</value>
</property>
<property>
<name>file.crawl.parent</name>
<value>true</value>
</property>
</configuration>
I have also tried leaving out protocol-http, adding and subtracting
different extensions from the "parse-" section. I have tried any value I
could find by searching on Google as well.
I have stripped the file down to the bare minimum and still no change.
--regex-urlfilter.xml--
+.
I have basically removed anything that filters anything here. It should not
be blocking anything anymore.
-- seed.txt --
file:///C:/TestFiles/
I have also tried every configuration of /cygdrive/c, file://, leaving the
trailing / off and on, *.*, http://file://, which doesnt make sense any
ways. Relative and absolute paths both.
Now, I have looked at every post on Nabble, every blog post (a lot of this
is outdated anyways), API docs. I am even looking in the IRC channel for
help. If it were fixable by Googling the error it would have been fixed
already.
There has got to be something about the Cygwin set up th at I am missing.
It seems to Crawl and fetch websites fine.
Thanks in advance for any help.
--
View this message in context: http://lucene.472066.n3.nabble.com/Crawling-Local-Files-within-Cygwin-tp3712116p3712116.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Crawling Local Files within Cygwin
Posted by feng lu <am...@gmail.com>.
Hi afraaa
you shoud add protocol-file to the plugin.includes property in
nutch-site.xml configuration file.
like this:
<property>
<name>plugin.includes</name>
<value>protocol-(http|file)|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|indexer-solr|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
<description>Regular expression naming plugin directory names to
include. Any plugin not matching this expression is excluded.
In any case you need at least include the nutch-extensionpoints plugin. By
default Nutch includes crawling just HTML and plain text via HTTP,
and basic indexing and search plugins. In order to use HTTPS please
enable
protocol-httpclient, but be aware of possible intermittent problems with
the
underlying commons-httpclient library.
</description>
</property>
On Sun, Mar 17, 2013 at 3:27 PM, afraaa <al...@gmail.com> wrote:
>
> after set all setting with regex-urlfilter.txt and seed.txt try this
> command
> :
>
>
> ./nutch plugin protocol-file org.apache.nutch.protocol.file.File
> file:\\\e:\test.html
>
> my file is test,htm in drive E .
> ------------------------------------------------------------------
> ... good to mention that before this i was command :
>
> ./nutch plugin protocol-file org.apache.nutch.protocol.file.File
> \\\e:\test.html
>
> and got this error java.netMalformedURLException : no protocol :
> \\\e:\test.html
>
> hope this help someone .
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Crawling-Local-Files-within-Cygwin-tp3712116p4048181.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
--
Don't Grow Old, Grow Up... :-)
RE: Crawling Local Files within Cygwin
Posted by afraaa <al...@gmail.com>.
after set all setting with regex-urlfilter.txt and seed.txt try this command
:
./nutch plugin protocol-file org.apache.nutch.protocol.file.File
file:\\\e:\test.html
my file is test,htm in drive E .
------------------------------------------------------------------
... good to mention that before this i was command :
./nutch plugin protocol-file org.apache.nutch.protocol.file.File
\\\e:\test.html
and got this error java.netMalformedURLException : no protocol :
\\\e:\test.html
hope this help someone .
--
View this message in context: http://lucene.472066.n3.nabble.com/Crawling-Local-Files-within-Cygwin-tp3712116p4048181.html
Sent from the Nutch - User mailing list archive at Nabble.com.
RE: Crawling Local Files within Cygwin
Posted by costas0811 <co...@hotmail.com>.
No, but I did copy from 1.3 to the 1.4 folder and started carefully porting over the changes one by one. I was never able to find out exactly what I missed but that's what I had to do.
Date: Tue, 21 Feb 2012 12:43:57 -0800
From: ml-node+s472066n3764766h34@n3.nabble.com
To: costas0811@hotmail.com
Subject: RE: Crawling Local Files within Cygwin
I am having the same issue with 1.4! It was working fine in 1.3 and 1.2. Any ideas what specific config changes that made the difference?
If you reply to this email, your message will be added to the discussion below:
http://lucene.472066.n3.nabble.com/Crawling-Local-Files-within-Cygwin-tp3712116p3764766.html
To unsubscribe from Crawling Local Files within Cygwin, click here.
NAML
--
View this message in context: http://lucene.472066.n3.nabble.com/Crawling-Local-Files-within-Cygwin-tp3712116p3764770.html
Sent from the Nutch - User mailing list archive at Nabble.com.
RE: Crawling Local Files within Cygwin
Posted by webdev1977 <we...@gmail.com>.
I am having the same issue with 1.4! It was working fine in 1.3 and 1.2.
Any ideas what specific config changes that made the difference?
--
View this message in context: http://lucene.472066.n3.nabble.com/Crawling-Local-Files-within-Cygwin-tp3712116p3764766.html
Sent from the Nutch - User mailing list archive at Nabble.com.
RE: Crawling Local Files within Cygwin
Posted by costas0811 <co...@hotmail.com>.
What I ended up doing was downloading 1.2 (I am using 1.4) tried that, it got further with a more specific error. So I copied the 1.2 config files to 1.4 conf dir. Then I started making my tweaks from there. Eventually it worked properly.
Thanks all for the help and I will write up a complete guide when I can.
Date: Tue, 7 Feb 2012 04:20:40 -0800
From: ml-node+s472066n3722375h13@n3.nabble.com
To: costas0811@hotmail.com
Subject: RE: Crawling Local Files within Cygwin
I have done this using a similar setup and you have to play around with the number of slashes that you use after the "file:" I don't remember off the top of my head the correct number, but try up to six. (Crazy I know!)
If you reply to this email, your message will be added to the discussion below:
http://lucene.472066.n3.nabble.com/Crawling-Local-Files-within-Cygwin-tp3712116p3722375.html
To unsubscribe from Crawling Local Files within Cygwin, click here.
NAML
--
View this message in context: http://lucene.472066.n3.nabble.com/Crawling-Local-Files-within-Cygwin-tp3712116p3722788.html
Sent from the Nutch - User mailing list archive at Nabble.com.
RE: Crawling Local Files within Cygwin
Posted by webdev1977 <we...@gmail.com>.
I have done this using a similar setup and you have to play around with the
number of slashes that you use after the "file:" I don't remember off the
top of my head the correct number, but try up to six. (Crazy I know!)
--
View this message in context: http://lucene.472066.n3.nabble.com/Crawling-Local-Files-within-Cygwin-tp3712116p3722375.html
Sent from the Nutch - User mailing list archive at Nabble.com.
RE: Crawling Local Files within Cygwin
Posted by costas0811 <co...@hotmail.com>.
Just for fun I tried it. No change.
Date: Fri, 3 Feb 2012 01:00:30 -0800
From: ml-node+s472066n3712488h16@n3.nabble.com
To: costas0811@hotmail.com
Subject: Re: Crawling Local Files within Cygwin
Hi!
I don't know if this can help you (I've only crawled web sites), but have
you tried setting "file.content.ignored" to false?
<property>
<name>file.content.ignored</name>
<value>false</value>
<description>If true, no file content will be saved during fetch.
And it is probably what we want to set most of time, since file:// URLs
are meant to be local and we can always use them directly at parsing
and indexing stages. Otherwise file contents will be saved.
!! NO IMPLEMENTED YET !!
</description>
</property>
However I don't know exactly what "NO IMPLEMENTED YET" means. I think nutch
is able to crawl file content...
2012/2/3 costas0811 <[hidden email]>
> Currently trying to simply crawl a folder and index the contents (Word
> Docs,
> HTML Files, etc..) but no matter how it is configured I get the error:
>
> fetch of file://C:/TestFiles/ failed with:
> org.apache.nutch.protocol.ProtocolNotFound: protocol not found for url=file
>
> Here is the current set up with notes below:
>
> -- nutch-site.xml --
> <configuration>
> <property>
> <name>http.agent.name</name>
> <value>My Nutch Spider</value>
> </property>
>
> <property>
> <name>http.robots.agents</name>
> <value>My Nutch Spider</value>
> </property>
>
>
> <property>
> <name>plugin.auto-activation</name>
> <value>true</value>
> </property>
>
> <property>
> <name>plugin.includes</name>
>
>
> <value>protocol-file|protocol-http|urlfilter-regex|parse-(text|html|js|msexcel|mspowerpoint|msword|oo|pdf|swf|zip)|index-basic|query-(basic|site|url)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
> </property>
>
>
> <property>
> <name>file.content.limit</name>
> <value>-1</value>
> </property>
>
> <property>
> <name>file.crawl.parent</name>
> <value>true</value>
> </property>
>
> </configuration>
>
> I have also tried leaving out protocol-http, adding and subtracting
> different extensions from the "parse-" section. I have tried any value I
> could find by searching on Google as well.
>
> I have stripped the file down to the bare minimum and still no change.
>
> --regex-urlfilter.xml--
> +.
>
> I have basically removed anything that filters anything here. It should
> not
> be blocking anything anymore.
>
>
>
> -- seed.txt --
> file:///C:/TestFiles/
>
> I have also tried every configuration of /cygdrive/c, file://, leaving the
> trailing / off and on, *.*, http://file://, which doesnt make sense any
> ways. Relative and absolute paths both.
>
>
> Now, I have looked at every post on Nabble, every blog post (a lot of this
> is outdated anyways), API docs. I am even looking in the IRC channel for
> help. If it were fixable by Googling the error it would have been fixed
> already.
>
> There has got to be something about the Cygwin set up th at I am missing.
> It seems to Crawl and fetch websites fine.
>
> Thanks in advance for any help.
>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Crawling-Local-Files-within-Cygwin-tp3712116p3712116.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
If you reply to this email, your message will be added to the discussion below:
http://lucene.472066.n3.nabble.com/Crawling-Local-Files-within-Cygwin-tp3712116p3712488.html
To unsubscribe from Crawling Local Files within Cygwin, click here.
NAML
--
View this message in context: http://lucene.472066.n3.nabble.com/Crawling-Local-Files-within-Cygwin-tp3712116p3713096.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Crawling Local Files within Cygwin
Posted by Adriana Farina <ad...@gmail.com>.
Hi!
I don't know if this can help you (I've only crawled web sites), but have
you tried setting "file.content.ignored" to false?
<property>
<name>file.content.ignored</name>
<value>false</value>
<description>If true, no file content will be saved during fetch.
And it is probably what we want to set most of time, since file:// URLs
are meant to be local and we can always use them directly at parsing
and indexing stages. Otherwise file contents will be saved.
!! NO IMPLEMENTED YET !!
</description>
</property>
However I don't know exactly what "NO IMPLEMENTED YET" means. I think nutch
is able to crawl file content...
2012/2/3 costas0811 <co...@hotmail.com>
> Currently trying to simply crawl a folder and index the contents (Word
> Docs,
> HTML Files, etc..) but no matter how it is configured I get the error:
>
> fetch of file://C:/TestFiles/ failed with:
> org.apache.nutch.protocol.ProtocolNotFound: protocol not found for url=file
>
> Here is the current set up with notes below:
>
> -- nutch-site.xml --
> <configuration>
> <property>
> <name>http.agent.name</name>
> <value>My Nutch Spider</value>
> </property>
>
> <property>
> <name>http.robots.agents</name>
> <value>My Nutch Spider</value>
> </property>
>
>
> <property>
> <name>plugin.auto-activation</name>
> <value>true</value>
> </property>
>
> <property>
> <name>plugin.includes</name>
>
>
> <value>protocol-file|protocol-http|urlfilter-regex|parse-(text|html|js|msexcel|mspowerpoint|msword|oo|pdf|swf|zip)|index-basic|query-(basic|site|url)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
> </property>
>
>
> <property>
> <name>file.content.limit</name>
> <value>-1</value>
> </property>
>
> <property>
> <name>file.crawl.parent</name>
> <value>true</value>
> </property>
>
> </configuration>
>
> I have also tried leaving out protocol-http, adding and subtracting
> different extensions from the "parse-" section. I have tried any value I
> could find by searching on Google as well.
>
> I have stripped the file down to the bare minimum and still no change.
>
> --regex-urlfilter.xml--
> +.
>
> I have basically removed anything that filters anything here. It should
> not
> be blocking anything anymore.
>
>
>
> -- seed.txt --
> file:///C:/TestFiles/
>
> I have also tried every configuration of /cygdrive/c, file://, leaving the
> trailing / off and on, *.*, http://file://, which doesnt make sense any
> ways. Relative and absolute paths both.
>
>
> Now, I have looked at every post on Nabble, every blog post (a lot of this
> is outdated anyways), API docs. I am even looking in the IRC channel for
> help. If it were fixable by Googling the error it would have been fixed
> already.
>
> There has got to be something about the Cygwin set up th at I am missing.
> It seems to Crawl and fetch websites fine.
>
> Thanks in advance for any help.
>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Crawling-Local-Files-within-Cygwin-tp3712116p3712116.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>