You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Kay <uh...@gmail.com> on 2012/07/24 01:10:43 UTC

Crawl HTTPS websites/Enable Plugin

Hello Everyone,

I am using apache nutch to crawl HTTP websites. But when I try to crawl
HTTPS site its throwing the following error.

“failed with: org.apache.nutch.protocol.ProtocolNotFound: protocol not found
for url=https:


I searched the forums and there were some suggestions to enable the
“protocol-httpclient” plugin in nutch-site.xml.   Here is the plugin that I
use in my nutch-site .xml  file
<configuration>
<property>
  <name>plugin.includes</name>
 
<value>protocol-httpclient|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
  <description>Regular expression naming plugin directory names to
  include.  Any plugin not matching this expression is excluded.
  In any case you need at least include the nutch-extensionpoints plugin. By
  default Nutch includes crawling just HTML and plain text via HTTP,
  and basic indexing and search plugins. In order to use HTTPS please enable 
  protocol-httpclient, but be aware of possible intermittent problems with
the 
  underlying commons-httpclient library.
  </description>
</property>

</configuration>

Do you guys have any other suggestions on how to crawl https websites ?

Your help is greatly appreciated,


Thanks a lot in advance,
-Kay



--
View this message in context: http://lucene.472066.n3.nabble.com/Crawl-HTTPS-websites-Enable-Plugin-tp3996861.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: Crawl HTTPS websites/Enable Plugin

Posted by Kay <uh...@gmail.com>.

Remi,

Thanks for your reply!

Both.By default Apache Nutch will not crawl HTTPS websites.Even after using
the plugin it didnt crawl HTTPS websites. It showed the following error.

 “failed with: org.apache.nutch.protocol.ProtocolNotFound: protocol not 
found  for url=https:"

i tried the different solutions that was suggested.do u have any other
fix/suggestions?

Thanks,
-Kay



--
View this message in context: http://lucene.472066.n3.nabble.com/Crawl-HTTPS-websites-Enable-Plugin-tp3996861p3997011.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: Crawl HTTPS websites/Enable Plugin

Posted by remi tassing <ta...@gmail.com>.

So did it fail before or after you used protocol-httpclient?

On 7/24/12, Kay <uh...@gmail.com> wrote:
> Hello Everyone,
>
> I am using apache nutch to crawl HTTP websites. But when I try to crawl
> HTTPS site its throwing the following error.
>
> “failed with: org.apache.nutch.protocol.ProtocolNotFound: protocol not
> found
> for url=https:
>
>
> I searched the forums and there were some suggestions to enable the
> “protocol-httpclient” plugin in nutch-site.xml.   Here is the plugin that I
> use in my nutch-site .xml  file
> <configuration>
> <property>
>   <name>plugin.includes</name>
>
> <value>protocol-httpclient|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
>   <description>Regular expression naming plugin directory names to
>   include.  Any plugin not matching this expression is excluded.
>   In any case you need at least include the nutch-extensionpoints plugin.
> By
>   default Nutch includes crawling just HTML and plain text via HTTP,
>   and basic indexing and search plugins. In order to use HTTPS please enable
>
>   protocol-httpclient, but be aware of possible intermittent problems with
> the
>   underlying commons-httpclient library.
>   </description>
> </property>
>
> </configuration>
>
> Do you guys have any other suggestions on how to crawl https websites ?
>
> Your help is greatly appreciated,
>
>
> Thanks a lot in advance,
> -Kay
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Crawl-HTTPS-websites-Enable-Plugin-tp3996861.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>


-- 
Remi Tassing