You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Kumar Limbu <ku...@gmail.com> on 2005/12/23 03:13:41 UTC

Crawling password-protected sites

Hi Everyone,

I am trying to crawl a site which is password protected. I don't mean the
http basic authentication, I mean the same way the gmail is protected. How
to I crawl these type of password protected sites. Does nutch allow this?  I
will be very thankful if you guys can put a light on this issue.

Thank you in advance.
--
Keep on smiling
:) Kumar

Re: Crawling password-protected sites

Posted by "Pushpesh Kr. Rajwanshi" <pu...@gmail.com>.
Probably for that you will have to authenticate urself by submitting the
login form but as i guess nutch uptill 0.7.1 doesnt support that. It doesn't
even generate the form urls which is called deep web crawling. I guess your
requirement is very specific to a particular site so better option would be
to write a  custom code for that particular site using http client API.

HTH
Pushpesh


On 12/23/05, gekkokid <me...@gekkokid.org.uk> wrote:
>
> you mean you have to enter a username and password via a browser to access
> these pages? does the robots.txt file allow crawling of these pages?
> ----- Original Message -----
> From: "Kumar Limbu" <ku...@gmail.com>
> To: <nu...@lucene.apache.org>
> Sent: Friday, December 23, 2005 2:13 AM
> Subject: Crawling password-protected sites
>
>
> Hi Everyone,
>
> I am trying to crawl a site which is password protected. I don't mean the
> http basic authentication, I mean the same way the gmail is protected. How
> to I crawl these type of password protected sites. Does nutch allow
> this?  I
> will be very thankful if you guys can put a light on this issue.
>
> Thank you in advance.
> --
> Keep on smiling
> :) Kumar
>
>

Re: Crawling password-protected sites

Posted by gekkokid <me...@gekkokid.org.uk>.
you mean you have to enter a username and password via a browser to access 
these pages? does the robots.txt file allow crawling of these pages?
----- Original Message ----- 
From: "Kumar Limbu" <ku...@gmail.com>
To: <nu...@lucene.apache.org>
Sent: Friday, December 23, 2005 2:13 AM
Subject: Crawling password-protected sites


Hi Everyone,

I am trying to crawl a site which is password protected. I don't mean the
http basic authentication, I mean the same way the gmail is protected. How
to I crawl these type of password protected sites. Does nutch allow this?  I
will be very thankful if you guys can put a light on this issue.

Thank you in advance.
--
Keep on smiling
:) Kumar