You are viewing a plain text version of this content. The canonical link for it is here.

Posted to httpclient-users@hc.apache.org by Jeetendra Mirchandani <je...@amazon.com> on 2005/12/23 10:37:38 UTC

Robots.txt support?

Hi,

Can some one point me to some extension to HTTP Client to honor
robots.txt?

Please CC me on the reply as I am not on the list.

Thanks,
Jeetu


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org

AW: Robots.txt support?

Posted by Ingo Meyer <dj...@gmx.net>.

> -----Ursprüngliche Nachricht-----
> Von: Jeetendra Mirchandani [mailto:jeetu@amazon.com] 
> Gesendet: Freitag, 23. Dezember 2005 10:38
> An: httpclient-user@jakarta.apache.org
> Cc: Jeetendra Mirchandani
> Betreff: Robots.txt support?
> 
> Hi,
> 
> Can some one point me to some extension to HTTP Client to 
> honor robots.txt?

How should HttpClient do that???
This is not in scope of HttpClient.

You should load the robots.txt and parse it.

> 
> Please CC me on the reply as I am not on the list.
> 
> Thanks,
> Jeetu
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: 
> httpclient-user-help@jakarta.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org