You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Marcin Okraszewski <ok...@o2.pl> on 2007/05/15 17:50:51 UTC
Nutch doesn't go through HTTP proxy.
I tried to run Nutch 0.9 from my network, which require HTTP proxy access. I have set up http.proxy.host and http.proxy.port properties in my nutch-site.xml. Proxy do not require authorization. Nutch takes it - I can see it in log (see below). But still I get java.net.UnknownHostException.
Interestingly, I used Wireshark (or Ethereal) to sniff if Nutch really tries to use proxy. And there is request from Nutch to proxy to get robots.txt. It says "404 Not Found". There is no fallowing request for particular page, only for robots.txt.
Any ideas what is wrong?
Marcin Okraszewski
007-05-15 17:38:59,465 INFO http.Http - http.proxy.host = <my_proxy_host>
2007-05-15 17:38:59,465 INFO http.Http - http.proxy.port = <my_proxy_port>
2007-05-15 17:38:59,465 INFO http.Http - http.timeout = 10000
2007-05-15 17:38:59,465 INFO http.Http - http.content.limit = 65536
2007-05-15 17:38:59,465 INFO http.Http - http.agent = YetAnotherSearchEngine/Nutch-0.9
2007-05-15 17:38:59,465 INFO http.Http - protocol.plugin.check.blocking = true
2007-05-15 17:38:59,465 INFO http.Http - protocol.plugin.check.robots = true
2007-05-15 17:38:59,466 INFO http.Http - fetcher.server.delay = 100
2007-05-15 17:38:59,466 INFO http.Http - http.max.delays = 100
2007-05-15 17:38:59,832 ERROR http.Http - org.apache.nutch.protocol.http.api.HttpException: java.net.UnknownHostException: <crawl_site>: <crawl_site>
2007-05-15 17:38:59,832 ERROR http.Http - at org.apache.nutch.protocol.http.api.HttpBase.blockAddr(HttpBase.java:340)
2007-05-15 17:38:59,832 ERROR http.Http - at org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:212)
2007-05-15 17:38:59,832 ERROR http.Http - at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:145)
2007-05-15 17:38:59,832 ERROR http.Http - Caused by: java.net.UnknownHostException: www.gral.pl: www.gral.pl
2007-05-15 17:38:59,832 ERROR http.Http - at java.net.InetAddress.getAllByName0(InetAddress.java:1128)
2007-05-15 17:38:59,833 ERROR http.Http - at java.net.InetAddress.getAllByName0(InetAddress.java:1098)
2007-05-15 17:38:59,833 ERROR http.Http - at java.net.InetAddress.getAllByName(InetAddress.java:1061)
2007-05-15 17:38:59,833 ERROR http.Http - at java.net.InetAddress.getByName(InetAddress.java:958)
2007-05-15 17:38:59,833 ERROR http.Http - at org.apache.nutch.protocol.http.api.HttpBase.blockAddr(HttpBase.java:336)
2007-05-15 17:38:59,833 ERROR http.Http - ... 2 more
2007-05-15 17:38:59,834 INFO fetcher.Fetcher - fetch of <crawl_site> failed with: org.apache.nutch.protocol.http.api.HttpException: java.net.UnknownHostException: <crawl_site>: <crawl_site>
Re: Nutch doesn't go through HTTP proxy.
Posted by Michael Wechner <mi...@wyona.com>.
Marcin Okraszewski wrote:
>I tried to run Nutch 0.9 from my network, which require HTTP proxy access. I have set up http.proxy.host and http.proxy.port properties in my nutch-site.xml. Proxy do not require authorization. Nutch takes it - I can see it in log (see below). But still I get java.net.UnknownHostException.
>
>Interestingly, I used Wireshark (or Ethereal) to sniff if Nutch really tries to use proxy. And there is request from Nutch to proxy to get robots.txt. It says "404 Not Found". There is no fallowing request for particular page, only for robots.txt.
>
>Any ideas what is wrong?
>
>
IIRC we had to patch Nutch in order to make it work with a proxy, but
that is Nutch 0.8 and I don't have this code available right now, but
you might want to search JIRA for possible patches. Whereas actually it
seems like something has been done
http://www.apache.org/dist/lucene/nutch/CHANGES-0.9.txt
issues 21
HTH
Michael
>Marcin Okraszewski
>
>007-05-15 17:38:59,465 INFO http.Http - http.proxy.host = <my_proxy_host>
>2007-05-15 17:38:59,465 INFO http.Http - http.proxy.port = <my_proxy_port>
>2007-05-15 17:38:59,465 INFO http.Http - http.timeout = 10000
>2007-05-15 17:38:59,465 INFO http.Http - http.content.limit = 65536
>2007-05-15 17:38:59,465 INFO http.Http - http.agent = YetAnotherSearchEngine/Nutch-0.9
>2007-05-15 17:38:59,465 INFO http.Http - protocol.plugin.check.blocking = true
>2007-05-15 17:38:59,465 INFO http.Http - protocol.plugin.check.robots = true
>2007-05-15 17:38:59,466 INFO http.Http - fetcher.server.delay = 100
>2007-05-15 17:38:59,466 INFO http.Http - http.max.delays = 100
>2007-05-15 17:38:59,832 ERROR http.Http - org.apache.nutch.protocol.http.api.HttpException: java.net.UnknownHostException: <crawl_site>: <crawl_site>
>2007-05-15 17:38:59,832 ERROR http.Http - at org.apache.nutch.protocol.http.api.HttpBase.blockAddr(HttpBase.java:340)
>2007-05-15 17:38:59,832 ERROR http.Http - at org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:212)
>2007-05-15 17:38:59,832 ERROR http.Http - at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:145)
>2007-05-15 17:38:59,832 ERROR http.Http - Caused by: java.net.UnknownHostException: www.gral.pl: www.gral.pl
>2007-05-15 17:38:59,832 ERROR http.Http - at java.net.InetAddress.getAllByName0(InetAddress.java:1128)
>2007-05-15 17:38:59,833 ERROR http.Http - at java.net.InetAddress.getAllByName0(InetAddress.java:1098)
>2007-05-15 17:38:59,833 ERROR http.Http - at java.net.InetAddress.getAllByName(InetAddress.java:1061)
>2007-05-15 17:38:59,833 ERROR http.Http - at java.net.InetAddress.getByName(InetAddress.java:958)
>2007-05-15 17:38:59,833 ERROR http.Http - at org.apache.nutch.protocol.http.api.HttpBase.blockAddr(HttpBase.java:336)
>2007-05-15 17:38:59,833 ERROR http.Http - ... 2 more
>2007-05-15 17:38:59,834 INFO fetcher.Fetcher - fetch of <crawl_site> failed with: org.apache.nutch.protocol.http.api.HttpException: java.net.UnknownHostException: <crawl_site>: <crawl_site>
>
>
>
>
--
Michael Wechner
Wyona - Open Source Content Management - Apache Lenya
http://www.wyona.com http://lenya.apache.org
michael.wechner@wyona.com michi@apache.org
+41 44 272 91 61