You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by AJ Chen <ca...@gmail.com> on 2005/09/28 19:27:55 UTC
what contibute to fetch slowing down
I started the crawler with about 2000 sites. The fetcher could achieve
7 pages/sec initially, but the performance gradually dropped to about 2
pages/sec, sometimes even 0.5 pages/sec. The fetch list had 300k pages
and I used 500 threads. What are the main causes of this slowing down?
Below are sample status:
050927 005952 status: segment 20050927005922, 100 pages, 3 errors,
1784615 bytes, 14611 ms
050927 005952 status: 6.8441586 pages/s, 954.2334 kb/s, 17846.15 bytes/page
050927 010005 status: segment 20050927005922, 200 pages, 9 errors,
3656863 bytes, 28170 ms
050927 010005 status: 7.0997515 pages/s, 1014.1726 kb/s, 18284.314
bytes/page
after sometime ...
050927 171818 status: segment 20050927070752, 101400 pages, 7201 errors,
2593026554 bytes, 36216316 ms
050927 171818 status: 2.799843 pages/s, 559.3617 kb/s, 25572.254 bytes/page
050927 171832 status: segment 20050927070752, 101500 pages, 7204 errors,
2595591632 bytes, 36230516 ms
050927 171832 status: 2.8015058 pages/s, 559.6956 kb/s, 25572.332 bytes/page
Thanks,
AJ
Re: what contibute to fetch slowing down
Posted by Daniele Menozzi <me...@ngi.it>.
On 09:59:45 03/Oct , Doug Cutting wrote:
> I suspect threads are hanging, probably in the parser,
I tried to not parse, but without good results.
If I use 100 threads, I can download pages at 500KB/s for about 5 seconds,
but after that, the download rate falls to 0. If I set 20 threads, I can download
at 200KB for 4/5 minutes, and the rate initially seems very stable. But, after
theese few minutes, the rate starts to get lower and lower, and tends to reach
zero pages/s.
> but sometimes TCP
> connections get stuck too.
If this is the problem, how can I fix it? And How can I recognize it?
Thanks!
-z
--
Free Software Enthusiast
Debian Powered Linux User #332564
http://menoz.homelinux.org
Re: what contibute to fetch slowing down
Posted by Doug Cutting <cu...@nutch.org>.
Daniele Menozzi wrote:
> On 10:27:55 28/Sep , AJ Chen wrote:
>
>>I started the crawler with about 2000 sites. The fetcher could achieve
>>7 pages/sec initially, but the performance gradually dropped to about 2
>>pages/sec, sometimes even 0.5 pages/sec. The fetch list had 300k pages
>>and I used 500 threads. What are the main causes of this slowing down?
>
>
> I have the same problem; I've tried with different number of fetchers
> (10,20,50,100,..), but the download rate always decrease sistematically,
> page after page.
I suspect threads are hanging, probably in the parser, but sometimes TCP
connections get stuck too. Use 'kill -QUIT' to generate a stack dump
for all threads. Or use 'lsof' to see open TCP connections. We should
probably modify the Fetcher to, when a thread takes more than a certain
amount of time to process an individual request, terminate that thread.
Doug
Re: what contibute to fetch slowing down
Posted by Doug Cutting <cu...@nutch.org>.
Fuad Efendi wrote:
> I found this in J2SE API for setReuseAddress(default: false):
> =====
> When a TCP connection is closed the connection may remain in a timeout
> state for a period of time after the connection is closed (typically
> known as the TIME_WAIT state or 2MSL wait state). For applications using
> a well known socket address or port it may not be possible to bind a
> socket to the required SocketAddress if there is a connection in the
> timeout state involving the socket address or port.
> =====
This is related to server sockets, not client sockets.
> It probably means that we are reaching huge amount (65000!) of "waiting"
> TCP ports after Socket.close(); and Fetcher Theads are blocking by OS
> waiting when OS release some of these ports... Am I right?
I don't think that is an issue.
Doug
Re: java.net.MalformedURLException: no protocol for parse-plugins.xml
Posted by Jérôme Charron <je...@gmail.com>.
> Likely missing file:/. If I get rid of lines 617-622
> > of conf/nutch-default.xml
>
> Resolved and committed:
http://svn.apache.org/viewcvs.cgi?rev=293370&view=rev
Thanks Earl.
Jérôme
--
http://motrech.free.fr/
http://www.frutch.org/
Re: java.net.MalformedURLException: no protocol for parse-plugins.xml
Posted by Jérôme Charron <je...@gmail.com>.
> Likely missing file:/. If I get rid of lines 617-622
> of conf/nutch-default.xml
Oups, sorry.
I made this last change just after testing the whole patch.
And I doesn't test it once again since I was sure it was a minor change.
I correct this right now. Sorry.
Regards
Jérôme
--
http://motrech.free.fr/
http://www.frutch.org/
java.net.MalformedURLException: no protocol for parse-plugins.xml
Posted by Earl Cahill <ca...@yahoo.com>.
I did a clean, full svn update, and ant on trunk, then
tried
bin/nutch crawl urls -dir crawl.test
and got
051002 224950 SEVERE Unable to load parse plugins file
from URL [parse-plugins.xml]
java.net.MalformedURLException: no protocol: ...
Likely missing file:/. If I get rid of lines 617-622
of conf/nutch-default.xml
<property>
<name>parse.plugin.file</name>
<value>parse-plugins.xml</value>
<description>The name of the file that defines the
associations between
content-types and parsers.</description>
</property>
it at least lets me run my crawl. Looks like that was
added in revision 292865 on friday by jerome. Putting
in the full path works, as per this patch
Index: conf/nutch-default.xml
===================================================================
--- conf/nutch-default.xml (revision 293226)
+++ conf/nutch-default.xml (working copy)
@@ -616,7 +616,7 @@
<property>
<name>parse.plugin.file</name>
- <value>parse-plugins.xml</value>
+
<value>file:/home/nutch/nutch/trunk/conf/parse-plugins.xml</value>
<description>The name of the file that defines the
associations between
content-types and parsers.</description>
</property>
But yeah, that's not a good option. I tried each
directory and none of them worked.
Hope this gets you at least close.
Earl
__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com
RE: what contibute to fetch slowing down
Posted by Fuad Efendi <fu...@efendi.ca>.
Unfortunately this is commented in Kelvin's code:
// reqStr.append("Connection: Keep-Alive\r\n");
I found only
reqStr.append(" HTTP/1.1\r\n");
- but it does not mean implementation of HTTP/1.1 features.
Teleport Ultra v.1.29 needs just a few hours to download all plain HTML
from SUN, Nutch needs few days. 8mbps/800kbps, download/upload.
-----Original Message-----
From: Michael Ji [mailto:fji_00@yahoo.com]
Sent: Sunday, October 02, 2005 5:37 PM
To: nutch-dev@lucene.apache.org
Subject: RE: what contibute to fetch slowing down
Kelvin's OC implementation is queuing fetching request according to the
host and using http 1.1 protocol. It is a nutch patch currently.
Michael Ji,
--- Fuad Efendi <fu...@efendi.ca> wrote:
> Some suggestion to improve performance:
>
>
> 1. Decrease randomization of FetchList.
>
> Here is comment from FetchListTool:
> /**
> * The TableSet class will allocate a given FetchListEntry
> * into one of several ArrayFiles. It chooses
> which
> * ArrayFile based on a hash of the URL's domain
> name.
> *
> * It uses a hash of the domain name so that
> pages are
> * allocated to a random ArrayFile, but
> same-host pages
> * go to the same file (for efficiency purposes
> during
> * fetch).
> *
> * Further, within a given file, the
> FetchListEntry items
> * appear in random order. This is so that we
> don't
> * hammer the same site over and over again
> during fetch.
> *
> * Each table should receive a roughly
> * even number of entries, but all URLs for a
> specific
> * domain name will be found in a single table.
> If
> * the dataset is weirdly skewed toward large
> domains,
> * there may be an uneven distribution.
> */
>
> Same "same-host pages go to the same file" - they
> should go in a
> sequence, without mixing/randomizing with other
> host-pages...
>
> We are fetching single URL, then we forget about
> existense of this
> TCP/IP connection, we even forget that Web Server
> created Client Process
> to handle our HTTP requests, it is called Keep
> Alive. Creation of TCP
> connection, and additionally creation of a such
> Client Process on a Web
> Server costs a lot of CPU on both sides, Nutch &
> WebServer.
>
> I suggest to use single Keep-Alive thread to fetch
> single Host, without
> randomization.
>
>
> 2. Use/Investigate more staff from Socket API such
> as
> public void setSoTimeout(int timeout)
> public void setReuseAddress(true)
>
> I found this in J2SE API for
> setReuseAddress(default: false):
> =====
> When a TCP connection is closed the connection may
> remain in a timeout
> state for a period of time after the connection is
> closed (typically
> known as the TIME_WAIT state or 2MSL wait state).
> For applications using
> a well known socket address or port it may not be
> possible to bind a
> socket to the required SocketAddress if there is a
> connection in the
> timeout state involving the socket address or port.
> =====
>
> It probably means that we are reaching huge amount
> (65000!) of "waiting"
> TCP ports after Socket.close(); and Fetcher Theads
> are blocking by OS
> waiting when OS release some of these ports... Am I
> right?
>
>
> P.S.
> Anyway, using Keep-Alive option is very important
> not only for us but
> also for Production Web Sites.
>
> Thanks,
> Fuad
>
>
>
>
>
> -----Original Message-----
> From: Fuad Efendi [mailto:fuad@efendi.ca]
> Sent: Friday, September 30, 2005 10:58 PM
> To: nutch-dev@lucene.apache.org; menoz@ngi.it
> Subject: RE: what contibute to fetch slowing down
>
>
> Dear Nutchers,
>
>
> I noticed same problem twise, with PentiumMobile2Mhz
> & WindowsXP & 2Gb,
> and with 2xOpteron252 x SuseLinux x 4Gb
>
> I have only one explanation which should be probably
> mirrored at JIRA:
>
>
> ================
> Network.
> ========
>
>
> 1.
> I never had such a problem with The Grinder,
> http://grinder.sourceforge.net, which is based on alternate HTTPClient
> http://www.innovation.ch/java/HTTPClient/index.html.
> Apache SF should
> really review their HttpClient RC3(!!!) accordingly,
> HTTPClient
> (upper--HTTP-case)is not "alpha", it is production
> version... I used
> Grinder a lot, it allows to execute 32 processes
> with 64 threads each on
> 2048Mb RAM...
>
>
> 2.
> I found at SUN API this:
> java.net.Socket
> public void setReuseAddress(boolean on) - please
> check API!!!
>
>
> 3.
> I saw in your PROTOCOL-HTTP this code:
> ... HTTP/1.0 ...
> Why? Why version 1.0??? It should understand
> server's reply such as
> "Connection: close" "Connection: keep-alive" etc.
> (pls ignore typo).
>
>
> 4.
> By the way, how many files UNIX needs in order to
> maintain 65536 network
> sockets?
>
>
> Respectfully,
> Fuad
>
> P.S.
> Sorry guys, I don't have anough time to
> participate... Could you please
> test this suspicious behaviour, and very strange
> opinion? Should I
> create a new bug report at JIRA?
>
> SUN's Socket, Apache's HttpClient, UNIX's
> networking...
>
>
>
>
> -----Original Message-----
> From: Daniele Menozzi [mailto:menoz@ngi.it]
> Sent: Wednesday, September 28, 2005 4:42 PM
> To: nutch-dev@lucene.apache.org
> Subject: Re: what contibute to fetch slowing down
>
>
> On 10:27:55 28/Sep , AJ Chen wrote:
> > I started the crawler with about 2000 sites. The
> fetcher could
> > achieve
> > 7 pages/sec initially, but the performance
> gradually dropped to about
> 2
> > pages/sec, sometimes even 0.5 pages/sec. The
> fetch list had 300k
> pages
> > and I used 500 threads. What are the main causes
> of this slowing down?
>
>
> I have the same problem; I've tried with different
> number of fetchers
> (10,20,50,100,..), but the download rate always
> decrease
=== message truncated ===
______________________________________________________
Yahoo! for Good
Donate to the Hurricane Katrina relief effort.
http://store.yahoo.com/redcross-donate3/
RE: what contibute to fetch slowing down
Posted by Fuad Efendi <fu...@efendi.ca>.
Doug,
Thanks for reply,
I'll try to perform specific tests against in-home Apache during this
week(end) (limited in time slightly... Sorry!). Everything possible,
usually Apache httpd has "timeout" setting for keep-alive, and default
setting is (I don't remember) probably 600 seconds. I performed such
tests a while ago using Grinder... Also, I can create baseline Nutch vs
Grinder, Nutch vs Teleport. Grinder can save HTTP reply in log files
(this is mostly load-generation tool); Teleport is commercial Web
Grabber...
For HTTP/1.0 we should send explicit message "Connection: close" before
Socket.close()... But we need to perform real tests anyway.
-----Original Message-----
From: Doug Cutting [mailto:cutting@nutch.org]
Sent: Monday, October 03, 2005 1:05 PM
To: nutch-dev@lucene.apache.org
Subject: Re: what contibute to fetch slowing down
Fuad Efendi wrote:
> If I am right, we are simply _killing_ many many sites with default
> Apache HTTPD installation (Microsoft IIS, etc.) (150 keep-alive client
> threads; I configured 6000 threads for Worker model, but it was very
> unusual). Those client threads are created each time for each single
> HTTP request from Nutch, after 150 pages we are simply overloading Web
> Server, and we receive "connection timeout exception".
I would be surprised if a web server, on exhausting it's keep-alive
cache, wouldn't simply close some of them. Repeated connections without
keep-alive should not harm a web server, as long as they're polite.
> We need to use real Web Server during tests, and HTTP Proxy
> (http://grinder.sourceforge.net - very simple Java based proxy)
That would be a great contribution. Do you have time to work on this?
Doug
Re: what contibute to fetch slowing down
Posted by Doug Cutting <cu...@nutch.org>.
Fuad Efendi wrote:
> If I am right, we are simply _killing_ many many sites with default
> Apache HTTPD installation (Microsoft IIS, etc.) (150 keep-alive client
> threads; I configured 6000 threads for Worker model, but it was very
> unusual). Those client threads are created each time for each single
> HTTP request from Nutch, after 150 pages we are simply overloading Web
> Server, and we receive "connection timeout exception".
I would be surprised if a web server, on exhausting it's keep-alive
cache, wouldn't simply close some of them. Repeated connections without
keep-alive should not harm a web server, as long as they're polite.
> We need to use real Web Server during tests, and HTTP Proxy
> (http://grinder.sourceforge.net - very simple Java based proxy)
That would be a great contribution. Do you have time to work on this?
Doug
RE: what contibute to fetch slowing down
Posted by Fuad Efendi <fu...@efendi.ca>.
I never tried Kelvin's OC, I only browsed source code a little.
We need to make test with JVM 1.4, and JVM 1.5 (Kelvin's OC).
If I am right, we are simply _killing_ many many sites with default
Apache HTTPD installation (Microsoft IIS, etc.) (150 keep-alive client
threads; I configured 6000 threads for Worker model, but it was very
unusual). Those client threads are created each time for each single
HTTP request from Nutch, after 150 pages we are simply overloading Web
Server, and we receive "connection timeout exception".
We need to use real Web Server during tests, and HTTP Proxy
(http://grinder.sourceforge.net - very simple Java based proxy)
-----Original Message-----
From: Michael Ji [mailto:fji_00@yahoo.com]
Sent: Sunday, October 02, 2005 5:37 PM
To: nutch-dev@lucene.apache.org
Subject: RE: what contibute to fetch slowing down
Kelvin's OC implementation is queuing fetching request according to the
host and using http 1.1 protocol. It is a nutch patch currently.
Michael Ji,
--- Fuad Efendi <fu...@efendi.ca> wrote:
> Some suggestion to improve performance:
>
>
> 1. Decrease randomization of FetchList.
>
> Here is comment from FetchListTool:
> /**
> * The TableSet class will allocate a given FetchListEntry
> * into one of several ArrayFiles. It chooses
> which
> * ArrayFile based on a hash of the URL's domain
> name.
> *
> * It uses a hash of the domain name so that
> pages are
> * allocated to a random ArrayFile, but
> same-host pages
> * go to the same file (for efficiency purposes
> during
> * fetch).
> *
> * Further, within a given file, the
> FetchListEntry items
> * appear in random order. This is so that we
> don't
> * hammer the same site over and over again
> during fetch.
> *
> * Each table should receive a roughly
> * even number of entries, but all URLs for a
> specific
> * domain name will be found in a single table.
> If
> * the dataset is weirdly skewed toward large
> domains,
> * there may be an uneven distribution.
> */
>
> Same "same-host pages go to the same file" - they
> should go in a
> sequence, without mixing/randomizing with other
> host-pages...
>
> We are fetching single URL, then we forget about
> existense of this
> TCP/IP connection, we even forget that Web Server
> created Client Process
> to handle our HTTP requests, it is called Keep
> Alive. Creation of TCP
> connection, and additionally creation of a such
> Client Process on a Web
> Server costs a lot of CPU on both sides, Nutch &
> WebServer.
>
> I suggest to use single Keep-Alive thread to fetch
> single Host, without
> randomization.
>
>
> 2. Use/Investigate more staff from Socket API such
> as
> public void setSoTimeout(int timeout)
> public void setReuseAddress(true)
>
> I found this in J2SE API for
> setReuseAddress(default: false):
> =====
> When a TCP connection is closed the connection may
> remain in a timeout
> state for a period of time after the connection is
> closed (typically
> known as the TIME_WAIT state or 2MSL wait state).
> For applications using
> a well known socket address or port it may not be
> possible to bind a
> socket to the required SocketAddress if there is a
> connection in the
> timeout state involving the socket address or port.
> =====
>
> It probably means that we are reaching huge amount
> (65000!) of "waiting"
> TCP ports after Socket.close(); and Fetcher Theads
> are blocking by OS
> waiting when OS release some of these ports... Am I
> right?
>
>
> P.S.
> Anyway, using Keep-Alive option is very important
> not only for us but
> also for Production Web Sites.
>
> Thanks,
> Fuad
>
>
>
>
>
> -----Original Message-----
> From: Fuad Efendi [mailto:fuad@efendi.ca]
> Sent: Friday, September 30, 2005 10:58 PM
> To: nutch-dev@lucene.apache.org; menoz@ngi.it
> Subject: RE: what contibute to fetch slowing down
>
>
> Dear Nutchers,
>
>
> I noticed same problem twise, with PentiumMobile2Mhz
> & WindowsXP & 2Gb,
> and with 2xOpteron252 x SuseLinux x 4Gb
>
> I have only one explanation which should be probably
> mirrored at JIRA:
>
>
> ================
> Network.
> ========
>
>
> 1.
> I never had such a problem with The Grinder,
> http://grinder.sourceforge.net, which is based on alternate HTTPClient
> http://www.innovation.ch/java/HTTPClient/index.html.
> Apache SF should
> really review their HttpClient RC3(!!!) accordingly,
> HTTPClient
> (upper--HTTP-case)is not "alpha", it is production
> version... I used
> Grinder a lot, it allows to execute 32 processes
> with 64 threads each on
> 2048Mb RAM...
>
>
> 2.
> I found at SUN API this:
> java.net.Socket
> public void setReuseAddress(boolean on) - please
> check API!!!
>
>
> 3.
> I saw in your PROTOCOL-HTTP this code:
> ... HTTP/1.0 ...
> Why? Why version 1.0??? It should understand
> server's reply such as
> "Connection: close" "Connection: keep-alive" etc.
> (pls ignore typo).
>
>
> 4.
> By the way, how many files UNIX needs in order to
> maintain 65536 network
> sockets?
>
>
> Respectfully,
> Fuad
>
> P.S.
> Sorry guys, I don't have anough time to
> participate... Could you please
> test this suspicious behaviour, and very strange
> opinion? Should I
> create a new bug report at JIRA?
>
> SUN's Socket, Apache's HttpClient, UNIX's
> networking...
>
>
>
>
> -----Original Message-----
> From: Daniele Menozzi [mailto:menoz@ngi.it]
> Sent: Wednesday, September 28, 2005 4:42 PM
> To: nutch-dev@lucene.apache.org
> Subject: Re: what contibute to fetch slowing down
>
>
> On 10:27:55 28/Sep , AJ Chen wrote:
> > I started the crawler with about 2000 sites. The
> fetcher could
> > achieve
> > 7 pages/sec initially, but the performance
> gradually dropped to about
> 2
> > pages/sec, sometimes even 0.5 pages/sec. The
> fetch list had 300k
> pages
> > and I used 500 threads. What are the main causes
> of this slowing down?
>
>
> I have the same problem; I've tried with different
> number of fetchers
> (10,20,50,100,..), but the download rate always
> decrease
=== message truncated ===
______________________________________________________
Yahoo! for Good
Donate to the Hurricane Katrina relief effort.
http://store.yahoo.com/redcross-donate3/
RE: what contibute to fetch slowing down
Posted by Michael Ji <fj...@yahoo.com>.
Kelvin's OC implementation is queuing fetching request
according to the host and using http 1.1 protocol. It
is a nutch patch currently.
Michael Ji,
--- Fuad Efendi <fu...@efendi.ca> wrote:
> Some suggestion to improve performance:
>
>
> 1. Decrease randomization of FetchList.
>
> Here is comment from FetchListTool:
> /**
> * The TableSet class will allocate a given
> FetchListEntry
> * into one of several ArrayFiles. It chooses
> which
> * ArrayFile based on a hash of the URL's domain
> name.
> *
> * It uses a hash of the domain name so that
> pages are
> * allocated to a random ArrayFile, but
> same-host pages
> * go to the same file (for efficiency purposes
> during
> * fetch).
> *
> * Further, within a given file, the
> FetchListEntry items
> * appear in random order. This is so that we
> don't
> * hammer the same site over and over again
> during fetch.
> *
> * Each table should receive a roughly
> * even number of entries, but all URLs for a
> specific
> * domain name will be found in a single table.
> If
> * the dataset is weirdly skewed toward large
> domains,
> * there may be an uneven distribution.
> */
>
> Same "same-host pages go to the same file" - they
> should go in a
> sequence, without mixing/randomizing with other
> host-pages...
>
> We are fetching single URL, then we forget about
> existense of this
> TCP/IP connection, we even forget that Web Server
> created Client Process
> to handle our HTTP requests, it is called Keep
> Alive. Creation of TCP
> connection, and additionally creation of a such
> Client Process on a Web
> Server costs a lot of CPU on both sides, Nutch &
> WebServer.
>
> I suggest to use single Keep-Alive thread to fetch
> single Host, without
> randomization.
>
>
> 2. Use/Investigate more staff from Socket API such
> as
> public void setSoTimeout(int timeout)
> public void setReuseAddress(true)
>
> I found this in J2SE API for
> setReuseAddress(default: false):
> =====
> When a TCP connection is closed the connection may
> remain in a timeout
> state for a period of time after the connection is
> closed (typically
> known as the TIME_WAIT state or 2MSL wait state).
> For applications using
> a well known socket address or port it may not be
> possible to bind a
> socket to the required SocketAddress if there is a
> connection in the
> timeout state involving the socket address or port.
> =====
>
> It probably means that we are reaching huge amount
> (65000!) of "waiting"
> TCP ports after Socket.close(); and Fetcher Theads
> are blocking by OS
> waiting when OS release some of these ports... Am I
> right?
>
>
> P.S.
> Anyway, using Keep-Alive option is very important
> not only for us but
> also for Production Web Sites.
>
> Thanks,
> Fuad
>
>
>
>
>
> -----Original Message-----
> From: Fuad Efendi [mailto:fuad@efendi.ca]
> Sent: Friday, September 30, 2005 10:58 PM
> To: nutch-dev@lucene.apache.org; menoz@ngi.it
> Subject: RE: what contibute to fetch slowing down
>
>
> Dear Nutchers,
>
>
> I noticed same problem twise, with PentiumMobile2Mhz
> & WindowsXP & 2Gb,
> and with 2xOpteron252 x SuseLinux x 4Gb
>
> I have only one explanation which should be probably
> mirrored at JIRA:
>
>
> ================
> Network.
> ========
>
>
> 1.
> I never had such a problem with The Grinder,
> http://grinder.sourceforge.net, which is based on
> alternate HTTPClient
> http://www.innovation.ch/java/HTTPClient/index.html.
> Apache SF should
> really review their HttpClient RC3(!!!) accordingly,
> HTTPClient
> (upper--HTTP-case)is not "alpha", it is production
> version... I used
> Grinder a lot, it allows to execute 32 processes
> with 64 threads each on
> 2048Mb RAM...
>
>
> 2.
> I found at SUN API this:
> java.net.Socket
> public void setReuseAddress(boolean on) - please
> check API!!!
>
>
> 3.
> I saw in your PROTOCOL-HTTP this code:
> ... HTTP/1.0 ...
> Why? Why version 1.0??? It should understand
> server's reply such as
> "Connection: close" "Connection: keep-alive" etc.
> (pls ignore typo).
>
>
> 4.
> By the way, how many files UNIX needs in order to
> maintain 65536 network
> sockets?
>
>
> Respectfully,
> Fuad
>
> P.S.
> Sorry guys, I don't have anough time to
> participate... Could you please
> test this suspicious behaviour, and very strange
> opinion? Should I
> create a new bug report at JIRA?
>
> SUN's Socket, Apache's HttpClient, UNIX's
> networking...
>
>
>
>
> -----Original Message-----
> From: Daniele Menozzi [mailto:menoz@ngi.it]
> Sent: Wednesday, September 28, 2005 4:42 PM
> To: nutch-dev@lucene.apache.org
> Subject: Re: what contibute to fetch slowing down
>
>
> On 10:27:55 28/Sep , AJ Chen wrote:
> > I started the crawler with about 2000 sites. The
> fetcher could
> > achieve
> > 7 pages/sec initially, but the performance
> gradually dropped to about
> 2
> > pages/sec, sometimes even 0.5 pages/sec. The
> fetch list had 300k
> pages
> > and I used 500 threads. What are the main causes
> of this slowing down?
>
>
> I have the same problem; I've tried with different
> number of fetchers
> (10,20,50,100,..), but the download rate always
> decrease
=== message truncated ===
______________________________________________________
Yahoo! for Good
Donate to the Hurricane Katrina relief effort.
http://store.yahoo.com/redcross-donate3/
RE: what contibute to fetch slowing down
Posted by Fuad Efendi <fu...@efendi.ca>.
Some suggestion to improve performance:
1. Decrease randomization of FetchList.
Here is comment from FetchListTool:
/**
* The TableSet class will allocate a given FetchListEntry
* into one of several ArrayFiles. It chooses which
* ArrayFile based on a hash of the URL's domain name.
*
* It uses a hash of the domain name so that pages are
* allocated to a random ArrayFile, but same-host pages
* go to the same file (for efficiency purposes during
* fetch).
*
* Further, within a given file, the FetchListEntry items
* appear in random order. This is so that we don't
* hammer the same site over and over again during fetch.
*
* Each table should receive a roughly
* even number of entries, but all URLs for a specific
* domain name will be found in a single table. If
* the dataset is weirdly skewed toward large domains,
* there may be an uneven distribution.
*/
Same "same-host pages go to the same file" - they should go in a
sequence, without mixing/randomizing with other host-pages...
We are fetching single URL, then we forget about existense of this
TCP/IP connection, we even forget that Web Server created Client Process
to handle our HTTP requests, it is called Keep Alive. Creation of TCP
connection, and additionally creation of a such Client Process on a Web
Server costs a lot of CPU on both sides, Nutch & WebServer.
I suggest to use single Keep-Alive thread to fetch single Host, without
randomization.
2. Use/Investigate more staff from Socket API such as
public void setSoTimeout(int timeout)
public void setReuseAddress(true)
I found this in J2SE API for setReuseAddress(default: false):
=====
When a TCP connection is closed the connection may remain in a timeout
state for a period of time after the connection is closed (typically
known as the TIME_WAIT state or 2MSL wait state). For applications using
a well known socket address or port it may not be possible to bind a
socket to the required SocketAddress if there is a connection in the
timeout state involving the socket address or port.
=====
It probably means that we are reaching huge amount (65000!) of "waiting"
TCP ports after Socket.close(); and Fetcher Theads are blocking by OS
waiting when OS release some of these ports... Am I right?
P.S.
Anyway, using Keep-Alive option is very important not only for us but
also for Production Web Sites.
Thanks,
Fuad
-----Original Message-----
From: Fuad Efendi [mailto:fuad@efendi.ca]
Sent: Friday, September 30, 2005 10:58 PM
To: nutch-dev@lucene.apache.org; menoz@ngi.it
Subject: RE: what contibute to fetch slowing down
Dear Nutchers,
I noticed same problem twise, with PentiumMobile2Mhz & WindowsXP & 2Gb,
and with 2xOpteron252 x SuseLinux x 4Gb
I have only one explanation which should be probably mirrored at JIRA:
================
Network.
========
1.
I never had such a problem with The Grinder,
http://grinder.sourceforge.net, which is based on alternate HTTPClient
http://www.innovation.ch/java/HTTPClient/index.html. Apache SF should
really review their HttpClient RC3(!!!) accordingly, HTTPClient
(upper--HTTP-case)is not "alpha", it is production version... I used
Grinder a lot, it allows to execute 32 processes with 64 threads each on
2048Mb RAM...
2.
I found at SUN API this:
java.net.Socket
public void setReuseAddress(boolean on) - please check API!!!
3.
I saw in your PROTOCOL-HTTP this code:
... HTTP/1.0 ...
Why? Why version 1.0??? It should understand server's reply such as
"Connection: close" "Connection: keep-alive" etc. (pls ignore typo).
4.
By the way, how many files UNIX needs in order to maintain 65536 network
sockets?
Respectfully,
Fuad
P.S.
Sorry guys, I don't have anough time to participate... Could you please
test this suspicious behaviour, and very strange opinion? Should I
create a new bug report at JIRA?
SUN's Socket, Apache's HttpClient, UNIX's networking...
-----Original Message-----
From: Daniele Menozzi [mailto:menoz@ngi.it]
Sent: Wednesday, September 28, 2005 4:42 PM
To: nutch-dev@lucene.apache.org
Subject: Re: what contibute to fetch slowing down
On 10:27:55 28/Sep , AJ Chen wrote:
> I started the crawler with about 2000 sites. The fetcher could
> achieve
> 7 pages/sec initially, but the performance gradually dropped to about
2
> pages/sec, sometimes even 0.5 pages/sec. The fetch list had 300k
pages
> and I used 500 threads. What are the main causes of this slowing down?
I have the same problem; I've tried with different number of fetchers
(10,20,50,100,..), but the download rate always decrease sistematically,
page after page. The machine is a p4 1.7, 768 MB ram, running debian on
2.6.12 kernel. The bandwidth isn't a problem (10Mbit in and 10Mbit out),
but I cannot obtain a stable, and high, page/s rate. I've also tried to
change machine and kernel, but the problem still remains. Can you please
give us some advice? Thank you for your help,
Menoz
--
Free Software Enthusiast
Debian Powered Linux User #332564
http://menoz.homelinux.org
RE: what contibute to fetch slowing down
Posted by Fuad Efendi <fu...@efendi.ca>.
Dear Nutchers,
I noticed same problem twise, with PentiumMobile2Mhz & WindowsXP & 2Gb,
and with 2xOpteron252 x SuseLinux x 4Gb
I have only one explanation which should be probably mirrored at JIRA:
================
Network.
========
1.
I never had such a problem with The Grinder,
http://grinder.sourceforge.net, which is based on alternate HTTPClient
http://www.innovation.ch/java/HTTPClient/index.html. Apache SF should
really review their HttpClient RC3(!!!) accordingly, HTTPClient
(upper--HTTP-case)is not "alpha", it is production version... I used
Grinder a lot, it allows to execute 32 processes with 64 threads each on
2048Mb RAM...
2.
I found at SUN API this:
java.net.Socket
public void setReuseAddress(boolean on) - please check API!!!
3.
I saw in your PROTOCOL-HTTP this code:
... HTTP/1.0 ...
Why? Why version 1.0??? It should understand server's reply such as
"Connection: close" "Connection: keep-alive" etc. (pls ignore typo).
4.
By the way, how many files UNIX needs in order to maintain 65536 network
sockets?
Respectfully,
Fuad
P.S.
Sorry guys, I don't have anough time to participate... Could you please
test this suspicious behaviour, and very strange opinion? Should I
create a new bug report at JIRA?
SUN's Socket, Apache's HttpClient, UNIX's networking...
-----Original Message-----
From: Daniele Menozzi [mailto:menoz@ngi.it]
Sent: Wednesday, September 28, 2005 4:42 PM
To: nutch-dev@lucene.apache.org
Subject: Re: what contibute to fetch slowing down
On 10:27:55 28/Sep , AJ Chen wrote:
> I started the crawler with about 2000 sites. The fetcher could
> achieve
> 7 pages/sec initially, but the performance gradually dropped to about
2
> pages/sec, sometimes even 0.5 pages/sec. The fetch list had 300k
pages
> and I used 500 threads. What are the main causes of this slowing down?
I have the same problem; I've tried with different number of fetchers
(10,20,50,100,..), but the download rate always decrease sistematically,
page after page. The machine is a p4 1.7, 768 MB ram, running debian on
2.6.12 kernel. The bandwidth isn't a problem (10Mbit in and 10Mbit out),
but I cannot obtain a stable, and high, page/s rate. I've also tried to
change machine and kernel, but the problem still remains. Can you please
give us some advice? Thank you for your help,
Menoz
--
Free Software Enthusiast
Debian Powered Linux User #332564
http://menoz.homelinux.org
Re: what contibute to fetch slowing down
Posted by Daniele Menozzi <me...@ngi.it>.
On 10:27:55 28/Sep , AJ Chen wrote:
> I started the crawler with about 2000 sites. The fetcher could achieve
> 7 pages/sec initially, but the performance gradually dropped to about 2
> pages/sec, sometimes even 0.5 pages/sec. The fetch list had 300k pages
> and I used 500 threads. What are the main causes of this slowing down?
I have the same problem; I've tried with different number of fetchers
(10,20,50,100,..), but the download rate always decrease sistematically,
page after page.
The machine is a p4 1.7, 768 MB ram, running debian on 2.6.12 kernel.
The bandwidth isn't a problem (10Mbit in and 10Mbit out), but I cannot
obtain a stable, and high, page/s rate.
I've also tried to change machine and kernel, but the problem still
remains.
Can you please give us some advice?
Thank you for your help,
Menoz
--
Free Software Enthusiast
Debian Powered Linux User #332564
http://menoz.homelinux.org
Other Engine that users Nutch
Posted by "tigger ." <b1...@hotmail.com>.
Hi All
Found another engine that users Nutch
http://www.bigclique.com/
But it does not says everything about Nutch on their web site.
Re: what contibute to fetch slowing down
Posted by Jack Tang <hi...@gmail.com>.
Hi AJ
I guess the growing of thread.
You can show the thread id in the log. I think it makes sence
Regards
/Jack
On 9/29/05, AJ Chen <ca...@gmail.com> wrote:
> I started the crawler with about 2000 sites. The fetcher could achieve
> 7 pages/sec initially, but the performance gradually dropped to about 2
> pages/sec, sometimes even 0.5 pages/sec. The fetch list had 300k pages
> and I used 500 threads. What are the main causes of this slowing down?
> Below are sample status:
>
> 050927 005952 status: segment 20050927005922, 100 pages, 3 errors,
> 1784615 bytes, 14611 ms
> 050927 005952 status: 6.8441586 pages/s, 954.2334 kb/s, 17846.15 bytes/page
> 050927 010005 status: segment 20050927005922, 200 pages, 9 errors,
> 3656863 bytes, 28170 ms
> 050927 010005 status: 7.0997515 pages/s, 1014.1726 kb/s, 18284.314
> bytes/page
>
> after sometime ...
> 050927 171818 status: segment 20050927070752, 101400 pages, 7201 errors,
> 2593026554 bytes, 36216316 ms
> 050927 171818 status: 2.799843 pages/s, 559.3617 kb/s, 25572.254 bytes/page
> 050927 171832 status: segment 20050927070752, 101500 pages, 7204 errors,
> 2595591632 bytes, 36230516 ms
> 050927 171832 status: 2.8015058 pages/s, 559.6956 kb/s, 25572.332 bytes/page
>
> Thanks,
> AJ
>
>
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
Re: what contibute to fetch slowing down
Posted by Ken Krugler <kk...@transpac.com>.
>Update on fetch performance of my current run: download speed has been
>stable at 3.8 pages/sec, about 640kbps. This is probably limited by my
>bandwidth - regular DSL service, promising up to 1.5 mbps inbound but
>realistically only 640 kbps.
>
>More than 1 million pages were fetched, but it took several days at current
>speed - just too slow. I'm planning to get more bandwidth. Could someone
>share their experience on what stable rate (pages/sec) can be achieved using
>3 mbps or 10 mbps inbound connection?
I'm currently getting about 6 pages/second, and peaking at 4.8MBps,
with a fast machine (Xeon quad 3.0GHz, 4GB RAM, SCSI RAID 5 disks)
and a big pipe.
I'm doing just fetching & parsing, no indexing, with a 10MB max
content size. Note that the max size can matter - we'd originally set
it to no limit, so that we wouldn't miss any PDFs. But this caused
other problems - for example, the regex-urlfilters.txt file that you
get with Nutch 0.7 doesn't exclude .bz2 files. So we were chewing up
lots of bandwidth, and occasionally running out of memory, when
trying to download these.
I think our pages/second would be faster, but I'm also getting lots
of max-retry errors. We're doing a limited domain crawl, so this gets
hit a lot because there are typically many URLs from the same host in
our fetchlist. Things improved a bit when we randomized the URL list
after doing the topN prune, so that they weren't in (essentially)
alphabetical order.
When you use the FetchListTool to emit multiple lists, it
intentionally divides up the list using the MD5 value for the link,
so that you get hosts scattered between the lists. But for a single
list, this doesn't happen, and thus the max threads/host value winds
up causing a lot of the threads to spend their time idling, if your
crawl (like mine) is focused.
Also 500 threads sounds excessive to me, but I don't really know.
-- Ken
>On 9/28/05, AJ Chen <ca...@gmail.com> wrote:
>>
>> I started the crawler with about 2000 sites. The fetcher could achieve
>> 7 pages/sec initially, but the performance gradually dropped to about 2
>> pages/sec, sometimes even 0.5 pages/sec. The fetch list had 300k pages
>> and I used 500 threads. What are the main causes of this slowing down?
>> Below are sample status:
>>
>> 050927 005952 status: segment 20050927005922, 100 pages, 3 errors,
>> 1784615 bytes, 14611 ms
>> 050927 005952 status: 6.8441586 pages/s, 954.2334 kb/s, 17846.15bytes/page
>> 050927 010005 status: segment 20050927005922, 200 pages, 9 errors,
>> 3656863 bytes, 28170 ms
>> 050927 010005 status: 7.0997515 pages/s, 1014.1726 kb/s, 18284.314
>> bytes/page
>>
>> after sometime ...
>> 050927 171818 status: segment 20050927070752, 101400 pages, 7201 errors,
>> 2593026554 bytes, 36216316 ms
>> 050927 171818 status: 2.799843 pages/s, 559.3617 kb/s, 25572.254bytes/page
>> 050927 171832 status: segment 20050927070752, 101500 pages, 7204 errors,
>> 2595591632 bytes, 36230516 ms
>> 050927 171832 status: 2.8015058 pages/s, 559.6956 kb/s, 25572.332bytes/page
>>
>> Thanks,
> > AJ
--
Ken Krugler
Krugle, Inc.
+1 530-470-9200
RE: Re[2]: what contibute to fetch slowing down
Posted by og...@yahoo.com.
Fuad,
I think you are constantly comparing apples and oranges here. It looks
like your new code simply hammers the server sending multiple requests
to a single server in parallel. That's a big no-no in a web
crawling/spidering/fetching world, as bad as not obeying robots.txt.
The fact that the speed difference is SO large is a clear hint that the
comparison may not be right, and that the 3 plugins you are comparing
are configured very differently.
Otis
--- Fuad Efendi <fu...@efendi.ca> wrote:
> Try new Protocol-HTTPClient-Innovation:
> http://issues.apache.org/jira/browse/NUTCH-109
>
>
> -----Original Message-----
> From: Daniele Menozzi [mailto:menoz@ngi.it]
> Sent: Monday, October 10, 2005 5:42 PM
> To: nutch-dev@lucene.apache.org
> Subject: Re: Re[2]: what contibute to fetch slowing down
>
>
> On 03:36:45 03/Oct , Michael wrote:
> > 3mbit, 100 threads = 15 pages/sec
> > cpu is low during fetch, so its bandwidth limit.
>
> yes, cpu is low, and even memory is quite free. But, with a 10MB
> in/out I
> cannot obtain good results (and I do not parse results, simply fetch
> them).
> If I use 100 threads, I can download pages at 500KB/s for about 5
> seconds,
> but after that, the download rate falls to 0. If I set 20 threads, I
> can
> download
> at 200KB for 4/5 minutes, and the rate initially seems very stable.
> But,
> after theese few minutes, the rate starts to get lower and lower, and
> tends
> to reach zero pages/s.
>
> I cannot understand what could be the problem. Every thread number I
> choose,
> the rate _always_ decrease, till it has reached 1/2 pages/s. I;ve
> tried 2
> different machines, but the problem is always the same.
>
> Can you please give me some advices?
> Thank you
> Daniele
>
>
>
> --
> Free Software Enthusiast
> Debian Powered Linux User #332564
> http://menoz.homelinux.org
>
>
>
RE: Re[2]: what contibute to fetch slowing down
Posted by Fuad Efendi <fu...@efendi.ca>.
Try new Protocol-HTTPClient-Innovation:
http://issues.apache.org/jira/browse/NUTCH-109
-----Original Message-----
From: Daniele Menozzi [mailto:menoz@ngi.it]
Sent: Monday, October 10, 2005 5:42 PM
To: nutch-dev@lucene.apache.org
Subject: Re: Re[2]: what contibute to fetch slowing down
On 03:36:45 03/Oct , Michael wrote:
> 3mbit, 100 threads = 15 pages/sec
> cpu is low during fetch, so its bandwidth limit.
yes, cpu is low, and even memory is quite free. But, with a 10MB in/out I
cannot obtain good results (and I do not parse results, simply fetch them).
If I use 100 threads, I can download pages at 500KB/s for about 5 seconds,
but after that, the download rate falls to 0. If I set 20 threads, I can
download
at 200KB for 4/5 minutes, and the rate initially seems very stable. But,
after theese few minutes, the rate starts to get lower and lower, and tends
to reach zero pages/s.
I cannot understand what could be the problem. Every thread number I choose,
the rate _always_ decrease, till it has reached 1/2 pages/s. I;ve tried 2
different machines, but the problem is always the same.
Can you please give me some advices?
Thank you
Daniele
--
Free Software Enthusiast
Debian Powered Linux User #332564
http://menoz.homelinux.org
RE: Re[2]: what contibute to fetch slowing down
Posted by Fuad Efendi <fu...@efendi.ca>.
http://nagoya.apache.org/jira
- it does not work right now, I am trying to upload new Http-Plugin which
seems to be 100 times faster.
1. TCP connection costs a lot, not only for Nutch and end-point but also for
intermediary network equipment
2. Web Server creates Client thread and hopes that Nutch really uses
HTTP/1.1, or at least Nutch sends "Connection: close" before closing in JVM
"Socket.close()"
...
I need to perform very objective tests, probably 2-3 days; new plugin
crawled/parsed 23,000 pages for 1,321 seconds; it seems that existing
http-plugin needs few days...
I am using separate network segment with Windows XP (Nutch), and Suse Linux
(Apache HTTPD + 120,000 pages)
-----Original Message-----
From: Daniele Menozzi [mailto:menoz@ngi.it]
Sent: Monday, October 10, 2005 5:42 PM
To: nutch-dev@lucene.apache.org
Subject: Re: Re[2]: what contibute to fetch slowing down
On 03:36:45 03/Oct , Michael wrote:
> 3mbit, 100 threads = 15 pages/sec
> cpu is low during fetch, so its bandwidth limit.
yes, cpu is low, and even memory is quite free. But, with a 10MB in/out I
cannot obtain good results (and I do not parse results, simply fetch them).
If I use 100 threads, I can download pages at 500KB/s for about 5 seconds,
but after that, the download rate falls to 0. If I set 20 threads, I can
download
at 200KB for 4/5 minutes, and the rate initially seems very stable. But,
after theese few minutes, the rate starts to get lower and lower, and tends
to reach zero pages/s.
I cannot understand what could be the problem. Every thread number I choose,
the rate _always_ decrease, till it has reached 1/2 pages/s. I;ve tried 2
different machines, but the problem is always the same.
Can you please give me some advices?
Thank you
Daniele
--
Free Software Enthusiast
Debian Powered Linux User #332564
http://menoz.homelinux.org
Re: Re[2]: what contibute to fetch slowing down
Posted by Daniele Menozzi <me...@ngi.it>.
On 03:36:45 03/Oct , Michael wrote:
> 3mbit, 100 threads = 15 pages/sec
> cpu is low during fetch, so its bandwidth limit.
yes, cpu is low, and even memory is quite free. But, with a 10MB in/out
I cannot obtain good results (and I do not parse results, simply fetch
them).
If I use 100 threads, I can download pages at 500KB/s for about 5 seconds,
but after that, the download rate falls to 0. If I set 20 threads, I can download
at 200KB for 4/5 minutes, and the rate initially seems very stable. But, after
theese few minutes, the rate starts to get lower and lower, and tends to reach
zero pages/s.
I cannot understand what could be the problem. Every thread number I choose, the rate _always_
decrease, till it has reached 1/2 pages/s.
I;ve tried 2 different machines, but the problem is always the same.
Can you please give me some advices?
Thank you
Daniele
--
Free Software Enthusiast
Debian Powered Linux User #332564
http://menoz.homelinux.org
Re[2]: what contibute to fetch slowing down
Posted by Michael <mi...@gameservice.ru>.
3mbit, 100 threads = 15 pages/sec
cpu is low during fetch, so its bandwidth limit.
AC> More than 1 million pages were fetched, but it took several days at current
AC> speed - just too slow. I'm planning to get more bandwidth. Could someone
AC> share their experience on what stable rate (pages/sec) can be achieved using
AC> 3 mbps or 10 mbps inbound connection?
Michael
Re: what contibute to fetch slowing down
Posted by AJ Chen <ca...@gmail.com>.
Update on fetch performance of my current run: download speed has been
stable at 3.8 pages/sec, about 640kbps. This is probably limited by my
bandwidth - regular DSL service, promising up to 1.5 mbps inbound but
realistically only 640 kbps.
More than 1 million pages were fetched, but it took several days at current
speed - just too slow. I'm planning to get more bandwidth. Could someone
share their experience on what stable rate (pages/sec) can be achieved using
3 mbps or 10 mbps inbound connection?
Thanks,
AJ
On 9/28/05, AJ Chen <ca...@gmail.com> wrote:
>
> I started the crawler with about 2000 sites. The fetcher could achieve
> 7 pages/sec initially, but the performance gradually dropped to about 2
> pages/sec, sometimes even 0.5 pages/sec. The fetch list had 300k pages
> and I used 500 threads. What are the main causes of this slowing down?
> Below are sample status:
>
> 050927 005952 status: segment 20050927005922, 100 pages, 3 errors,
> 1784615 bytes, 14611 ms
> 050927 005952 status: 6.8441586 pages/s, 954.2334 kb/s, 17846.15bytes/page
> 050927 010005 status: segment 20050927005922, 200 pages, 9 errors,
> 3656863 bytes, 28170 ms
> 050927 010005 status: 7.0997515 pages/s, 1014.1726 kb/s, 18284.314
> bytes/page
>
> after sometime ...
> 050927 171818 status: segment 20050927070752, 101400 pages, 7201 errors,
> 2593026554 bytes, 36216316 ms
> 050927 171818 status: 2.799843 pages/s, 559.3617 kb/s, 25572.254bytes/page
> 050927 171832 status: segment 20050927070752, 101500 pages, 7204 errors,
> 2595591632 bytes, 36230516 ms
> 050927 171832 status: 2.8015058 pages/s, 559.6956 kb/s, 25572.332bytes/page
>
> Thanks,
> AJ
>
>