You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Harmesh <ha...@in.v2solutions.com> on 2007/03/07 10:14:52 UTC
Re: [SOLVED] memory consumpition by nutch
Hi thanks for the sugestion , but as i am not that much familiar with linux
can u please tell me what is throttling firewall and how to deploy it , more
over i would like to add that i had already change the fetcher.threads.fetch
property to 1 but it was still using the same bandbith.
Sean Dean-3 wrote:
>
> I guess your looking to limit the bandwidth used by the Nutch crawler?
>
> One idea is to set a "very low" thread count for the fetcher, maybe even
> going right down to 1 depending on the effect your looking for. Another
> idea, which would be much more controlled is to setup a throttling
> firewall on the same machine or anywhere in-between it and your gateway.
>
> If you go the thread route, you will need to place the below lines in your
> nutch-site.xml file;
>
> <property>
> <name>fetcher.threads.fetch</name>
> <value>1</value>
> <description>The number of FetcherThreads the fetcher should use.
> This is also determines the maximum number of requests that are
> made at once (each FetcherThread handles one
> connection).</description>
> </property>
>
>
> ----- Original Message ----
> From: Harmesh <ha...@in.v2solutions.com>
> To: nutch-user@lucene.apache.org
> Sent: Wednesday, March 7, 2007 2:01:50 AM
> Subject: memory consumpition by nutch
>
>
> while i am crawling most of the band with is used by nutch crawler is
> there
> any way to optimize it
> --
> View this message in context:
> http://www.nabble.com/memory-consumpition-by-nutch-tf3360441.html#a9347708
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
--
View this message in context: http://www.nabble.com/memory-consumpition-by-nutch-tf3360441.html#a9349223
Sent from the Nutch - User mailing list archive at Nabble.com.