Nutch search and hard drive hot spots

[Dan Morrill <ra...@baker.edu>]

Just want to ask if anyone else has noticed that the index and segments
under the searcher dir are causing a hot spot on the hard drive in a heavy
transaction use search. 

 

I am on windows, Nutch 7.1, tomcat 5.15, and have tuned the system for some
decent performance, Modified both tomcat and Nutch configuration files to
increase threads, memory, and limit returns to 1 per site, but on the
hardware side I am seeing a hot spot on the drive when making a very large
query (returns more than 4000 results). 

 

Has anyone else noticed this behavior when dealing with an index that is
greater than 750 megs, and 48 gigs worth of search data? 

 

If you have noticed this, did you throw hardware at the problem or is there
a way in software to fix the issue? Would the nutch dfs work, or should I
use the os level dfs? 

 

Interested in hearing what others have seen, or if anyone else has even seen
this issue. 

 

r/d