ant build + speed

[sdeck <sc...@gmail.com>]

So, I had posted a message many weeks ago in the forum about how my speed of
a crawl would be really fast on my laptop dev, but when put on the server,
it would be really slow. Turns out that the socket reads actually do
something to the file system, so if you have a security manager that sets
permissions to be checked on the file system, your java app will spend a
long period of time checking for permissions and getting thread
locks/waiting.
I have been using ant to control my directories and crawl process, and guess
what, ant, by default, sets a security manager.

So, if you are using ant to spawn off a crawl, or any type of java app, use
the fork="true" of the java task, which will spawn it into its own thread
without a security manager.  I went from around 1 page per second to about
30-40, all because of that little issue.

Scott

-- 
View this message in context: http://www.nabble.com/ant-build-%2B-speed-tf3460778.html#a9655848
Sent from the Nutch - User mailing list archive at Nabble.com.