Fetcher hangs for a long time

[Johannes Dorn <jo...@johannet.de>]

Hello,

i have implemented a Protocol that allows me to parse "xyz://something:something" urls.
The parsing works fine, however, when fetching a large amount of URLs, it takes a very long time for the fetcher to finish.

All my URLs are stored in a single text file containing 63000 entries. I am working in local mode. 
When i set -topN to 10000, it takes about 10 minutes for the fetcher to finish. 
Now i have set -topN to 65000 but i am stuck at this point now for 20 hours.

...
fetching xyz://fzu11:vfjd
fetching xyz://iiur:nepw
-finishing thread FetcherThread, activeThreads=9
-finishing thread FetcherThread, activeThreads=8
-activeThreads=8, spinWaiting=0, fetchQueues.totalSize=0
-finishing thread FetcherThread, activeThreads=7
-finishing thread FetcherThread, activeThreads=6
-finishing thread FetcherThread, activeThreads=5
-finishing thread FetcherThread, activeThreads=4
-finishing thread FetcherThread, activeThreads=3
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=1
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0

What is happening here? No log output is available even though i set the fetcher logging to verbose.

Is there any way to speed things up?

Johannes