Problem with time out on QueueFeeder

[Paul Rogers <pa...@gmail.com>]

Hi Guys

I am using nutch 1.8 to fetch pdf documents from an http server.  The jobs
have been running OK until recently when I started getting the following
error:

-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=2500
fetching
http://server1/doccontrol/DC-10%20Incoming%20Correspondence(IAE-US)/15C_221427_IAE_LTR_IAE_0845%20Letter%20from%20Alvarez.pdf
(queue crawl delay=5000ms)
-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=2500
-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=2500
QueueFeeder finished: total 4655 records + hit by time limit :1184
-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=2500
* queue: http://ws0895 >> dropping!
-finishing thread FetcherThread, activeThreads=49
-finishing thread FetcherThread, activeThreads=48
.
.
.
.
-finishing thread FetcherThread, activeThreads=3
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=1
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
        at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1340)
        at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:1376)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:1349)

New pdf's are added to server everynight and the whole of the content then
re-fetched, ie the content is growing so I can understand that a limit
might be reached.

I have searched on the error and it seems that this behaviour should be
governed by fetcher.timelimit.mins property

I've checked the nutch-default and nutch-site files and can only find a
single entry:

<property>
  <name>fetcher.timelimit.mins</name>
  <value>-1</value>
  <description>This is the number of minutes allocated to the fetching.
  Once this value is reached, any remaining entry from the input URL list
is skipped
  and all active queues are emptied. The default value of -1 deactivates
the time limit.
  </description>
</property>

Should there not therefore be  no time limit?

Any suggestions on what else might be causing this problem?

Thanks

P