You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by "Karl Wright (JIRA)" <ji...@apache.org> on 2013/04/24 16:43:15 UTC

[jira] [Created] (CONNECTORS-679) Web connector seems to be able to get stuck waiting on throttling conditions

Karl Wright created CONNECTORS-679:
--------------------------------------

             Summary: Web connector seems to be able to get stuck waiting on throttling conditions
                 Key: CONNECTORS-679
                 URL: https://issues.apache.org/jira/browse/CONNECTORS-679
             Project: ManifoldCF
          Issue Type: Bug
          Components: Web connector
    Affects Versions: ManifoldCF 1.2
            Reporter: Karl Wright
            Assignee: Karl Wright
            Priority: Critical
             Fix For: ManifoldCF 1.2


While crawling, a production system wound up stuck with the following thread dump:

{code}
2013-04-24 16:06:52
Full thread dump Java HotSpot(TM) Server VM (20.13-b02 mixed mode):

"Attach Listener" daemon prio=10 tid=0x09022000 nid=0x2d46 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Thread-78762" daemon prio=10 tid=0xdac6f400 nid=0x5bff in Object.wait() [0xdc7d8000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-76505" daemon prio=10 tid=0x0932e000 nid=0x5304 in Object.wait() [0xdbb18000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-76346" daemon prio=10 tid=0xdc342800 nid=0x5263 in Object.wait() [0xdbbba000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-76181" daemon prio=10 tid=0xde240400 nid=0x51a0 in Object.wait() [0xdb75c000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-76117" daemon prio=10 tid=0xdac01c00 nid=0x5160 in Object.wait() [0xdc4fe000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-76034" daemon prio=10 tid=0xdab05000 nid=0x510d in Object.wait() [0xdc45c000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-75986" daemon prio=10 tid=0xde240c00 nid=0x50dd in Object.wait() [0xdbac7000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-75893" daemon prio=10 tid=0xde2ac000 nid=0x5080 in Object.wait() [0xdbcad000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-75840" daemon prio=10 tid=0xdad54c00 nid=0x504b in Object.wait() [0xdc87a000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-75810" daemon prio=10 tid=0xde23f800 nid=0x502c in Object.wait() [0xdba76000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-75709" daemon prio=10 tid=0xdde00400 nid=0x4fc7 in Object.wait() [0xdc694000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-75629" daemon prio=10 tid=0xdad05000 nid=0x4f77 in Object.wait() [0xdbc0b000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-75607" daemon prio=10 tid=0xdbd21c00 nid=0x4f61 in Object.wait() [0xdb7ad000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-72978" daemon prio=10 tid=0xdad54000 nid=0x447c in Object.wait() [0xdc736000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-72921" daemon prio=10 tid=0x08a6f800 nid=0x4443 in Object.wait() [0xde3eb000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-72847" daemon prio=10 tid=0xdf524000 nid=0x43f9 in Object.wait() [0xdc4ad000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-72771" daemon prio=10 tid=0xdc33a400 nid=0x43ad in Object.wait() [0xdbcfe000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-72650" daemon prio=10 tid=0xdeae7000 nid=0x4334 in Object.wait() [0xdc6e5000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-72571" daemon prio=10 tid=0xdac09c00 nid=0x42e6 in Object.wait() [0xdbb69000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-72315" daemon prio=10 tid=0xdbd22000 nid=0x41e4 in Object.wait() [0xdb5fe000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-72045" daemon prio=10 tid=0xde241c00 nid=0x40d6 in Object.wait() [0xdc787000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-72028" daemon prio=10 tid=0xdab07400 nid=0x40c5 in Object.wait() [0xdb3fe000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-72000" daemon prio=10 tid=0xde2ab000 nid=0x40ae in Object.wait() [0xdc8cb000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-72003" daemon prio=10 tid=0xde25b000 nid=0x40aa in Object.wait() [0xdbc5c000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Thread-72007" daemon prio=10 tid=0xda6fe400 nid=0x40a5 in Object.wait() [0xdba25000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820)
	- locked <0xec11c430> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032)
	at org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537)

"Idle cleanup thread" daemon prio=10 tid=0xde9e8000 nid=0x1922 in Object.wait() [0xdc829000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:956)
	- locked <0xf0e14a30> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.system.IdleCleanupThread.run(IdleCleanupThread.java:70)

"Seeding thread" daemon prio=10 tid=0xde9e5000 nid=0x191f in Object.wait() [0xdc91c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:956)
	- locked <0xf1160a10> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.system.SeedingThread.run(SeedingThread.java:105)

"Job reset thread" daemon prio=10 tid=0xde9e3800 nid=0x191e in Object.wait() [0xdc96d000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:956)
	- locked <0xf1bfd378> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.system.JobResetThread.run(JobResetThread.java:114)

"Document delete thread '9'" daemon prio=10 tid=0xde9e2000 nid=0x191d in Object.wait() [0xdc9be000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe4246448> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
	- locked <0xe4246448> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)

"Document delete thread '8'" daemon prio=10 tid=0xde9e0800 nid=0x191c in Object.wait() [0xdca0f000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe4246448> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
	- locked <0xe4246448> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)

"Document delete thread '7'" daemon prio=10 tid=0xde9df000 nid=0x191b in Object.wait() [0xdca60000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe4246448> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
	- locked <0xe4246448> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)

"Document delete thread '6'" daemon prio=10 tid=0xde9dd800 nid=0x191a in Object.wait() [0xdcab1000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe4246448> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
	- locked <0xe4246448> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)

"Document delete thread '5'" daemon prio=10 tid=0xde9dc400 nid=0x1919 in Object.wait() [0xdcb02000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe4246448> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
	- locked <0xe4246448> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)

"Document delete thread '4'" daemon prio=10 tid=0xde9db800 nid=0x1918 in Object.wait() [0xdcddb000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe4246448> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
	- locked <0xe4246448> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)

"Document delete thread '3'" daemon prio=10 tid=0xde9da000 nid=0x1917 in Object.wait() [0xdcb53000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe4246448> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
	- locked <0xe4246448> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)

"Document delete thread '2'" daemon prio=10 tid=0xde9d8800 nid=0x1916 in Object.wait() [0xdcba4000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe4246448> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
	- locked <0xe4246448> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)

"Document delete thread '1'" daemon prio=10 tid=0xde9d7000 nid=0x1915 in Object.wait() [0xdcbf5000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe4246448> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
	- locked <0xe4246448> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)

"Document delete thread '0'" daemon prio=10 tid=0xde9d5800 nid=0x1914 in Object.wait() [0xdcc46000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe4246448> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
	- locked <0xe4246448> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)

"Document delete stuffer thread" daemon prio=10 tid=0xde9d4000 nid=0x1913 in Object.wait() [0xdcc98000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:956)
	- locked <0xf1e68988> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.system.DocumentDeleteStufferThread.run(DocumentDeleteStufferThread.java:112)

"Document cleanup thread '9'" daemon prio=10 tid=0xde9d2800 nid=0x1912 in Object.wait() [0xdcce8000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe42630f0> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe42630f0> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)

"Document cleanup thread '8'" daemon prio=10 tid=0xde9d1000 nid=0x1911 in Object.wait() [0xdcd39000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe42630f0> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe42630f0> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)

"Document cleanup thread '7'" daemon prio=10 tid=0xde9cf800 nid=0x1910 in Object.wait() [0xdcd8a000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe42630f0> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe42630f0> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)

"Document cleanup thread '6'" daemon prio=10 tid=0xde9cf000 nid=0x190e in Object.wait() [0xdd063000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe42630f0> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe42630f0> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)

"Document cleanup thread '5'" daemon prio=10 tid=0xde9cd800 nid=0x190d in Object.wait() [0xdce2c000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe42630f0> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe42630f0> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)

"Document cleanup thread '4'" daemon prio=10 tid=0xde9cc000 nid=0x190c in Object.wait() [0xdce7d000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe42630f0> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe42630f0> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)

"Document cleanup thread '3'" daemon prio=10 tid=0xde9ca800 nid=0x190b in Object.wait() [0xdcece000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe42630f0> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe42630f0> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)

"Document cleanup thread '2'" daemon prio=10 tid=0xde9c9000 nid=0x190a in Object.wait() [0xdcf1f000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe42630f0> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe42630f0> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)

"Document cleanup thread '1'" daemon prio=10 tid=0xde9c8000 nid=0x1909 in Object.wait() [0xdcf70000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe42630f0> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe42630f0> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)

"Document cleanup thread '0'" daemon prio=10 tid=0xde9c6800 nid=0x1908 in Object.wait() [0xdcfc1000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe42630f0> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe42630f0> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)

"Document cleanup stuffer thread" daemon prio=10 tid=0xde9c5000 nid=0x1907 in Object.wait() [0xdd012000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:956)
	- locked <0xf1db1530> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupStufferThread.run(DocumentCleanupStufferThread.java:114)

"Expiration thread '9'" daemon prio=10 tid=0xde9c3800 nid=0x1905 in Object.wait() [0xdd0b4000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe427ddd8> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe427ddd8> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)

"Expiration thread '8'" daemon prio=10 tid=0xde9c2c00 nid=0x1904 in Object.wait() [0xdd38d000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe427ddd8> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe427ddd8> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)

"Expiration thread '7'" daemon prio=10 tid=0xde9c1400 nid=0x1903 in Object.wait() [0xdd105000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe427ddd8> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe427ddd8> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)

"Expiration thread '6'" daemon prio=10 tid=0xde9bfc00 nid=0x1902 in Object.wait() [0xdd156000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe427ddd8> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe427ddd8> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)

"Expiration thread '5'" daemon prio=10 tid=0xde9be400 nid=0x1901 in Object.wait() [0xdd1a7000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe427ddd8> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe427ddd8> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)

"Expiration thread '4'" daemon prio=10 tid=0xde9bcc00 nid=0x1900 in Object.wait() [0xdd1f8000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe427ddd8> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe427ddd8> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)

"Expiration thread '3'" daemon prio=10 tid=0xde9bb400 nid=0x18ff in Object.wait() [0xdd249000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe427ddd8> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe427ddd8> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)

"Expiration thread '2'" daemon prio=10 tid=0xde9b9c00 nid=0x18fe in Object.wait() [0xdd29a000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe427ddd8> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe427ddd8> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)

"Expiration thread '1'" daemon prio=10 tid=0xde9b8400 nid=0x18fd in Object.wait() [0xdd2eb000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe427ddd8> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe427ddd8> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)

"Expiration thread '0'" daemon prio=10 tid=0xde9b6c00 nid=0x18fc in Object.wait() [0xdd33c000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe427ddd8> (a java.util.ArrayList)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
	- locked <0xe427ddd8> (a java.util.ArrayList)
	at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)

"Worker thread '24'" daemon prio=10 tid=0xde9b5400 nid=0x18fa in Object.wait() [0xdd3de000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Thread.join(Thread.java:1186)
	- locked <0xeca059d8> (a org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread)
	at java.lang.Thread.join(Thread.java:1239)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.finishUp(ThrottledFetcher.java:2694)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1857)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:804)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '23'" daemon prio=10 tid=0xde9b4c00 nid=0x18f9 in Object.wait() [0xdd522000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xec4efee8> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
	- locked <0xec5619a0> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(InputStreamReader.java:167)
	at java.io.Reader.read(Reader.java:123)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.getLimitedResponseBody(ThrottledFetcher.java:1784)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:773)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '22'" daemon prio=10 tid=0xde9b3400 nid=0x18f8 in Object.wait() [0xdd42f000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xec98fc08> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:101)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:747)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '21'" daemon prio=10 tid=0xde9b1c00 nid=0x18f7 in Object.wait() [0xdd480000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xec54a508> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:101)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:747)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '20'" daemon prio=10 tid=0xde9b0400 nid=0x18f6 in Object.wait() [0xdd4d1000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xec5098f8> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:101)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:747)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '19'" daemon prio=10 tid=0xde9aec00 nid=0x18f3 in Object.wait() [0xdd573000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xec4efd58> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:101)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:747)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '18'" daemon prio=10 tid=0xde9ad400 nid=0x18f2 in Object.wait() [0xdd5c4000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xecc99328> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:101)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:747)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '17'" daemon prio=10 tid=0xde9ac800 nid=0x18ef in Object.wait() [0xdd7fb000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xec493980> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
	- locked <0xec562328> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(InputStreamReader.java:167)
	at java.io.Reader.read(Reader.java:123)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.getLimitedResponseBody(ThrottledFetcher.java:1784)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:773)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '16'" daemon prio=10 tid=0xde9a8400 nid=0x18ee in Object.wait() [0xdd615000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Thread.join(Thread.java:1186)
	- locked <0xec9c2998> (a org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread)
	at java.lang.Thread.join(Thread.java:1239)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.finishUp(ThrottledFetcher.java:2694)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1857)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:804)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '15'" daemon prio=10 tid=0xde9a6c00 nid=0x18ed in Object.wait() [0xdd666000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xec9a1ab8> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:101)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:747)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '14'" daemon prio=10 tid=0xde9a5400 nid=0x18ec in Object.wait() [0xdd6b7000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xec4dbe60> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:101)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:747)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '13'" daemon prio=10 tid=0xde9a3c00 nid=0x18eb in Object.wait() [0xdd708000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xec98f2e8> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
	- locked <0xeca81f60> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(InputStreamReader.java:167)
	at java.io.Reader.read(Reader.java:123)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.getLimitedResponseBody(ThrottledFetcher.java:1784)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:773)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '12'" daemon prio=10 tid=0xde9a2400 nid=0x18ea in Object.wait() [0xdd759000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xecc98328> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
	- locked <0xeccd0280> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(InputStreamReader.java:167)
	at java.io.Reader.read(Reader.java:123)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.getLimitedResponseBody(ThrottledFetcher.java:1784)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:773)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '11'" daemon prio=10 tid=0xde9a0c00 nid=0x18e9 in Object.wait() [0xdd7aa000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Thread.join(Thread.java:1186)
	- locked <0xeca169e8> (a org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread)
	at java.lang.Thread.join(Thread.java:1239)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.finishUp(ThrottledFetcher.java:2694)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1857)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:804)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '10'" daemon prio=10 tid=0xde99f400 nid=0x18e7 in Object.wait() [0xdd84c000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Thread.join(Thread.java:1186)
	- locked <0xec15b908> (a org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread)
	at java.lang.Thread.join(Thread.java:1239)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.finishUp(ThrottledFetcher.java:2694)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1857)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:804)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '9'" daemon prio=10 tid=0xde99dc00 nid=0x18e6 in Object.wait() [0xdd89d000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Thread.join(Thread.java:1186)
	- locked <0xec358620> (a org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread)
	at java.lang.Thread.join(Thread.java:1239)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.finishUp(ThrottledFetcher.java:2694)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1857)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:804)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '8'" daemon prio=10 tid=0xde99c400 nid=0x18e5 in Object.wait() [0xdd8ee000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xecd96958> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:101)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:747)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '7'" daemon prio=10 tid=0xde99ac00 nid=0x18e4 in Object.wait() [0xdd93f000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xec9c3ad8> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:101)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:747)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '6'" daemon prio=10 tid=0xde999800 nid=0x18e3 in Object.wait() [0xdd990000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xec9d4ae8> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:101)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:747)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '5'" daemon prio=10 tid=0xde998400 nid=0x18e2 in Object.wait() [0xddd0b000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xec491390> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:101)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:747)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '4'" daemon prio=10 tid=0xde997c00 nid=0x18e1 in Object.wait() [0xdd9e1000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Thread.join(Thread.java:1186)
	- locked <0xec11d450> (a org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread)
	at java.lang.Thread.join(Thread.java:1239)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.finishUp(ThrottledFetcher.java:2694)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1857)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:804)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '3'" daemon prio=10 tid=0xde996000 nid=0x18e0 in Object.wait() [0xdda32000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xec98faa8> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
	- locked <0xec9c3488> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(InputStreamReader.java:167)
	at java.io.Reader.read(Reader.java:123)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.getLimitedResponseBody(ThrottledFetcher.java:1784)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:773)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '2'" daemon prio=10 tid=0xde994800 nid=0x18df in Object.wait() [0xdda83000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xec5a67d0> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
	- locked <0xec5ee1a8> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(InputStreamReader.java:167)
	at java.io.Reader.read(Reader.java:123)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.getLimitedResponseBody(ThrottledFetcher.java:1784)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:773)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '1'" daemon prio=10 tid=0xde993000 nid=0x18de in Object.wait() [0xddad4000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xec9a0ab8> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:101)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:747)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Worker thread '0'" daemon prio=10 tid=0xdeaa3000 nid=0x18dd in Object.wait() [0xddb25000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.manifoldcf.core.common.XThreadInputStream.read(XThreadInputStream.java:160)
	- locked <0xec492980> (a org.apache.manifoldcf.core.common.XThreadInputStream)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:101)
	at org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:747)
	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:322)

"Set priority thread" daemon prio=10 tid=0xdf5bb000 nid=0x18dc in Object.wait() [0xddb76000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:956)
	- locked <0xef4bd708> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.system.SetPriorityThread.run(SetPriorityThread.java:148)

"Expire stuffer thread" daemon prio=10 tid=0x0899e000 nid=0x18db in Object.wait() [0xddbc7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:956)
	- locked <0xf143f768> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.system.ExpireStufferThread.run(ExpireStufferThread.java:134)

"Stuffer thread" daemon prio=10 tid=0x089a5800 nid=0x18d9 in Object.wait() [0xddc18000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:956)
	- locked <0xeeac8468> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.system.StufferThread.run(StufferThread.java:131)

"Job delete thread" daemon prio=10 tid=0x089a4000 nid=0x18d8 in Object.wait() [0xddc6a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:956)
	- locked <0xf1ef3ae8> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.system.JobDeleteThread.run(JobDeleteThread.java:75)

"Job notification thread" daemon prio=10 tid=0x089a2400 nid=0x18d7 in Object.wait() [0xddcba000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:956)
	- locked <0xf1e57150> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.system.JobNotificationThread.run(JobNotificationThread.java:191)

"Finisher thread" daemon prio=10 tid=0x0899cc00 nid=0x18d5 in Object.wait() [0xddd5c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:956)
	- locked <0xf1d3feb8> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.system.FinisherThread.run(FinisherThread.java:66)

"Delete startup thread" daemon prio=10 tid=0x0899bc00 nid=0x18d4 in Object.wait() [0xdddad000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:956)
	- locked <0xf1e57d28> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.system.StartDeleteThread.run(StartDeleteThread.java:89)

"Startup thread" daemon prio=10 tid=0x08994800 nid=0x18d3 in Object.wait() [0xdddfe000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:956)
	- locked <0xf1b0bda0> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.system.StartupThread.run(StartupThread.java:93)

"Job start thread" daemon prio=10 tid=0x0898fc00 nid=0x18d2 in Object.wait() [0xde39a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:956)
	- locked <0xf1c51e08> (a java.lang.Integer)
	at org.apache.manifoldcf.crawler.system.JobStartThread.run(JobStartThread.java:92)

"Connection pool reaper" daemon prio=10 tid=0xdeb7f000 nid=0x18aa waiting on condition [0xde7fe000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.manifoldcf.core.jdbcpool.ConnectionPoolManager$ConnectionCloserThread.run(ConnectionPoolManager.java:136)

"Low Memory Detector" daemon prio=10 tid=0xdf24a000 nid=0x18a8 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0xdf248000 nid=0x18a7 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0xdf246000 nid=0x18a6 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0xdf244800 nid=0x18a5 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0xdf236000 nid=0x18a4 in Object.wait() [0xdf1fe000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
	- locked <0xe4201910> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0xdf234c00 nid=0x18a3 in Object.wait() [0xdf366000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
	- locked <0xe42314c8> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x0896f000 nid=0x1894 in Object.wait() [0xf715d000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:956)
	- locked <0xf1ced040> (a java.lang.Integer)
	at org.apache.manifoldcf.agents.AgentRun.doExecute(AgentRun.java:68)
	at org.apache.manifoldcf.agents.BaseAgentsInitializationCommand.execute(BaseAgentsInitializationCommand.java:37)
	at org.apache.manifoldcf.agents.AgentRun.main(AgentRun.java:101)

"VM Thread" prio=10 tid=0xdf231000 nid=0x18a2 runnable 

"GC task thread#0 (ParallelGC)" prio=10 tid=0x08976400 nid=0x1895 runnable 

"GC task thread#1 (ParallelGC)" prio=10 tid=0x08977800 nid=0x1896 runnable 

"GC task thread#2 (ParallelGC)" prio=10 tid=0x08979000 nid=0x1897 runnable 

"GC task thread#3 (ParallelGC)" prio=10 tid=0x0897a400 nid=0x1898 runnable 

"GC task thread#4 (ParallelGC)" prio=10 tid=0x0897bc00 nid=0x1899 runnable 

"GC task thread#5 (ParallelGC)" prio=10 tid=0x0897d400 nid=0x189a runnable 

"GC task thread#6 (ParallelGC)" prio=10 tid=0x0897e800 nid=0x189b runnable 

"GC task thread#7 (ParallelGC)" prio=10 tid=0x08980000 nid=0x189c runnable 

"GC task thread#8 (ParallelGC)" prio=10 tid=0x08981400 nid=0x189d runnable 

"GC task thread#9 (ParallelGC)" prio=10 tid=0x08982c00 nid=0x189e runnable 

"GC task thread#10 (ParallelGC)" prio=10 tid=0x08984000 nid=0x189f runnable 

"GC task thread#11 (ParallelGC)" prio=10 tid=0x08985800 nid=0x18a0 runnable 

"GC task thread#12 (ParallelGC)" prio=10 tid=0x08987000 nid=0x18a1 runnable 

"VM Periodic Task Thread" prio=10 tid=0xdf254000 nid=0x18a9 waiting on condition 

JNI global references: 1080
{code}

It is clear from this that the throttle code can somehow wind up stuck waiting, with nobody available to wake it up.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira