You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by "Karl Wright (JIRA)" <ji...@apache.org> on 2017/03/08 13:01:37 UTC
[jira] [Comment Edited] (CONNECTORS-1395) Unexpected jobqueue
status - record id 1488898668325, expecting active status, saw 4
[ https://issues.apache.org/jira/browse/CONNECTORS-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901174#comment-15901174 ]
Karl Wright edited comment on CONNECTORS-1395 at 3/8/17 1:01 PM:
-----------------------------------------------------------------
Thread dump is consistent with a lock that hasn't been freed.
{code}
Conversation opened. 11 messages. All messages read.
Skip to content
Using Gmail with screen readers
Search
Gmail
COMPOSE
Labels
Inbox (1,296)
Starred
Important
Sent Mail
Drafts (48)
Circles
Friends (1)
Family (32)
Acquaintances
Following
Colleagues (82)
Personal
Travel
More
Hangouts
More
3 of 7,201
Expand all Print all In new window
Advice on which PostgreSQL to use with ManifoldCF 2.6
Inbox
x
Standen Guy Mar 3 (5 days ago)
Hi Karl, I am currently using MCF 2.0.1 with PostgreSQL 9.3.5 on Windows and ...
Karl Wright Mar 3 (5 days ago)
Hi Guy, A issue with concurrent jobs is known for jobs sharing the same JCIFS...
Standen Guy AttachmentsMar 3 (5 days ago)
Hi Karl, Thanks for coming back so quickly. Unfortunately I wasn’t using a JC...
Karl Wright Mar 3 (5 days ago)
Hi Guy: It is expected that sometimes database deadlock will develop, and the...
Standen Guy Mar 6 (2 days ago)
Hi Karl, Thanks for that I will try version 2.6. Whilst moving to MCF 2.6 I w...
Karl Wright Mar 6 (2 days ago)
Hi Guy, (1) I have no experience with PostgresSQL versions beyond 9.3, but I ...
Standen Guy Mar 6 (2 days ago)
Many Thanks Karl From: Karl Wright [mailto:daddywri@gmail.com] Sent: 06 March...
Standen Guy Guy.Standen@uk.fujitsu.com via manifoldcf.apache.org
Attachments5:11 AM (2 hours ago)
to user
Hi Karl,
I have upgraded to MCF 2.6 on Windows using Zookeeper for synchronisation and PostgreSQL 9.3.16 for the database.
I re-ran the pair of jobs ( 1 web crawl of local intranet site every 15 minutes and 1 crawl of JDBC connected database every 3 minutes) that would previously lock up after a couple of iterations. These seemed to run much better together but after 6 hours running there was a lock up for both these jobs (both indicating they were starting up).
Attempting to restart or abort the jobs through the MCF webapp failed.
After restarting the MCF agents process all sprang into life and the jobs aborted/restarted as expected.
In the MCF log the following error was found
“ERROR 2017-03-08 00:25:30,433 (Worker thread '14') - Exception tossed: Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
at org.apache.manifoldcf.crawler.jobs.JobQueue.updateCompletedRecord(JobQueue.java:1019)
at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentCompletedMultiple(JobManager.java:3271)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:710)
WARN 2017-03-08 00:25:30,449 (Worker thread '23') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:30,449 (Worker thread '24') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:30,464 (Worker thread '9') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:30,464 (Worker thread '0') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:31,900 (Worker thread '11') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:31,900 (Worker thread '29') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:32,867 (Worker thread '10') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:32,867 (Worker thread '2') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:33,335 (Worker thread '8') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:36,642 (Worker thread '20') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:37,422 (Worker thread '21') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:38,280 (Worker thread '22') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:38,280 (Worker thread '3') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:38,280 (Worker thread '5') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:38,826 (Worker thread '28') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:39,045 (Worker thread '13') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:45,425 (Worker thread '4') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:45,425 (Worker thread '15') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:45,425 (Worker thread '17') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:46,392 (Worker thread '25') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:25:46,392 (Worker thread '27') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:26:11,043 (Worker thread '1') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:26:35,817 (Worker thread '19') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:26:35,817 (Worker thread '26') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:26:36,753 (Worker thread '7') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:26:39,248 (Worker thread '6') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:26:39,248 (Worker thread '18') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
WARN 2017-03-08 00:26:43,129 (Worker thread '16') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
FATAL 2017-03-08 00:32:24,819 (Idle cleanup thread) - Error tossed: Can't release lock we don't hold
java.lang.IllegalStateException: Can't release lock we don't hold”
ManifoldCF does not seem to have recovered from this error and the jobs can’t be re-started without restarting ManifoldCF agents.
The manifoldcf, PostgreSQL and Agents JVM stack trace are attached
Best Regards,
Guy
From: Karl Wright [mailto:daddywri@gmail.com]
Sent: 06 March 2017 09:24
3 Attachments
Karl Wright 6:12 AM (1 hour ago)
Hi Guy, See https://issues.apache.org/jira/browse/CONNECTORS-590. When you se...
Karl Wright 6:52 AM (1 hour ago)
Hi Guy, The agents thread dump shows that there's a lock stuck from somewhere...
Karl Wright <da...@gmail.com>
7:31 AM (28 minutes ago)
to user
Hi Guy,
I've now had a look at everything available to me on your issue. I've created CONNECTORS-1395 to track this issue; please attach any new materials to that ticket.
There's an exception trace you didn't include, specifically for this FATAL exception:
FATAL 2017-03-08 00:32:24,819 (Idle cleanup thread) - Error tossed: Can't release lock we don't hold
java.lang.IllegalStateException: Can't release lock we don't hold”
This is very probably the smoking gun for the stuck lock. I'd really like that exception trace from the log if you wouldn't mind.
What has happened is this:
(1) The document fetch cycle runs like this: (a) "stuff" the document (by changing its state to "active", (b) fetch the document, (c) mark the document completed. For a specific document, after it's been fetched, when we're trying to mark the document as being "completed" we do not see the expected "active" status. Instead we see a status of "pending purgatory", which means that either some other thread changed the document's status out from under us, or that document's status was never in fact set to "active" during the stuffing phase. Neither of these is possible given the code paths available, but we can prove it one way or another by turning on "forensic" debugging, as I described above.
(2) Once the failure happens, the job in question should abort. But it seems like that abort does not complete because there's a second problem with lock management (which generates the FATAL message above). This should be readily fixed if I can get that trace.
Thanks,
Karl
Click here to Reply or Forward
2.39 GB (15%) of 15 GB used
Manage
Terms - Privacy
Last account activity: 0 minutes ago
Details
People (2)
Standen Guy
Add to circles
Show details
2017-03-08 08:29:22
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.31-b07 mixed mode):
"Set priority thread-EventThread" #30657286 daemon prio=10 os_prio=2 tid=0x000000006e79b800 nid=0x149c waiting on condition [0x0000000054cee000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f710cc40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
"Set priority thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30657285 daemon prio=10 os_prio=2 tid=0x000000006e79d000 nid=0xd4c runnable [0x000000005830f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000000f710fdc0> (a sun.nio.ch.Util$2)
- locked <0x00000000f710fdb0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f710fc88> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
"Agents thread-EventThread" #30657284 daemon prio=5 os_prio=0 tid=0x0000000056dbb800 nid=0xab4 waiting on condition [0x0000000057dff000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f7112c40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
"Agents thread-SendThread(127.0.0.1:8349)" #30657283 daemon prio=5 os_prio=0 tid=0x000000006b2e3800 nid=0x11e8 runnable [0x000000005f96f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000000f710df30> (a sun.nio.ch.Util$2)
- locked <0x00000000f710df20> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f710dea0> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
"Idle cleanup thread-EventThread" #30657282 daemon prio=5 os_prio=0 tid=0x000000006c1b8000 nid=0x928 waiting on condition [0x000000005f24f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f7113470> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
"Idle cleanup thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30657281 daemon prio=5 os_prio=0 tid=0x000000006c1b7000 nid=0x1040 runnable [0x0000000055f8e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000000f71144d0> (a sun.nio.ch.Util$2)
- locked <0x00000000f71144c0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f7113f90> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
"Finisher thread-EventThread" #30650263 daemon prio=5 os_prio=0 tid=0x000000005768f800 nid=0x163c waiting on condition [0x000000005ef2f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f70e6bb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
"Finisher thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30650262 daemon prio=5 os_prio=0 tid=0x000000005768e800 nid=0x1158 runnable [0x000000005f02e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000000f70e9d30> (a sun.nio.ch.Util$2)
- locked <0x00000000f70e9d20> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f70e9bf8> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
"Job delete thread-EventThread" #30650034 daemon prio=5 os_prio=0 tid=0x000000006b2e4800 nid=0x10e8 waiting on condition [0x000000005f82f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f7098c48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
"Job delete thread-SendThread(127.0.0.1:8349)" #30650033 daemon prio=5 os_prio=0 tid=0x000000006c1b8800 nid=0x8fc runnable [0x00000000566af000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000000f709ba78> (a sun.nio.ch.Util$2)
- locked <0x00000000f709ba68> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f709b9e8> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
"Job reset thread-EventThread" #30650028 daemon prio=5 os_prio=0 tid=0x000000006e79c800 nid=0x17a8 waiting on condition [0x000000005c89e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f709dcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
"Job reset thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30650027 daemon prio=5 os_prio=0 tid=0x000000006e79b000 nid=0xcec runnable [0x000000005c4df000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000000f709e9d8> (a sun.nio.ch.Util$2)
- locked <0x00000000f709e9c8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f709e530> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
"Job reset thread-EventThread" #30649986 daemon prio=5 os_prio=0 tid=0x0000000056f19800 nid=0x1490 waiting on condition [0x000000005f42f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f70a6aa8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
"Job reset thread-SendThread(127.0.0.1:8349)" #30649985 daemon prio=5 os_prio=0 tid=0x0000000056f12800 nid=0xacc runnable [0x000000005619e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000000f70aab20> (a sun.nio.ch.Util$2)
- locked <0x00000000f70aab10> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f70aaa90> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
"Agents thread-EventThread" #30640967 daemon prio=5 os_prio=0 tid=0x000000006c1b6800 nid=0x1120 waiting on condition [0x000000005fafe000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f6fe3c30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
"Agents thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30640966 daemon prio=5 os_prio=0 tid=0x000000006c1b5800 nid=0x12e0 runnable [0x000000005d28f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000000f6fe6db0> (a sun.nio.ch.Util$2)
- locked <0x00000000f6fe6da0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f6fe6c78> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
"Set priority thread-EventThread" #30595106 daemon prio=10 os_prio=2 tid=0x0000000056c28800 nid=0x15f8 waiting on condition [0x000000005fc0f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f63607e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
"Set priority thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30595105 daemon prio=10 os_prio=2 tid=0x0000000056c28000 nid=0x12f0 runnable [0x000000005fd8f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000000f6362598> (a sun.nio.ch.Util$2)
- locked <0x00000000f6362588> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f6362318> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
"Idle cleanup thread-EventThread" #30594300 daemon prio=5 os_prio=0 tid=0x000000006b2e1800 nid=0x10c8 waiting on condition [0x000000005fedf000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f6367fa0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
"Idle cleanup thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30594299 daemon prio=5 os_prio=0 tid=0x000000006b2e2000 nid=0x1540 runnable [0x000000005f68f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000000f6368a58> (a sun.nio.ch.Util$2)
- locked <0x00000000f6368a48> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f6368740> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
"Expire stuffer thread-EventThread" #30594028 daemon prio=6 os_prio=0 tid=0x00000000577bb000 nid=0x157c waiting on condition [0x000000005eddf000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f6369b98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
"Expire stuffer thread-SendThread(127.0.0.1:8349)" #30594027 daemon prio=6 os_prio=0 tid=0x00000000550c9000 nid=0x118c runnable [0x000000005804f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000000f63786f0> (a sun.nio.ch.Util$2)
- locked <0x00000000f63786e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f6378398> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
"Document cleanup stuffer thread-EventThread" #30594022 daemon prio=5 os_prio=0 tid=0x0000000056f13000 nid=0x1634 waiting on condition [0x000000005ec0e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f637e0a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
"Document cleanup stuffer thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30594021 daemon prio=5 os_prio=0 tid=0x0000000056f18800 nid=0x1488 runnable [0x000000005646e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000000f637b120> (a sun.nio.ch.Util$2)
- locked <0x00000000f637b110> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f637b090> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
"Connection pool reaper" #2098506 daemon prio=5 os_prio=0 tid=0x000000006e79a000 nid=0xf58 waiting on condition [0x000000006cf3f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.manifoldcf.core.jdbcpool.ConnectionPoolManager$ConnectionCloserThread.run(ConnectionPoolManager.java:152)
"Assessment thread" #138 daemon prio=5 os_prio=0 tid=0x0000000056fa6800 nid=0xd90 in Object.wait() [0x000000005e2ef000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
- locked <0x00000000ff522940> (a java.lang.Integer)
at org.apache.manifoldcf.crawler.system.AssessmentThread.run(AssessmentThread.java:68)
"Idle cleanup thread" #137 daemon prio=5 os_prio=0 tid=0x0000000056fa5800 nid=0xc94 in Object.wait() [0x000000005e91f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
- locked <0x00000000ff575968> (a java.lang.Integer)
at org.apache.manifoldcf.crawler.system.IdleCleanupThread.run(IdleCleanupThread.java:74)
"Seeding thread" #136 daemon prio=5 os_prio=0 tid=0x0000000056fa5000 nid=0xd94 in Object.wait() [0x000000005e6ee000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
- locked <0x00000000feea03e0> (a java.lang.Integer)
at org.apache.manifoldcf.crawler.system.SeedingThread.run(SeedingThread.java:104)
"Job reset thread" #135 daemon prio=5 os_prio=0 tid=0x0000000056fa4000 nid=0x810 in Object.wait() [0x000000005dedf000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
- locked <0x00000000ff3cd530> (a java.lang.Integer)
at org.apache.manifoldcf.crawler.system.JobResetThread.run(JobResetThread.java:126)
"Document delete thread '9'" #123 daemon prio=5 os_prio=0 tid=0x0000000056fa3800 nid=0x91c in Object.wait() [0x000000005e5df000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0216580> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0x00000000f0216580> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)
"Document delete thread '8'" #122 daemon prio=5 os_prio=0 tid=0x0000000056fa2800 nid=0xd74 in Object.wait() [0x000000005e42f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0216580> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0x00000000f0216580> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)
"Document delete thread '7'" #121 daemon prio=5 os_prio=0 tid=0x0000000056fa2000 nid=0xd80 in Object.wait() [0x000000005856f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0216580> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0x00000000f0216580> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)
"Document delete thread '6'" #120 daemon prio=5 os_prio=0 tid=0x0000000056fa1000 nid=0xe4c in Object.wait() [0x000000005e1cf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0216580> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0x00000000f0216580> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)
"Document delete thread '5'" #119 daemon prio=5 os_prio=0 tid=0x0000000056fa0800 nid=0x988 in Object.wait() [0x000000005dfde000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0216580> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0x00000000f0216580> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)
"Document delete thread '4'" #118 daemon prio=5 os_prio=0 tid=0x0000000056f92000 nid=0x4e0 in Object.wait() [0x000000005ddbe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0216580> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0x00000000f0216580> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)
"Document delete thread '3'" #117 daemon prio=5 os_prio=0 tid=0x0000000056f91800 nid=0xf9c in Object.wait() [0x000000005d08f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0216580> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0x00000000f0216580> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)
"Document delete thread '2'" #116 daemon prio=5 os_prio=0 tid=0x0000000056f90800 nid=0xf88 in Object.wait() [0x000000005dc6f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0216580> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0x00000000f0216580> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)
"Document delete thread '1'" #115 daemon prio=5 os_prio=0 tid=0x0000000056f90000 nid=0x2dc in Object.wait() [0x000000005da7f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0216580> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0x00000000f0216580> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)
"Document delete thread '0'" #114 daemon prio=5 os_prio=0 tid=0x0000000056f8f000 nid=0xc74 in Object.wait() [0x000000005d94e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0216580> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0x00000000f0216580> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)
"Document delete stuffer thread" #113 daemon prio=5 os_prio=0 tid=0x0000000056f8e800 nid=0xc48 in Object.wait() [0x000000005ae8f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.core.lockmanager.LockGate.waitForPermission(LockGate.java:136)
- locked <0x00000000f627b588> (a org.apache.manifoldcf.core.lockmanager.LockGate)
at org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:258)
at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enter(BaseLockManager.java:1522)
at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enterLocks(BaseLockManager.java:803)
at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCache(CacheManager.java:280)
at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:100)
at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:862)
at org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:236)
at org.apache.manifoldcf.crawler.jobs.Jobs.deletingJobsPresent(Jobs.java:3133)
at org.apache.manifoldcf.crawler.jobs.JobManager.getNextDeletableDocuments(JobManager.java:1862)
at org.apache.manifoldcf.crawler.system.DocumentDeleteStufferThread.run(DocumentDeleteStufferThread.java:108)
"Document cleanup thread '9'" #134 daemon prio=5 os_prio=0 tid=0x0000000056f8d800 nid=0x8dc in Object.wait() [0x000000005d7ef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f025aae8> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f025aae8> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)
"Document cleanup thread '8'" #133 daemon prio=5 os_prio=0 tid=0x0000000056f8c800 nid=0xe04 in Object.wait() [0x000000005d6cf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f025aae8> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f025aae8> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)
"Document cleanup thread '7'" #132 daemon prio=5 os_prio=0 tid=0x0000000056f8c000 nid=0x4b4 in Object.wait() [0x000000005d4fe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f025aae8> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f025aae8> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)
"Document cleanup thread '6'" #131 daemon prio=5 os_prio=0 tid=0x0000000056f8b000 nid=0xd38 in Object.wait() [0x000000005d39e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f025aae8> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)
"Document cleanup thread '5'" #130 daemon prio=5 os_prio=0 tid=0x0000000056f3d800 nid=0xec8 in Object.wait() [0x000000005d18f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f025aae8> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f025aae8> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)
"Document cleanup thread '4'" #129 daemon prio=5 os_prio=0 tid=0x0000000056f3d000 nid=0xee8 in Object.wait() [0x000000005cf1f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f025aae8> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f025aae8> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)
"Document cleanup thread '3'" #128 daemon prio=5 os_prio=0 tid=0x0000000056f3c000 nid=0x9d8 in Object.wait() [0x000000005cd8e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f025aae8> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f025aae8> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)
"Document cleanup thread '2'" #127 daemon prio=5 os_prio=0 tid=0x0000000056f3b800 nid=0x2b8 in Object.wait() [0x000000005b7ce000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f025aae8> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f025aae8> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)
"Document cleanup thread '1'" #126 daemon prio=5 os_prio=0 tid=0x0000000056f3a800 nid=0xf84 in Object.wait() [0x000000005cbef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f025aae8> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f025aae8> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)
"Document cleanup thread '0'" #125 daemon prio=5 os_prio=0 tid=0x0000000056f3a000 nid=0xf4c in Object.wait() [0x000000005c78f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f025aae8> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f025aae8> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)
"Document cleanup stuffer thread" #124 daemon prio=5 os_prio=0 tid=0x0000000056f39000 nid=0xf98 in Object.wait() [0x000000005ca6e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.core.lockmanager.LockGate.waitForPermission(LockGate.java:136)
- locked <0x00000000f627b588> (a org.apache.manifoldcf.core.lockmanager.LockGate)
at org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:258)
at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enter(BaseLockManager.java:1522)
at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enterLocks(BaseLockManager.java:803)
at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCache(CacheManager.java:280)
at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:100)
at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:862)
at org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:236)
at org.apache.manifoldcf.crawler.jobs.Jobs.cleaningJobsPresent(Jobs.java:3150)
at org.apache.manifoldcf.crawler.jobs.JobManager.getNextCleanableDocuments(JobManager.java:1618)
at org.apache.manifoldcf.crawler.system.DocumentCleanupStufferThread.run(DocumentCleanupStufferThread.java:108)
"Expiration thread '9'" #112 daemon prio=5 os_prio=0 tid=0x0000000056f37000 nid=0xc0c in Object.wait() [0x000000005b9de000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0275528> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f0275528> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)
"Expiration thread '8'" #111 daemon prio=5 os_prio=0 tid=0x0000000056c72000 nid=0x134 in Object.wait() [0x000000005c64e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0275528> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f0275528> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)
"Expiration thread '7'" #110 daemon prio=5 os_prio=0 tid=0x0000000056c71800 nid=0xff8 in Object.wait() [0x000000005c06f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0275528> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f0275528> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)
"Expiration thread '6'" #109 daemon prio=5 os_prio=0 tid=0x0000000056c70800 nid=0x9f0 in Object.wait() [0x000000005c3ce000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0275528> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f0275528> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)
"Expiration thread '5'" #108 daemon prio=5 os_prio=0 tid=0x0000000056c70000 nid=0x498 in Object.wait() [0x000000005c2cf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0275528> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f0275528> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)
"Expiration thread '4'" #107 daemon prio=5 os_prio=0 tid=0x0000000056c6f000 nid=0x568 in Object.wait() [0x000000005c1bf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0275528> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f0275528> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)
"Expiration thread '3'" #106 daemon prio=5 os_prio=0 tid=0x0000000056c6e800 nid=0xd84 in Object.wait() [0x000000005bf4f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0275528> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f0275528> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)
"Expiration thread '2'" #105 daemon prio=5 os_prio=0 tid=0x0000000056c6d800 nid=0x244 in Object.wait() [0x000000005be0f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0275528> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f0275528> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)
"Expiration thread '1'" #104 daemon prio=5 os_prio=0 tid=0x0000000056c6d000 nid=0xe38 in Object.wait() [0x000000005bc3f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0275528> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f0275528> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)
"Expiration thread '0'" #103 daemon prio=5 os_prio=0 tid=0x0000000056c6c000 nid=0xff4 in Object.wait() [0x000000005bade000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0275528> (a java.util.ArrayList)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
- locked <0x00000000f0275528> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)
"Worker thread '29'" #102 daemon prio=5 os_prio=0 tid=0x0000000056c6b800 nid=0xf48 in Object.wait() [0x000000005b8df000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '28'" #101 daemon prio=5 os_prio=0 tid=0x0000000056bf0000 nid=0x9b4 in Object.wait() [0x000000005b6cf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '27'" #100 daemon prio=5 os_prio=0 tid=0x0000000056bef000 nid=0x5fc in Object.wait() [0x0000000059d8f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '26'" #99 daemon prio=5 os_prio=0 tid=0x0000000056bee800 nid=0x2a4 in Object.wait() [0x000000005b4fe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '25'" #98 daemon prio=5 os_prio=0 tid=0x0000000056bed800 nid=0xa20 in Object.wait() [0x000000005b3be000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '24'" #97 daemon prio=5 os_prio=0 tid=0x0000000056bed000 nid=0xa90 in Object.wait() [0x000000005b28e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '23'" #96 daemon prio=5 os_prio=0 tid=0x0000000056bec000 nid=0xe90 in Object.wait() [0x000000005b12f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '22'" #95 daemon prio=5 os_prio=0 tid=0x0000000056beb800 nid=0x998 in Object.wait() [0x000000005af8f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '21'" #94 daemon prio=5 os_prio=0 tid=0x0000000056bea800 nid=0x80c in Object.wait() [0x000000005ad8e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '20'" #93 daemon prio=5 os_prio=0 tid=0x0000000056bea000 nid=0x2e0 in Object.wait() [0x000000005ac0e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '19'" #92 daemon prio=5 os_prio=0 tid=0x0000000056be9000 nid=0x2ec in Object.wait() [0x000000005a1bf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '18'" #91 daemon prio=5 os_prio=0 tid=0x0000000056bde000 nid=0xd54 in Object.wait() [0x000000005ab0f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '17'" #90 daemon prio=5 os_prio=0 tid=0x0000000056bdd000 nid=0x5a4 in Object.wait() [0x000000005a95f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '16'" #89 daemon prio=5 os_prio=0 tid=0x0000000056bdc800 nid=0xc98 in Object.wait() [0x000000005a79f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '15'" #88 daemon prio=5 os_prio=0 tid=0x0000000056bdb800 nid=0xb18 in Object.wait() [0x000000005a68e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '14'" #87 daemon prio=5 os_prio=0 tid=0x0000000056bdb000 nid=0x8bc in Object.wait() [0x000000005a4fe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '13'" #86 daemon prio=5 os_prio=0 tid=0x0000000056bda000 nid=0x8d8 in Object.wait() [0x0000000058b3e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '12'" #85 daemon prio=5 os_prio=0 tid=0x0000000056bd9800 nid=0x668 in Object.wait() [0x000000005a30e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '11'" #84 daemon prio=5 os_prio=0 tid=0x0000000056bd8800 nid=0xfcc in Object.wait() [0x000000005a0be000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '10'" #83 daemon prio=5 os_prio=0 tid=0x0000000056bd8000 nid=0xef0 in Object.wait() [0x0000000059fae000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '9'" #82 daemon prio=5 os_prio=0 tid=0x0000000056bd7000 nid=0x3f0 in Object.wait() [0x0000000059eae000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '8'" #81 daemon prio=5 os_prio=0 tid=0x0000000056b3f800 nid=0x290 in Object.wait() [0x0000000059c6f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '7'" #80 daemon prio=5 os_prio=0 tid=0x0000000056b3e800 nid=0x39c in Object.wait() [0x0000000057f3e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '6'" #79 daemon prio=5 os_prio=0 tid=0x0000000056b3e000 nid=0x924 in Object.wait() [0x0000000059a9f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '5'" #78 daemon prio=5 os_prio=0 tid=0x0000000056b3d000 nid=0xd60 in Object.wait() [0x00000000598cf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '4'" #77 daemon prio=5 os_prio=0 tid=0x0000000056b3c000 nid=0x938 in Object.wait() [0x00000000597cf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '3'" #76 daemon prio=5 os_prio=0 tid=0x0000000056b3b800 nid=0xe7c in Object.wait() [0x00000000596af000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '2'" #75 daemon prio=5 os_prio=0 tid=0x0000000056b3a800 nid=0xf30 in Object.wait() [0x000000005952e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '1'" #74 daemon prio=5 os_prio=0 tid=0x0000000056b3a000 nid=0x12c in Object.wait() [0x000000005935f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"Worker thread '0'" #73 daemon prio=5 os_prio=0 tid=0x0000000056b39000 nid=0x4f4 in Object.wait() [0x000000005919f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
- locked <0x00000000f027d248> (a java.util.ArrayList)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)
"History cleanup thread" #72 daemon prio=5 os_prio=0 tid=0x0000000056b38800 nid=0xec4 in Object.wait() [0x00000000588ff000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
- locked <0x00000000ffe18000> (a java.lang.Integer)
at org.apache.manifoldcf.crawler.system.HistoryCleanupThread.run(HistoryCleanupThread.java:78)
"Set priority thread" #71 daemon prio=10 os_prio=2 tid=0x0000000056abb000 nid=0x6bc in Object.wait() [0x0000000058fce000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
- locked <0x00000000ff4abfe8> (a java.lang.Integer)
at org.apache.manifoldcf.crawler.system.SetPriorityThread.run(SetPriorityThread.java:148)
"Expire stuffer thread" #70 daemon prio=6 os_prio=0 tid=0x0000000056aba800 nid=0xdf8 in Object.wait() [0x0000000058ece000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.core.lockmanager.LockGate.waitForPermission(LockGate.java:136)
- locked <0x00000000f627b588> (a org.apache.manifoldcf.core.lockmanager.LockGate)
at org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:258)
at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enter(BaseLockManager.java:1522)
at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enterLocks(BaseLockManager.java:803)
at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCache(CacheManager.java:280)
at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:100)
at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:862)
at org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:236)
at org.apache.manifoldcf.crawler.jobs.Jobs.activeJobsPresent(Jobs.java:3171)
at org.apache.manifoldcf.crawler.jobs.JobManager.getExpiredDocuments(JobManager.java:2441)
at org.apache.manifoldcf.crawler.system.ExpireStufferThread.run(ExpireStufferThread.java:121)
"Stuffer thread" #69 daemon prio=6 os_prio=0 tid=0x0000000056ab9800 nid=0x990 in Object.wait() [0x0000000058d6e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.core.lockmanager.LockGate.waitForPermission(LockGate.java:136)
- locked <0x00000000f627b588> (a org.apache.manifoldcf.core.lockmanager.LockGate)
at org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:258)
at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enter(BaseLockManager.java:1522)
at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enterLocks(BaseLockManager.java:803)
at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCache(CacheManager.java:280)
at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:100)
at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:862)
at org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:236)
at org.apache.manifoldcf.crawler.jobs.Jobs.activeJobsPresent(Jobs.java:3171)
at org.apache.manifoldcf.crawler.jobs.JobManager.getNextDocuments(JobManager.java:2698)
at org.apache.manifoldcf.crawler.system.StufferThread.run(StufferThread.java:186)
"Job delete thread" #68 daemon prio=5 os_prio=0 tid=0x0000000056ab9000 nid=0x270 in Object.wait() [0x0000000058c5f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
- locked <0x00000000ff3f2de0> (a java.lang.Integer)
at org.apache.manifoldcf.crawler.system.JobDeleteThread.run(JobDeleteThread.java:78)
"Job notification thread" #67 daemon prio=5 os_prio=0 tid=0x0000000056ab8000 nid=0x820 in Object.wait() [0x0000000058a3f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
- locked <0x00000000ff5d45b8> (a java.lang.Integer)
at org.apache.manifoldcf.crawler.system.JobNotificationThread.run(JobNotificationThread.java:431)
"Finisher thread" #66 daemon prio=5 os_prio=0 tid=0x0000000056ab7800 nid=0xda4 in Object.wait() [0x00000000587ff000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
- locked <0x00000000ff451868> (a java.lang.Integer)
at org.apache.manifoldcf.crawler.system.FinisherThread.run(FinisherThread.java:69)
"Delete startup thread" #65 daemon prio=5 os_prio=0 tid=0x0000000056ab6800 nid=0x580 in Object.wait() [0x000000005867f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
- locked <0x00000000ff50fe98> (a java.lang.Integer)
at org.apache.manifoldcf.crawler.system.StartDeleteThread.run(StartDeleteThread.java:93)
"Startup thread" #64 daemon prio=5 os_prio=0 tid=0x0000000056ab6000 nid=0xda0 in Object.wait() [0x000000005846e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at org.apache.manifoldcf.core.lockmanager.LockObject.enterWriteLock(LockObject.java:80)
- locked <0x00000000f627b5f8> (a org.apache.manifoldcf.core.lockmanager.ZooKeeperLockObject)
at org.apache.manifoldcf.core.lockmanager.LockGate.enterWriteLock(LockGate.java:177)
at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enter(BaseLockManager.java:1473)
at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enterLocks(BaseLockManager.java:803)
at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCache(CacheManager.java:299)
at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:100)
at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:661)
at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performUpdate(DBInterfacePostgreSQL.java:254)
at org.apache.manifoldcf.core.database.BaseTable.performUpdate(BaseTable.java:80)
at org.apache.manifoldcf.crawler.jobs.Jobs.noteJobStarted(Jobs.java:2322)
at org.apache.manifoldcf.crawler.jobs.JobManager.noteJobStarted(JobManager.java:6508)
at org.apache.manifoldcf.crawler.system.StartupThread.run(StartupThread.java:207)
"Job start thread" #63 daemon prio=5 os_prio=0 tid=0x0000000056ab5000 nid=0xc60 in Object.wait() [0x00000000581df000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
- locked <0x00000000ff639398> (a java.lang.Integer)
at org.apache.manifoldcf.crawler.system.JobStartThread.run(JobStartThread.java:96)
"Connection pool reaper" #17 daemon prio=5 os_prio=0 tid=0x0000000056aa0000 nid=0xa94 waiting on condition [0x00000000578df000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.manifoldcf.core.jdbcpool.ConnectionPoolManager$ConnectionCloserThread.run(ConnectionPoolManager.java:152)
"Agents thread" #16 daemon prio=5 os_prio=0 tid=0x0000000056bd2800 nid=0xf3c in Object.wait() [0x0000000057aff000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
- locked <0x00000000ff4ddce8> (a java.lang.Integer)
at org.apache.manifoldcf.agents.system.AgentsDaemon$AgentsThread.run(AgentsDaemon.java:210)
"Idle cleanup thread" #15 daemon prio=5 os_prio=0 tid=0x0000000056bd1000 nid=0xe0c in Object.wait() [0x00000000579df000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
- locked <0x00000000ff58b430> (a java.lang.Integer)
at org.apache.manifoldcf.agents.system.IdleCleanupThread.run(IdleCleanupThread.java:96)
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x000000005506c800 nid=0xd2c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #8 daemon prio=9 os_prio=2 tid=0x0000000054fe8000 nid=0xcbc waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #7 daemon prio=9 os_prio=2 tid=0x0000000054fe0800 nid=0x90c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x000000005352a000 nid=0xdb8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x0000000053529000 nid=0xc20 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x0000000053526000 nid=0xdc4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x00000000534c3800 nid=0xef4 in Object.wait() [0x00000000549bf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
- locked <0x00000000f0056410> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x00000000534ba000 nid=0xd68 in Object.wait() [0x000000005484f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
- locked <0x00000000f0171988> (a java.lang.ref.Reference$Lock)
"main" #1 prio=5 os_prio=0 tid=0x000000000231f000 nid=0xcf8 in Object.wait() [0x000000000253f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
- locked <0x00000000ff556a88> (a java.lang.Integer)
at org.apache.manifoldcf.agents.system.AgentsDaemon.runAgents(AgentsDaemon.java:115)
at org.apache.manifoldcf.agents.AgentRun.doExecute(AgentRun.java:64)
at org.apache.manifoldcf.agents.BaseAgentsInitializationCommand.execute(BaseAgentsInitializationCommand.java:37)
at org.apache.manifoldcf.agents.AgentRun.main(AgentRun.java:93)
"VM Thread" os_prio=2 tid=0x00000000534b6000 nid=0xc44 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x000000000218d000 nid=0x1e8 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x000000000218e800 nid=0xdb0 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000000002190000 nid=0x900 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000000002193800 nid=0xca0 runnable
"VM Periodic Task Thread" os_prio=2 tid=0x000000005506f800 nid=0xe74 waiting on condition
JNI global references: 269
Heap
PSYoungGen total 86016K, used 77935K [0x00000000fab00000, 0x0000000100000000, 0x0000000100000000)
eden space 84992K, 91% used [0x00000000fab00000,0x00000000ff6a5b20,0x00000000ffe00000)
from space 1024K, 62% used [0x00000000ffe00000,0x00000000ffea0000,0x00000000fff00000)
to space 1024K, 0% used [0x00000000fff00000,0x00000000fff00000,0x0000000100000000)
ParOldGen total 175104K, used 116435K [0x00000000f0000000, 0x00000000fab00000, 0x00000000fab00000)
object space 175104K, 66% used [0x00000000f0000000,0x00000000f71b4c90,0x00000000fab00000)
Metaspace used 19235K, capacity 19524K, committed 20096K, reserved 1067008K
class space used 1939K, capacity 2034K, committed 2176K, reserved 1048576K
{code}
was (Author: kwright@metacarta.com):
Thread dump is consistent with a lock that hasn't been freed.
> Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
> ------------------------------------------------------------------------------------
>
> Key: CONNECTORS-1395
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1395
> Project: ManifoldCF
> Issue Type: Bug
> Components: Framework core
> Affects Versions: ManifoldCF 2.6
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 2.7
>
>
> User saw this in the log, after which the system hung:
> {code}
> ERROR 2017-03-08 00:25:30,433 (Worker thread '14') - Exception tossed: Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
> at org.apache.manifoldcf.crawler.jobs.JobQueue.updateCompletedRecord(JobQueue.java:1019)
> at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentCompletedMultiple(JobManager.java:3271)
> at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:710)
> WARN 2017-03-08 00:25:30,449 (Worker thread '23') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:30,449 (Worker thread '24') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:30,464 (Worker thread '9') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:30,464 (Worker thread '0') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:31,900 (Worker thread '11') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:31,900 (Worker thread '29') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:32,867 (Worker thread '10') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:32,867 (Worker thread '2') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:33,335 (Worker thread '8') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:36,642 (Worker thread '20') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:37,422 (Worker thread '21') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:38,280 (Worker thread '22') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:38,280 (Worker thread '3') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:38,280 (Worker thread '5') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:38,826 (Worker thread '28') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:39,045 (Worker thread '13') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:45,425 (Worker thread '4') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:45,425 (Worker thread '15') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:45,425 (Worker thread '17') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:46,392 (Worker thread '25') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:46,392 (Worker thread '27') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:11,043 (Worker thread '1') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:35,817 (Worker thread '19') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:35,817 (Worker thread '26') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:36,753 (Worker thread '7') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:39,248 (Worker thread '6') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:39,248 (Worker thread '18') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:43,129 (Worker thread '16') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> FATAL 2017-03-08 00:32:24,819 (Idle cleanup thread) - Error tossed: Can't release lock we don't hold
> java.lang.IllegalStateException: Can't release lock we don't hold”
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)