You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by Erlend Garåsen <e....@usit.uio.no> on 2013/01/25 11:26:15 UTC
max_pred_locks_per_transaction
After we started to crawl journals, the crawler just stopped after a
couple of hours. Running MCF in debug mode gave me the stack trace shown
below. I thing we need to adjust some PG parameters, perhaps
max_pred_locks_per_transaction. The database admins are now asking me
about which value to set. They have increased it, but I don't know
whether it is sufficient.
Another thing. I did not see this message until I changed the log level
to debug, but the log4j should catch this error messages with warn level
enabled. So maybe it is a dead end, i.e. a totally different cause and
that this just occurred as a coincidence.
ERROR 2013-01-19 03:47:15,049 (Worker thread '49') - Worker thread
aborting and restarting due to database connection reset: Database
exception: Exception doing query: ERROR: out of shared memory
Hint: You might need to increase max_pred_locks_per_transaction.
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
exception: Exception doing query: ERROR: out of shared memory
Hint: You might need to increase max_pred_locks_per_transaction.
at
org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:681)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
at
org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4089)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:1932)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.flush(WorkerThread.java:1863)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:554)
Caused by: org.postgresql.util.PSQLException: ERROR: out of shared memory
Hint: You might need to increase max_pred_locks_per_transaction.
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
at org.apache.manifoldcf.core.database.Database.execute(Database.java:826)
at
org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641)
DEBUG 2013-01-19 03:47:23,386 (Idle cleanup thread) - Checking for
connections, idleTimeout: 1358563583386
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Erlend Garåsen <e....@usit.uio.no>.
Sorry for not following up with a thread dump shortly after, but I will
if we still need one. I'm looking forward to try out the latest version
from trunk tomorrow morning. It will be the first thing I'll do. Thanks
for looking at this issue!
Erlend
On 07.02.13 19.57, Karl Wright wrote:
> I think I found the issue.
>
> Please sync up and try again.
> Karl
>
>
> On Thu, Feb 7, 2013 at 9:43 AM, Karl Wright <da...@gmail.com> wrote:
>> It is not necessary to wait until the job ends; each time it runs out
>> of handles it prints the entire set of allocated handles. You can
>> stop the job now.
>>
>> Karl
>>
>> On Thu, Feb 7, 2013 at 9:41 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>>>
>>> I will upload the log to Jira or place it on web as soon as the job ends.
>>>
>>> Erlend
>>>
>>>
>>> On 07.02.13 15.34, Karl Wright wrote:
>>>>
>>>> FWIW, the way this diagnostic code works is that it simply dumps ALL
>>>> the handles that are currently considered to be in use. Some of them
>>>> are leaked, and some are just being used. So it's really the sum
>>>> total of all of them that will help us narrow down where leaks are
>>>> occurring.
>>>>
>>>> I looked at the one location you provided; it's entirely unremarkable;
>>>> it's not even happening in a transaction. If similar locations show
>>>> up again and again, it will mean that the problem must be in the part
>>>> of the code that allocates connections at the lowest level. Before I
>>>> start picking that code apart line by line, though, I'd like all the
>>>> data.
>>>>
>>>> Karl
>>>>
>>>> On Thu, Feb 7, 2013 at 9:07 AM, Karl Wright <da...@gmail.com> wrote:
>>>>>
>>>>> Thanks, this is great.
>>>>>
>>>>> Could you include ALL of the 'possibly leaked' connections? I'm
>>>>> looking for the ones that show up over and over again. Please attach
>>>>> that part of the log to the ticket.
>>>>>
>>>>> Karl
>>>>>
>>>>> On Thu, Feb 7, 2013 at 9:04 AM, Erlend Garåsen <e....@usit.uio.no>
>>>>> wrote:
>>>>>>
>>>>>> On 06.02.13 15.24, Karl Wright wrote:
>>>>>>
>>>>>>> FWIW, schema creation has been working fine here, and all tests that
>>>>>>> exercise the schema have been passing fine as well.
>>>>>>
>>>>>>
>>>>>>
>>>>>> My fault, but everything is successfully set up now.
>>>>>>
>>>>>> I think I have reproduced the issue. The crawl is still running, but I
>>>>>> guess
>>>>>> it will stop very soon due to the following entries in my log. The last
>>>>>> fetched URL before the first of these log entries below was:
>>>>>>
>>>>>> https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid
>>>>>>
>>>>>> It seems that this URL is part of a search, so maybe we will get around
>>>>>> this
>>>>>> problem by filtering out all search-related URLs. The problem is that it
>>>>>> is
>>>>>> already filtered out, but by doing post filtering. I can try to adjust
>>>>>> that
>>>>>> at next attempt.
>>>>>>
>>>>>> INFO 2013-02-07 14:32:08,166 (Worker thread '13') - WEB: FETCH
>>>>>>
>>>>>> URL|https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid|1360243927960+204|200|24992|
>>>>>> WARN 2013-02-07 14:32:08,228 (Worker thread '16') - Out of db
>>>>>> connections,
>>>>>> list of outstanding ones follows.
>>>>>> WARN 2013-02-07 14:32:08,229 (Worker thread '16') - Found a possibly
>>>>>> leaked
>>>>>> db connection
>>>>>> java.lang.Exception: Possibly leaked db connection
>>>>>> at
>>>>>>
>>>>>> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:72)
>>>>>>
>>>>>> at
>>>>>>
>>>>>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>>>>>> at
>>>>>>
>>>>>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>>>>>> at
>>>>>>
>>>>>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:758)
>>>>>> at
>>>>>>
>>>>>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1430)
>>>>>>
>>>>>> at
>>>>>>
>>>>>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>>>>>> at
>>>>>>
>>>>>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>>>>>> at
>>>>>>
>>>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
>>>>>> at
>>>>>>
>>>>>> org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
>>>>>> at
>>>>>>
>>>>>> org.apache.manifoldcf.crawler.jobs.HopCount.processFind(HopCount.java:674)
>>>>>> at
>>>>>>
>>>>>> org.apache.manifoldcf.crawler.jobs.HopCount.findHopCounts(HopCount.java:649)
>>>>>> at
>>>>>>
>>>>>> org.apache.manifoldcf.crawler.jobs.JobManager.findHopCounts(JobManager.java:3777)
>>>>>> at
>>>>>>
>>>>>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:224)
>>>>>> WARN 2013-02-07 14:32:08,233 (Worker thread '16') - Found a possibly
>>>>>> leaked
>>>>>> db connection
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Erlend Garåsen
>>>>>> Center for Information Technology Services
>>>>>> University of Oslo
>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>> 31050
>>>
>>>
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Karl Wright <da...@gmail.com>.
I think I found the issue.
Please sync up and try again.
Karl
On Thu, Feb 7, 2013 at 9:43 AM, Karl Wright <da...@gmail.com> wrote:
> It is not necessary to wait until the job ends; each time it runs out
> of handles it prints the entire set of allocated handles. You can
> stop the job now.
>
> Karl
>
> On Thu, Feb 7, 2013 at 9:41 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>>
>> I will upload the log to Jira or place it on web as soon as the job ends.
>>
>> Erlend
>>
>>
>> On 07.02.13 15.34, Karl Wright wrote:
>>>
>>> FWIW, the way this diagnostic code works is that it simply dumps ALL
>>> the handles that are currently considered to be in use. Some of them
>>> are leaked, and some are just being used. So it's really the sum
>>> total of all of them that will help us narrow down where leaks are
>>> occurring.
>>>
>>> I looked at the one location you provided; it's entirely unremarkable;
>>> it's not even happening in a transaction. If similar locations show
>>> up again and again, it will mean that the problem must be in the part
>>> of the code that allocates connections at the lowest level. Before I
>>> start picking that code apart line by line, though, I'd like all the
>>> data.
>>>
>>> Karl
>>>
>>> On Thu, Feb 7, 2013 at 9:07 AM, Karl Wright <da...@gmail.com> wrote:
>>>>
>>>> Thanks, this is great.
>>>>
>>>> Could you include ALL of the 'possibly leaked' connections? I'm
>>>> looking for the ones that show up over and over again. Please attach
>>>> that part of the log to the ticket.
>>>>
>>>> Karl
>>>>
>>>> On Thu, Feb 7, 2013 at 9:04 AM, Erlend Garåsen <e....@usit.uio.no>
>>>> wrote:
>>>>>
>>>>> On 06.02.13 15.24, Karl Wright wrote:
>>>>>
>>>>>> FWIW, schema creation has been working fine here, and all tests that
>>>>>> exercise the schema have been passing fine as well.
>>>>>
>>>>>
>>>>>
>>>>> My fault, but everything is successfully set up now.
>>>>>
>>>>> I think I have reproduced the issue. The crawl is still running, but I
>>>>> guess
>>>>> it will stop very soon due to the following entries in my log. The last
>>>>> fetched URL before the first of these log entries below was:
>>>>>
>>>>> https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid
>>>>>
>>>>> It seems that this URL is part of a search, so maybe we will get around
>>>>> this
>>>>> problem by filtering out all search-related URLs. The problem is that it
>>>>> is
>>>>> already filtered out, but by doing post filtering. I can try to adjust
>>>>> that
>>>>> at next attempt.
>>>>>
>>>>> INFO 2013-02-07 14:32:08,166 (Worker thread '13') - WEB: FETCH
>>>>>
>>>>> URL|https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid|1360243927960+204|200|24992|
>>>>> WARN 2013-02-07 14:32:08,228 (Worker thread '16') - Out of db
>>>>> connections,
>>>>> list of outstanding ones follows.
>>>>> WARN 2013-02-07 14:32:08,229 (Worker thread '16') - Found a possibly
>>>>> leaked
>>>>> db connection
>>>>> java.lang.Exception: Possibly leaked db connection
>>>>> at
>>>>>
>>>>> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:72)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>>>>> at
>>>>>
>>>>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>>>>> at
>>>>>
>>>>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:758)
>>>>> at
>>>>>
>>>>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1430)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>>>>> at
>>>>>
>>>>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>>>>> at
>>>>>
>>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
>>>>> at
>>>>>
>>>>> org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
>>>>> at
>>>>>
>>>>> org.apache.manifoldcf.crawler.jobs.HopCount.processFind(HopCount.java:674)
>>>>> at
>>>>>
>>>>> org.apache.manifoldcf.crawler.jobs.HopCount.findHopCounts(HopCount.java:649)
>>>>> at
>>>>>
>>>>> org.apache.manifoldcf.crawler.jobs.JobManager.findHopCounts(JobManager.java:3777)
>>>>> at
>>>>>
>>>>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:224)
>>>>> WARN 2013-02-07 14:32:08,233 (Worker thread '16') - Found a possibly
>>>>> leaked
>>>>> db connection
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Erlend Garåsen
>>>>> Center for Information Technology Services
>>>>> University of Oslo
>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>> 31050
>>
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Karl Wright <da...@gmail.com>.
It is not necessary to wait until the job ends; each time it runs out
of handles it prints the entire set of allocated handles. You can
stop the job now.
Karl
On Thu, Feb 7, 2013 at 9:41 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> I will upload the log to Jira or place it on web as soon as the job ends.
>
> Erlend
>
>
> On 07.02.13 15.34, Karl Wright wrote:
>>
>> FWIW, the way this diagnostic code works is that it simply dumps ALL
>> the handles that are currently considered to be in use. Some of them
>> are leaked, and some are just being used. So it's really the sum
>> total of all of them that will help us narrow down where leaks are
>> occurring.
>>
>> I looked at the one location you provided; it's entirely unremarkable;
>> it's not even happening in a transaction. If similar locations show
>> up again and again, it will mean that the problem must be in the part
>> of the code that allocates connections at the lowest level. Before I
>> start picking that code apart line by line, though, I'd like all the
>> data.
>>
>> Karl
>>
>> On Thu, Feb 7, 2013 at 9:07 AM, Karl Wright <da...@gmail.com> wrote:
>>>
>>> Thanks, this is great.
>>>
>>> Could you include ALL of the 'possibly leaked' connections? I'm
>>> looking for the ones that show up over and over again. Please attach
>>> that part of the log to the ticket.
>>>
>>> Karl
>>>
>>> On Thu, Feb 7, 2013 at 9:04 AM, Erlend Garåsen <e....@usit.uio.no>
>>> wrote:
>>>>
>>>> On 06.02.13 15.24, Karl Wright wrote:
>>>>
>>>>> FWIW, schema creation has been working fine here, and all tests that
>>>>> exercise the schema have been passing fine as well.
>>>>
>>>>
>>>>
>>>> My fault, but everything is successfully set up now.
>>>>
>>>> I think I have reproduced the issue. The crawl is still running, but I
>>>> guess
>>>> it will stop very soon due to the following entries in my log. The last
>>>> fetched URL before the first of these log entries below was:
>>>>
>>>> https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid
>>>>
>>>> It seems that this URL is part of a search, so maybe we will get around
>>>> this
>>>> problem by filtering out all search-related URLs. The problem is that it
>>>> is
>>>> already filtered out, but by doing post filtering. I can try to adjust
>>>> that
>>>> at next attempt.
>>>>
>>>> INFO 2013-02-07 14:32:08,166 (Worker thread '13') - WEB: FETCH
>>>>
>>>> URL|https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid|1360243927960+204|200|24992|
>>>> WARN 2013-02-07 14:32:08,228 (Worker thread '16') - Out of db
>>>> connections,
>>>> list of outstanding ones follows.
>>>> WARN 2013-02-07 14:32:08,229 (Worker thread '16') - Found a possibly
>>>> leaked
>>>> db connection
>>>> java.lang.Exception: Possibly leaked db connection
>>>> at
>>>>
>>>> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:72)
>>>>
>>>> at
>>>>
>>>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>>>> at
>>>>
>>>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>>>> at
>>>>
>>>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:758)
>>>> at
>>>>
>>>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1430)
>>>>
>>>> at
>>>>
>>>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>>>> at
>>>>
>>>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>>>> at
>>>>
>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
>>>> at
>>>>
>>>> org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
>>>> at
>>>>
>>>> org.apache.manifoldcf.crawler.jobs.HopCount.processFind(HopCount.java:674)
>>>> at
>>>>
>>>> org.apache.manifoldcf.crawler.jobs.HopCount.findHopCounts(HopCount.java:649)
>>>> at
>>>>
>>>> org.apache.manifoldcf.crawler.jobs.JobManager.findHopCounts(JobManager.java:3777)
>>>> at
>>>>
>>>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:224)
>>>> WARN 2013-02-07 14:32:08,233 (Worker thread '16') - Found a possibly
>>>> leaked
>>>> db connection
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>> 31050
>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Erlend Garåsen <e....@usit.uio.no>.
I will upload the log to Jira or place it on web as soon as the job ends.
Erlend
On 07.02.13 15.34, Karl Wright wrote:
> FWIW, the way this diagnostic code works is that it simply dumps ALL
> the handles that are currently considered to be in use. Some of them
> are leaked, and some are just being used. So it's really the sum
> total of all of them that will help us narrow down where leaks are
> occurring.
>
> I looked at the one location you provided; it's entirely unremarkable;
> it's not even happening in a transaction. If similar locations show
> up again and again, it will mean that the problem must be in the part
> of the code that allocates connections at the lowest level. Before I
> start picking that code apart line by line, though, I'd like all the
> data.
>
> Karl
>
> On Thu, Feb 7, 2013 at 9:07 AM, Karl Wright <da...@gmail.com> wrote:
>> Thanks, this is great.
>>
>> Could you include ALL of the 'possibly leaked' connections? I'm
>> looking for the ones that show up over and over again. Please attach
>> that part of the log to the ticket.
>>
>> Karl
>>
>> On Thu, Feb 7, 2013 at 9:04 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>>> On 06.02.13 15.24, Karl Wright wrote:
>>>
>>>> FWIW, schema creation has been working fine here, and all tests that
>>>> exercise the schema have been passing fine as well.
>>>
>>>
>>> My fault, but everything is successfully set up now.
>>>
>>> I think I have reproduced the issue. The crawl is still running, but I guess
>>> it will stop very soon due to the following entries in my log. The last
>>> fetched URL before the first of these log entries below was:
>>> https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid
>>>
>>> It seems that this URL is part of a search, so maybe we will get around this
>>> problem by filtering out all search-related URLs. The problem is that it is
>>> already filtered out, but by doing post filtering. I can try to adjust that
>>> at next attempt.
>>>
>>> INFO 2013-02-07 14:32:08,166 (Worker thread '13') - WEB: FETCH
>>> URL|https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid|1360243927960+204|200|24992|
>>> WARN 2013-02-07 14:32:08,228 (Worker thread '16') - Out of db connections,
>>> list of outstanding ones follows.
>>> WARN 2013-02-07 14:32:08,229 (Worker thread '16') - Found a possibly leaked
>>> db connection
>>> java.lang.Exception: Possibly leaked db connection
>>> at
>>> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:72)
>>>
>>> at
>>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>>> at
>>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>>> at
>>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:758)
>>> at
>>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1430)
>>>
>>> at
>>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>>> at
>>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>>> at
>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
>>> at
>>> org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
>>> at
>>> org.apache.manifoldcf.crawler.jobs.HopCount.processFind(HopCount.java:674)
>>> at
>>> org.apache.manifoldcf.crawler.jobs.HopCount.findHopCounts(HopCount.java:649)
>>> at
>>> org.apache.manifoldcf.crawler.jobs.JobManager.findHopCounts(JobManager.java:3777)
>>> at
>>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:224)
>>> WARN 2013-02-07 14:32:08,233 (Worker thread '16') - Found a possibly leaked
>>> db connection
>>>
>>>
>>>
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Karl Wright <da...@gmail.com>.
FWIW, the way this diagnostic code works is that it simply dumps ALL
the handles that are currently considered to be in use. Some of them
are leaked, and some are just being used. So it's really the sum
total of all of them that will help us narrow down where leaks are
occurring.
I looked at the one location you provided; it's entirely unremarkable;
it's not even happening in a transaction. If similar locations show
up again and again, it will mean that the problem must be in the part
of the code that allocates connections at the lowest level. Before I
start picking that code apart line by line, though, I'd like all the
data.
Karl
On Thu, Feb 7, 2013 at 9:07 AM, Karl Wright <da...@gmail.com> wrote:
> Thanks, this is great.
>
> Could you include ALL of the 'possibly leaked' connections? I'm
> looking for the ones that show up over and over again. Please attach
> that part of the log to the ticket.
>
> Karl
>
> On Thu, Feb 7, 2013 at 9:04 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>> On 06.02.13 15.24, Karl Wright wrote:
>>
>>> FWIW, schema creation has been working fine here, and all tests that
>>> exercise the schema have been passing fine as well.
>>
>>
>> My fault, but everything is successfully set up now.
>>
>> I think I have reproduced the issue. The crawl is still running, but I guess
>> it will stop very soon due to the following entries in my log. The last
>> fetched URL before the first of these log entries below was:
>> https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid
>>
>> It seems that this URL is part of a search, so maybe we will get around this
>> problem by filtering out all search-related URLs. The problem is that it is
>> already filtered out, but by doing post filtering. I can try to adjust that
>> at next attempt.
>>
>> INFO 2013-02-07 14:32:08,166 (Worker thread '13') - WEB: FETCH
>> URL|https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid|1360243927960+204|200|24992|
>> WARN 2013-02-07 14:32:08,228 (Worker thread '16') - Out of db connections,
>> list of outstanding ones follows.
>> WARN 2013-02-07 14:32:08,229 (Worker thread '16') - Found a possibly leaked
>> db connection
>> java.lang.Exception: Possibly leaked db connection
>> at
>> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:72)
>>
>> at
>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>> at
>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>> at
>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:758)
>> at
>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1430)
>>
>> at
>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>> at
>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>> at
>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
>> at
>> org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
>> at
>> org.apache.manifoldcf.crawler.jobs.HopCount.processFind(HopCount.java:674)
>> at
>> org.apache.manifoldcf.crawler.jobs.HopCount.findHopCounts(HopCount.java:649)
>> at
>> org.apache.manifoldcf.crawler.jobs.JobManager.findHopCounts(JobManager.java:3777)
>> at
>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:224)
>> WARN 2013-02-07 14:32:08,233 (Worker thread '16') - Found a possibly leaked
>> db connection
>>
>>
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Karl Wright <da...@gmail.com>.
Thanks, this is great.
Could you include ALL of the 'possibly leaked' connections? I'm
looking for the ones that show up over and over again. Please attach
that part of the log to the ticket.
Karl
On Thu, Feb 7, 2013 at 9:04 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
> On 06.02.13 15.24, Karl Wright wrote:
>
>> FWIW, schema creation has been working fine here, and all tests that
>> exercise the schema have been passing fine as well.
>
>
> My fault, but everything is successfully set up now.
>
> I think I have reproduced the issue. The crawl is still running, but I guess
> it will stop very soon due to the following entries in my log. The last
> fetched URL before the first of these log entries below was:
> https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid
>
> It seems that this URL is part of a search, so maybe we will get around this
> problem by filtering out all search-related URLs. The problem is that it is
> already filtered out, but by doing post filtering. I can try to adjust that
> at next attempt.
>
> INFO 2013-02-07 14:32:08,166 (Worker thread '13') - WEB: FETCH
> URL|https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid|1360243927960+204|200|24992|
> WARN 2013-02-07 14:32:08,228 (Worker thread '16') - Out of db connections,
> list of outstanding ones follows.
> WARN 2013-02-07 14:32:08,229 (Worker thread '16') - Found a possibly leaked
> db connection
> java.lang.Exception: Possibly leaked db connection
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:72)
>
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:758)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1430)
>
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
> at
> org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
> at
> org.apache.manifoldcf.crawler.jobs.HopCount.processFind(HopCount.java:674)
> at
> org.apache.manifoldcf.crawler.jobs.HopCount.findHopCounts(HopCount.java:649)
> at
> org.apache.manifoldcf.crawler.jobs.JobManager.findHopCounts(JobManager.java:3777)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:224)
> WARN 2013-02-07 14:32:08,233 (Worker thread '16') - Found a possibly leaked
> db connection
>
>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Erlend Garåsen <e....@usit.uio.no>.
On 06.02.13 15.24, Karl Wright wrote:
> FWIW, schema creation has been working fine here, and all tests that
> exercise the schema have been passing fine as well.
My fault, but everything is successfully set up now.
I think I have reproduced the issue. The crawl is still running, but I
guess it will stop very soon due to the following entries in my log. The
last fetched URL before the first of these log entries below was:
https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid
It seems that this URL is part of a search, so maybe we will get around
this problem by filtering out all search-related URLs. The problem is
that it is already filtered out, but by doing post filtering. I can try
to adjust that at next attempt.
INFO 2013-02-07 14:32:08,166 (Worker thread '13') - WEB: FETCH
URL|https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid|1360243927960+204|200|24992|
WARN 2013-02-07 14:32:08,228 (Worker thread '16') - Out of db
connections, list of outstanding ones follows.
WARN 2013-02-07 14:32:08,229 (Worker thread '16') - Found a possibly
leaked db connection
java.lang.Exception: Possibly leaked db connection
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:72)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:758)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1430)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
at
org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
at
org.apache.manifoldcf.crawler.jobs.HopCount.processFind(HopCount.java:674)
at
org.apache.manifoldcf.crawler.jobs.HopCount.findHopCounts(HopCount.java:649)
at
org.apache.manifoldcf.crawler.jobs.JobManager.findHopCounts(JobManager.java:3777)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:224)
WARN 2013-02-07 14:32:08,233 (Worker thread '16') - Found a possibly
leaked db connection
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Karl Wright <da...@gmail.com>.
The jobsforcedparam table is a crawler core database table that was
added as part of CONNECTORS-552. It should be created or added as
part of the crawleragent register command. If this command is
failing, you will need to determine why.
FWIW, schema creation has been working fine here, and all tests that
exercise the schema have been passing fine as well.
Karl
On Wed, Feb 6, 2013 at 9:01 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
> On 06.02.13 14.14, Karl Wright wrote:
>>
>> The database instance create logic should do nothing if the instance
>> already exists. In fact, that is how it is coded throughout. The
>> expected deployment model is to execute all the operations you would
>> normally do, in the same order, including connector registration,
>> every time you update the ManifoldCF software. If you want to skip
>> the database instance creation, you can't use initialize.bat it is
>> true, but you can do everything that it would be doing through the use
>> of individual ManifoldCF commands, e.g.:
>>
>> org.apache.manifoldcf.agents.Initialize
>> org.apache.manifoldcf.agents.RegisterAgent
>> etc.
>
>
> Yes, and this is actually a part of our deployment steps since we cannot use
> initialize.sh. One of these commands failed, so I just emptied the database
> manually.
>
> The problem now is that I'm still getting the error while I'm trying to view
> my jobs. Same error:
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
> exception: SQLException doing query (42P01): ERROR: relation
> "jobforcedparams" does not exist
>
> This is what I ran after I emptied my database:
>
> 1. org.apache.manifoldcf.agents.Install
>
> 2. org.apache.manifoldcf.agents.Register
> org.apache.manifoldcf.crawler.system.CrawlerAgent
>
> 3. org.apache.manifoldcf.crawler.Register
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector
> WebCrawler
>
> 4. org.apache.manifoldcf.agents.RegisterOutput
> org.apache.manifoldcf.agents.output.solr.SolrConnector SolrConnector
>
> Should I have performed org.apache.manifoldcf.agents.Initialize as well? I
> have never ran that command class before, probably bevause it tries to
> create the database?
>
> Is jobforcedparams something related to connection tracking?
>
> Erlend
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Erlend Garåsen <e....@usit.uio.no>.
On 06.02.13 14.14, Karl Wright wrote:
> The database instance create logic should do nothing if the instance
> already exists. In fact, that is how it is coded throughout. The
> expected deployment model is to execute all the operations you would
> normally do, in the same order, including connector registration,
> every time you update the ManifoldCF software. If you want to skip
> the database instance creation, you can't use initialize.bat it is
> true, but you can do everything that it would be doing through the use
> of individual ManifoldCF commands, e.g.:
>
> org.apache.manifoldcf.agents.Initialize
> org.apache.manifoldcf.agents.RegisterAgent
> etc.
Yes, and this is actually a part of our deployment steps since we cannot
use initialize.sh. One of these commands failed, so I just emptied the
database manually.
The problem now is that I'm still getting the error while I'm trying to
view my jobs. Same error:
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
exception: SQLException doing query (42P01): ERROR: relation
"jobforcedparams" does not exist
This is what I ran after I emptied my database:
1. org.apache.manifoldcf.agents.Install
2. org.apache.manifoldcf.agents.Register
org.apache.manifoldcf.crawler.system.CrawlerAgent
3. org.apache.manifoldcf.crawler.Register
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector
WebCrawler
4. org.apache.manifoldcf.agents.RegisterOutput
org.apache.manifoldcf.agents.output.solr.SolrConnector SolrConnector
Should I have performed org.apache.manifoldcf.agents.Initialize as well?
I have never ran that command class before, probably bevause it tries to
create the database?
Is jobforcedparams something related to connection tracking?
Erlend
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Karl Wright <da...@gmail.com>.
The database instance create logic should do nothing if the instance
already exists. In fact, that is how it is coded throughout. The
expected deployment model is to execute all the operations you would
normally do, in the same order, including connector registration,
every time you update the ManifoldCF software. If you want to skip
the database instance creation, you can't use initialize.bat it is
true, but you can do everything that it would be doing through the use
of individual ManifoldCF commands, e.g.:
org.apache.manifoldcf.agents.Initialize
org.apache.manifoldcf.agents.RegisterAgent
etc.
They are all laid out in how-to-build-and-deploy.
Kalr
On Wed, Feb 6, 2013 at 8:07 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> Nope, I don't think so. The initialize script will also create the database
> which I do not have access to. Remember that we are using an external PG
> Server. :)
>
> From the source code:
> ManifoldCF.createSystemDatabase(tc);
>
> Erlend
>
>
> On 06.02.13 13.57, Karl Wright wrote:
>>
>> Hi Erlend,
>> You don't need to delete the tables; the upgrade is automatic if you
>> simply repeat the standard database initialization steps. For
>> example:
>>
>> initialize.bat
>>
>> ... will complete the upgrade.
>>
>> Karl
>>
>> On Wed, Feb 6, 2013 at 7:50 AM, Erlend Garåsen <e....@usit.uio.no>
>> wrote:
>>>
>>>
>>> You are probably right. I found this in another log:
>>> Caused by: org.postgresql.util.PSQLException: ERROR: relation
>>> "jobforcedparams" does not exist
>>> Position: 15
>>>
>>> It seems that I need to do additional steps in order to update the
>>> tables.
>>> I'll delete all tables manually and start all over again. :)
>>>
>>> Erlend
>>>
>>>
>>> On 06.02.13 13.31, Karl Wright wrote:
>>>>
>>>>
>>>> The jsp forward error is occurring because you are getting some kind
>>>> of error listing jobs, and the UI is attempting to forward you to the
>>>> error page. Unfortunately under Resin that is not apparently possible
>>>> since Resin apparently sets some Http headers before we have any
>>>> chance of doing the forward operation. So I think this is likely a
>>>> Resin-only problem.
>>>>
>>>> Karl
>>>>
>>>> On Wed, Feb 6, 2013 at 7:15 AM, Erlend Garåsen <e....@usit.uio.no>
>>>> wrote:
>>>>>
>>>>>
>>>>> On 06.02.13 12.56, Erlend Garåsen wrote:
>>>>>
>>>>>> Just a question. I see the following dumping into my logs every second
>>>>>> even though no jobs are running at the moment. Is this related to
>>>>>> connection tracking? I'm just afraid that it will be difficult to find
>>>>>> the relevant log entries with such an active logging.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Nevermind, I had several other log levels set to debug, so that was the
>>>>> reason.
>>>>>
>>>>> I'm afraid that trunk does not work well on Resin. If I go to "List all
>>>>> jobs", I'm getting the following error:
>>>>> "java.lang.IllegalStateException:
>>>>> can't forward after writing HTTP headers".
>>>>>
>>>>> I will open a ticket if this error does not disappear after a second
>>>>> deployment. I have seen this before for a year ago, but I cannot
>>>>> remember
>>>>> the cause. I will investigate this as well.
>>>>>
>>>>> Full stack trace:
>>>>>
>>>>> [2013-02-06 13:09:09.443] {resin-port-127.0.0.1:6945-43}
>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>> headers
>>>>> at
>>>>> com.caucho.jsp.PageContextImpl.forward(PageContextImpl.java:1122)
>>>>> at
>>>>> _jsp._listjobs__jsp._jspService(_listjobs__jsp.java:267)
>>>>> at
>>>>> _jsp._listjobs__jsp._jspService(_listjobs__jsp.java:36)
>>>>> at
>>>>> com.caucho.jsp.JavaPage.service(JavaPage.java:64)
>>>>> at
>>>>> com.caucho.jsp.Page.pageservice(Page.java:542)
>>>>> at
>>>>>
>>>>>
>>>>> com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194)
>>>>> at
>>>>>
>>>>>
>>>>> com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:156)
>>>>> at
>>>>>
>>>>>
>>>>> com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:289)
>>>>> at
>>>>>
>>>>> com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:468)
>>>>> at
>>>>>
>>>>>
>>>>> com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:369)
>>>>> at
>>>>> com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:336)
>>>>> at
>>>>>
>>>>>
>>>>> com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:1301)
>>>>> at
>>>>>
>>>>>
>>>>> com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:1257)
>>>>> at
>>>>>
>>>>>
>>>>> com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:1241)
>>>>> at
>>>>>
>>>>>
>>>>> com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:1149)
>>>>> at
>>>>>
>>>>>
>>>>> com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl(TcpSocketLink.java:950)
>>>>> at
>>>>>
>>>>>
>>>>> com.caucho.network.listen.ConnectionTask.runThread(ConnectionTask.java:117)
>>>>> at
>>>>> com.caucho.network.listen.ConnectionTask.run(ConnectionTask.java:93)
>>>>> at
>>>>>
>>>>>
>>>>> com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks(SocketLinkThreadLauncher.java:169)
>>>>> at
>>>>>
>>>>>
>>>>> com.caucho.network.listen.TcpSocketAcceptThread.run(TcpSocketAcceptThread.java:61)
>>>>> at
>>>>> com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
>>>>> at
>>>>> com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Erlend Garåsen
>>>>> Center for Information Technology Services
>>>>> University of Oslo
>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>> 31050
>>>
>>>
>>>
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>> 31050
>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Erlend Garåsen <e....@usit.uio.no>.
Nope, I don't think so. The initialize script will also create the
database which I do not have access to. Remember that we are using an
external PG Server. :)
From the source code:
ManifoldCF.createSystemDatabase(tc);
Erlend
On 06.02.13 13.57, Karl Wright wrote:
> Hi Erlend,
> You don't need to delete the tables; the upgrade is automatic if you
> simply repeat the standard database initialization steps. For
> example:
>
> initialize.bat
>
> ... will complete the upgrade.
>
> Karl
>
> On Wed, Feb 6, 2013 at 7:50 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>>
>> You are probably right. I found this in another log:
>> Caused by: org.postgresql.util.PSQLException: ERROR: relation
>> "jobforcedparams" does not exist
>> Position: 15
>>
>> It seems that I need to do additional steps in order to update the tables.
>> I'll delete all tables manually and start all over again. :)
>>
>> Erlend
>>
>>
>> On 06.02.13 13.31, Karl Wright wrote:
>>>
>>> The jsp forward error is occurring because you are getting some kind
>>> of error listing jobs, and the UI is attempting to forward you to the
>>> error page. Unfortunately under Resin that is not apparently possible
>>> since Resin apparently sets some Http headers before we have any
>>> chance of doing the forward operation. So I think this is likely a
>>> Resin-only problem.
>>>
>>> Karl
>>>
>>> On Wed, Feb 6, 2013 at 7:15 AM, Erlend Garåsen <e....@usit.uio.no>
>>> wrote:
>>>>
>>>> On 06.02.13 12.56, Erlend Garåsen wrote:
>>>>
>>>>> Just a question. I see the following dumping into my logs every second
>>>>> even though no jobs are running at the moment. Is this related to
>>>>> connection tracking? I'm just afraid that it will be difficult to find
>>>>> the relevant log entries with such an active logging.
>>>>
>>>>
>>>>
>>>> Nevermind, I had several other log levels set to debug, so that was the
>>>> reason.
>>>>
>>>> I'm afraid that trunk does not work well on Resin. If I go to "List all
>>>> jobs", I'm getting the following error: "java.lang.IllegalStateException:
>>>> can't forward after writing HTTP headers".
>>>>
>>>> I will open a ticket if this error does not disappear after a second
>>>> deployment. I have seen this before for a year ago, but I cannot remember
>>>> the cause. I will investigate this as well.
>>>>
>>>> Full stack trace:
>>>>
>>>> [2013-02-06 13:09:09.443] {resin-port-127.0.0.1:6945-43}
>>>> java.lang.IllegalStateException: can't forward after writing HTTP headers
>>>> at
>>>> com.caucho.jsp.PageContextImpl.forward(PageContextImpl.java:1122)
>>>> at
>>>> _jsp._listjobs__jsp._jspService(_listjobs__jsp.java:267)
>>>> at
>>>> _jsp._listjobs__jsp._jspService(_listjobs__jsp.java:36)
>>>> at
>>>> com.caucho.jsp.JavaPage.service(JavaPage.java:64)
>>>> at
>>>> com.caucho.jsp.Page.pageservice(Page.java:542)
>>>> at
>>>>
>>>> com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194)
>>>> at
>>>>
>>>> com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:156)
>>>> at
>>>>
>>>> com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:289)
>>>> at
>>>> com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:468)
>>>> at
>>>>
>>>> com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:369)
>>>> at
>>>> com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:336)
>>>> at
>>>>
>>>> com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:1301)
>>>> at
>>>>
>>>> com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:1257)
>>>> at
>>>>
>>>> com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:1241)
>>>> at
>>>>
>>>> com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:1149)
>>>> at
>>>>
>>>> com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl(TcpSocketLink.java:950)
>>>> at
>>>>
>>>> com.caucho.network.listen.ConnectionTask.runThread(ConnectionTask.java:117)
>>>> at
>>>> com.caucho.network.listen.ConnectionTask.run(ConnectionTask.java:93)
>>>> at
>>>>
>>>> com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks(SocketLinkThreadLauncher.java:169)
>>>> at
>>>>
>>>> com.caucho.network.listen.TcpSocketAcceptThread.run(TcpSocketAcceptThread.java:61)
>>>> at
>>>> com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
>>>> at
>>>> com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>> 31050
>>
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Karl Wright <da...@gmail.com>.
Hi Erlend,
You don't need to delete the tables; the upgrade is automatic if you
simply repeat the standard database initialization steps. For
example:
initialize.bat
... will complete the upgrade.
Karl
On Wed, Feb 6, 2013 at 7:50 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> You are probably right. I found this in another log:
> Caused by: org.postgresql.util.PSQLException: ERROR: relation
> "jobforcedparams" does not exist
> Position: 15
>
> It seems that I need to do additional steps in order to update the tables.
> I'll delete all tables manually and start all over again. :)
>
> Erlend
>
>
> On 06.02.13 13.31, Karl Wright wrote:
>>
>> The jsp forward error is occurring because you are getting some kind
>> of error listing jobs, and the UI is attempting to forward you to the
>> error page. Unfortunately under Resin that is not apparently possible
>> since Resin apparently sets some Http headers before we have any
>> chance of doing the forward operation. So I think this is likely a
>> Resin-only problem.
>>
>> Karl
>>
>> On Wed, Feb 6, 2013 at 7:15 AM, Erlend Garåsen <e....@usit.uio.no>
>> wrote:
>>>
>>> On 06.02.13 12.56, Erlend Garåsen wrote:
>>>
>>>> Just a question. I see the following dumping into my logs every second
>>>> even though no jobs are running at the moment. Is this related to
>>>> connection tracking? I'm just afraid that it will be difficult to find
>>>> the relevant log entries with such an active logging.
>>>
>>>
>>>
>>> Nevermind, I had several other log levels set to debug, so that was the
>>> reason.
>>>
>>> I'm afraid that trunk does not work well on Resin. If I go to "List all
>>> jobs", I'm getting the following error: "java.lang.IllegalStateException:
>>> can't forward after writing HTTP headers".
>>>
>>> I will open a ticket if this error does not disappear after a second
>>> deployment. I have seen this before for a year ago, but I cannot remember
>>> the cause. I will investigate this as well.
>>>
>>> Full stack trace:
>>>
>>> [2013-02-06 13:09:09.443] {resin-port-127.0.0.1:6945-43}
>>> java.lang.IllegalStateException: can't forward after writing HTTP headers
>>> at
>>> com.caucho.jsp.PageContextImpl.forward(PageContextImpl.java:1122)
>>> at
>>> _jsp._listjobs__jsp._jspService(_listjobs__jsp.java:267)
>>> at
>>> _jsp._listjobs__jsp._jspService(_listjobs__jsp.java:36)
>>> at
>>> com.caucho.jsp.JavaPage.service(JavaPage.java:64)
>>> at
>>> com.caucho.jsp.Page.pageservice(Page.java:542)
>>> at
>>>
>>> com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194)
>>> at
>>>
>>> com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:156)
>>> at
>>>
>>> com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:289)
>>> at
>>> com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:468)
>>> at
>>>
>>> com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:369)
>>> at
>>> com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:336)
>>> at
>>>
>>> com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:1301)
>>> at
>>>
>>> com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:1257)
>>> at
>>>
>>> com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:1241)
>>> at
>>>
>>> com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:1149)
>>> at
>>>
>>> com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl(TcpSocketLink.java:950)
>>> at
>>>
>>> com.caucho.network.listen.ConnectionTask.runThread(ConnectionTask.java:117)
>>> at
>>> com.caucho.network.listen.ConnectionTask.run(ConnectionTask.java:93)
>>> at
>>>
>>> com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks(SocketLinkThreadLauncher.java:169)
>>> at
>>>
>>> com.caucho.network.listen.TcpSocketAcceptThread.run(TcpSocketAcceptThread.java:61)
>>> at
>>> com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
>>> at
>>> com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)
>>>
>>>
>>>
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>> 31050
>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Erlend Garåsen <e....@usit.uio.no>.
You are probably right. I found this in another log:
Caused by: org.postgresql.util.PSQLException: ERROR: relation
"jobforcedparams" does not exist
Position: 15
It seems that I need to do additional steps in order to update the
tables. I'll delete all tables manually and start all over again. :)
Erlend
On 06.02.13 13.31, Karl Wright wrote:
> The jsp forward error is occurring because you are getting some kind
> of error listing jobs, and the UI is attempting to forward you to the
> error page. Unfortunately under Resin that is not apparently possible
> since Resin apparently sets some Http headers before we have any
> chance of doing the forward operation. So I think this is likely a
> Resin-only problem.
>
> Karl
>
> On Wed, Feb 6, 2013 at 7:15 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>> On 06.02.13 12.56, Erlend Garåsen wrote:
>>
>>> Just a question. I see the following dumping into my logs every second
>>> even though no jobs are running at the moment. Is this related to
>>> connection tracking? I'm just afraid that it will be difficult to find
>>> the relevant log entries with such an active logging.
>>
>>
>> Nevermind, I had several other log levels set to debug, so that was the
>> reason.
>>
>> I'm afraid that trunk does not work well on Resin. If I go to "List all
>> jobs", I'm getting the following error: "java.lang.IllegalStateException:
>> can't forward after writing HTTP headers".
>>
>> I will open a ticket if this error does not disappear after a second
>> deployment. I have seen this before for a year ago, but I cannot remember
>> the cause. I will investigate this as well.
>>
>> Full stack trace:
>>
>> [2013-02-06 13:09:09.443] {resin-port-127.0.0.1:6945-43}
>> java.lang.IllegalStateException: can't forward after writing HTTP headers
>> at
>> com.caucho.jsp.PageContextImpl.forward(PageContextImpl.java:1122)
>> at
>> _jsp._listjobs__jsp._jspService(_listjobs__jsp.java:267)
>> at
>> _jsp._listjobs__jsp._jspService(_listjobs__jsp.java:36)
>> at
>> com.caucho.jsp.JavaPage.service(JavaPage.java:64)
>> at
>> com.caucho.jsp.Page.pageservice(Page.java:542)
>> at
>> com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194)
>> at
>> com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:156)
>> at
>> com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:289)
>> at
>> com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:468)
>> at
>> com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:369)
>> at
>> com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:336)
>> at
>> com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:1301)
>> at
>> com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:1257)
>> at
>> com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:1241)
>> at
>> com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:1149)
>> at
>> com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl(TcpSocketLink.java:950)
>> at
>> com.caucho.network.listen.ConnectionTask.runThread(ConnectionTask.java:117)
>> at
>> com.caucho.network.listen.ConnectionTask.run(ConnectionTask.java:93)
>> at
>> com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks(SocketLinkThreadLauncher.java:169)
>> at
>> com.caucho.network.listen.TcpSocketAcceptThread.run(TcpSocketAcceptThread.java:61)
>> at
>> com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
>> at
>> com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)
>>
>>
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Karl Wright <da...@gmail.com>.
The jsp forward error is occurring because you are getting some kind
of error listing jobs, and the UI is attempting to forward you to the
error page. Unfortunately under Resin that is not apparently possible
since Resin apparently sets some Http headers before we have any
chance of doing the forward operation. So I think this is likely a
Resin-only problem.
Karl
On Wed, Feb 6, 2013 at 7:15 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
> On 06.02.13 12.56, Erlend Garåsen wrote:
>
>> Just a question. I see the following dumping into my logs every second
>> even though no jobs are running at the moment. Is this related to
>> connection tracking? I'm just afraid that it will be difficult to find
>> the relevant log entries with such an active logging.
>
>
> Nevermind, I had several other log levels set to debug, so that was the
> reason.
>
> I'm afraid that trunk does not work well on Resin. If I go to "List all
> jobs", I'm getting the following error: "java.lang.IllegalStateException:
> can't forward after writing HTTP headers".
>
> I will open a ticket if this error does not disappear after a second
> deployment. I have seen this before for a year ago, but I cannot remember
> the cause. I will investigate this as well.
>
> Full stack trace:
>
> [2013-02-06 13:09:09.443] {resin-port-127.0.0.1:6945-43}
> java.lang.IllegalStateException: can't forward after writing HTTP headers
> at
> com.caucho.jsp.PageContextImpl.forward(PageContextImpl.java:1122)
> at
> _jsp._listjobs__jsp._jspService(_listjobs__jsp.java:267)
> at
> _jsp._listjobs__jsp._jspService(_listjobs__jsp.java:36)
> at
> com.caucho.jsp.JavaPage.service(JavaPage.java:64)
> at
> com.caucho.jsp.Page.pageservice(Page.java:542)
> at
> com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194)
> at
> com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:156)
> at
> com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:289)
> at
> com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:468)
> at
> com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:369)
> at
> com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:336)
> at
> com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:1301)
> at
> com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:1257)
> at
> com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:1241)
> at
> com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:1149)
> at
> com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl(TcpSocketLink.java:950)
> at
> com.caucho.network.listen.ConnectionTask.runThread(ConnectionTask.java:117)
> at
> com.caucho.network.listen.ConnectionTask.run(ConnectionTask.java:93)
> at
> com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks(SocketLinkThreadLauncher.java:169)
> at
> com.caucho.network.listen.TcpSocketAcceptThread.run(TcpSocketAcceptThread.java:61)
> at
> com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
> at
> com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)
>
>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Erlend Garåsen <e....@usit.uio.no>.
On 06.02.13 12.56, Erlend Garåsen wrote:
> Just a question. I see the following dumping into my logs every second
> even though no jobs are running at the moment. Is this related to
> connection tracking? I'm just afraid that it will be difficult to find
> the relevant log entries with such an active logging.
Nevermind, I had several other log levels set to debug, so that was the
reason.
I'm afraid that trunk does not work well on Resin. If I go to "List all
jobs", I'm getting the following error:
"java.lang.IllegalStateException: can't forward after writing HTTP headers".
I will open a ticket if this error does not disappear after a second
deployment. I have seen this before for a year ago, but I cannot
remember the cause. I will investigate this as well.
Full stack trace:
[2013-02-06 13:09:09.443] {resin-port-127.0.0.1:6945-43}
java.lang.IllegalStateException: can't forward after writing HTTP headers
at
com.caucho.jsp.PageContextImpl.forward(PageContextImpl.java:1122)
at
_jsp._listjobs__jsp._jspService(_listjobs__jsp.java:267)
at
_jsp._listjobs__jsp._jspService(_listjobs__jsp.java:36)
at
com.caucho.jsp.JavaPage.service(JavaPage.java:64)
at
com.caucho.jsp.Page.pageservice(Page.java:542)
at
com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194)
at
com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:156)
at
com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:289)
at
com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:468)
at
com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:369)
at
com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:336)
at
com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:1301)
at
com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:1257)
at
com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:1241)
at
com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:1149)
at
com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl(TcpSocketLink.java:950)
at
com.caucho.network.listen.ConnectionTask.runThread(ConnectionTask.java:117)
at
com.caucho.network.listen.ConnectionTask.run(ConnectionTask.java:93)
at
com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks(SocketLinkThreadLauncher.java:169)
at
com.caucho.network.listen.TcpSocketAcceptThread.run(TcpSocketAcceptThread.java:61)
at
com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
at
com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Erlend Garåsen <e....@usit.uio.no>.
It's not difficult to reproduce, because it *will* fail with the job I
have defined. I'm almost sure which host is causing these problems -
it's related to complex articles. I can even create a new job with only
the problematic host defined. Yes, that's exactly what I'm going to do,
I think.
Just a question. I see the following dumping into my logs every second
even though no jobs are running at the moment. Is this related to
connection tracking? I'm just afraid that it will be difficult to find
the relevant log entries with such an active logging.
Erlend
On 06.02.13 12.43, Karl Wright wrote:
> Hi Erlend,
>
> Various crawler features are not where the problem is going to be
> coming from. It will be coming from the basic process of queuing
> documents and processing them, and the interaction with PostgreSQL.
>
> If you are worried that you won't be able to reproduce the issue, my
> suggestion is to set up a job similar to what you had before that
> hung. As long as you are using the same PostgreSQL server (not
> instance), the conditions will be ripe for the same kind of handle
> leakage as before. If you had tons of jobs running, and have no idea
> which one caused the problem, don't worry - the only thing that might
> actually matters is the kind of jobs you are doing: web, rss, etc.,
> along with possibly the overall system load.
>
> Karl
>
> On Wed, Feb 6, 2013 at 4:48 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>>
>> I have some time this week to investigate this further after I finally
>> delivered a heavy job about SAML integration. I will look through the source
>> code as well and try to log even more if necessary.
>>
>> Maciej: Can you please send me a detailed description about your similar
>> problems?
>>
>> I previously mentioned that the "exclude from index" functionality in the
>> exclusion tab could be the source of the problem, but that does not seem to
>> be the case anymore after I got the same problem for another job without any
>> settings there.
>>
>> Just a guessing, but I think the source of the problem is problematic
>> documents, for instance invalid XML documents which the crawler tries to
>> parse.
>>
>> Karl: If necessary, you can try to crawl these documents yourself, but first
>> I need to isolate the problem to one particular host.
>>
>> Thanks about the warnings about no backward compatibility for this test, so
>> I will only use our test server where data integrity is not that important.
>> I'm starting a new crawl right away and will report thereafter.
>>
>> Erlend
>>
>>
>> On 05.02.13 14.10, Karl Wright wrote:
>>>
>>> Ok, it is clear from this that most of your threads are waiting to get
>>> a connection, and there are no connections to be found. This is
>>> exactly the problem that Maciej reported, which I created the
>>> CONNECTORS-638 ticket for. There has to be a connection leak
>>> somewhere. Obviously it is not a common situation, or the problem
>>> would arise almost right away; it probably occurs as a result of some
>>> error condition or pathway that is relatively uncommon.
>>>
>>> The diagnostic code that is now checked into trunk should work as follows:
>>>
>>> (1) First, checkout and build trunk. Since there are schema changes
>>> in trunk vs. older versions of ManifoldCF, you cannot "go backwards"
>>> and run an older version on a particular database instance once you've
>>> run trunk. Keep that in mind.
>>>
>>> (2) Add a line to the properties.xml file, as follows:
>>>
>>> <property name="org.apache.manifoldcf.database.connectiontracking"
>>> value="true"/>
>>>
>>> (3) Start the system up and let it run.
>>>
>>> (4) When it fails, you should start to see dumps in the log like this:
>>>
>>> Logging.db.warn("Out of db connections, list of
>>> outstanding ones follows.");
>>> for (WrappedConnection c : outstandingConnections)
>>> {
>>> Logging.db.warn("Found a possibly leaked db
>>> connection",c.getInstantiationException());
>>> }
>>>
>>> ... which will dump where all the offending connections were
>>> allocated. Hopefully this will point us at what the problem is. If
>>> there seems to be no consistency here, I'll have to explore the
>>> possibility that there are bugs in the connection allocation/free
>>> code, but we'll see.
>>>
>>> Karl
>>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Karl Wright <da...@gmail.com>.
Hi Erlend,
Various crawler features are not where the problem is going to be
coming from. It will be coming from the basic process of queuing
documents and processing them, and the interaction with PostgreSQL.
If you are worried that you won't be able to reproduce the issue, my
suggestion is to set up a job similar to what you had before that
hung. As long as you are using the same PostgreSQL server (not
instance), the conditions will be ripe for the same kind of handle
leakage as before. If you had tons of jobs running, and have no idea
which one caused the problem, don't worry - the only thing that might
actually matters is the kind of jobs you are doing: web, rss, etc.,
along with possibly the overall system load.
Karl
On Wed, Feb 6, 2013 at 4:48 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> I have some time this week to investigate this further after I finally
> delivered a heavy job about SAML integration. I will look through the source
> code as well and try to log even more if necessary.
>
> Maciej: Can you please send me a detailed description about your similar
> problems?
>
> I previously mentioned that the "exclude from index" functionality in the
> exclusion tab could be the source of the problem, but that does not seem to
> be the case anymore after I got the same problem for another job without any
> settings there.
>
> Just a guessing, but I think the source of the problem is problematic
> documents, for instance invalid XML documents which the crawler tries to
> parse.
>
> Karl: If necessary, you can try to crawl these documents yourself, but first
> I need to isolate the problem to one particular host.
>
> Thanks about the warnings about no backward compatibility for this test, so
> I will only use our test server where data integrity is not that important.
> I'm starting a new crawl right away and will report thereafter.
>
> Erlend
>
>
> On 05.02.13 14.10, Karl Wright wrote:
>>
>> Ok, it is clear from this that most of your threads are waiting to get
>> a connection, and there are no connections to be found. This is
>> exactly the problem that Maciej reported, which I created the
>> CONNECTORS-638 ticket for. There has to be a connection leak
>> somewhere. Obviously it is not a common situation, or the problem
>> would arise almost right away; it probably occurs as a result of some
>> error condition or pathway that is relatively uncommon.
>>
>> The diagnostic code that is now checked into trunk should work as follows:
>>
>> (1) First, checkout and build trunk. Since there are schema changes
>> in trunk vs. older versions of ManifoldCF, you cannot "go backwards"
>> and run an older version on a particular database instance once you've
>> run trunk. Keep that in mind.
>>
>> (2) Add a line to the properties.xml file, as follows:
>>
>> <property name="org.apache.manifoldcf.database.connectiontracking"
>> value="true"/>
>>
>> (3) Start the system up and let it run.
>>
>> (4) When it fails, you should start to see dumps in the log like this:
>>
>> Logging.db.warn("Out of db connections, list of
>> outstanding ones follows.");
>> for (WrappedConnection c : outstandingConnections)
>> {
>> Logging.db.warn("Found a possibly leaked db
>> connection",c.getInstantiationException());
>> }
>>
>> ... which will dump where all the offending connections were
>> allocated. Hopefully this will point us at what the problem is. If
>> there seems to be no consistency here, I'll have to explore the
>> possibility that there are bugs in the connection allocation/free
>> code, but we'll see.
>>
>> Karl
>>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Erlend Garåsen <e....@usit.uio.no>.
I have some time this week to investigate this further after I finally
delivered a heavy job about SAML integration. I will look through the
source code as well and try to log even more if necessary.
Maciej: Can you please send me a detailed description about your similar
problems?
I previously mentioned that the "exclude from index" functionality in
the exclusion tab could be the source of the problem, but that does not
seem to be the case anymore after I got the same problem for another job
without any settings there.
Just a guessing, but I think the source of the problem is problematic
documents, for instance invalid XML documents which the crawler tries to
parse.
Karl: If necessary, you can try to crawl these documents yourself, but
first I need to isolate the problem to one particular host.
Thanks about the warnings about no backward compatibility for this test,
so I will only use our test server where data integrity is not that
important. I'm starting a new crawl right away and will report thereafter.
Erlend
On 05.02.13 14.10, Karl Wright wrote:
> Ok, it is clear from this that most of your threads are waiting to get
> a connection, and there are no connections to be found. This is
> exactly the problem that Maciej reported, which I created the
> CONNECTORS-638 ticket for. There has to be a connection leak
> somewhere. Obviously it is not a common situation, or the problem
> would arise almost right away; it probably occurs as a result of some
> error condition or pathway that is relatively uncommon.
>
> The diagnostic code that is now checked into trunk should work as follows:
>
> (1) First, checkout and build trunk. Since there are schema changes
> in trunk vs. older versions of ManifoldCF, you cannot "go backwards"
> and run an older version on a particular database instance once you've
> run trunk. Keep that in mind.
>
> (2) Add a line to the properties.xml file, as follows:
>
> <property name="org.apache.manifoldcf.database.connectiontracking"
> value="true"/>
>
> (3) Start the system up and let it run.
>
> (4) When it fails, you should start to see dumps in the log like this:
>
> Logging.db.warn("Out of db connections, list of
> outstanding ones follows.");
> for (WrappedConnection c : outstandingConnections)
> {
> Logging.db.warn("Found a possibly leaked db
> connection",c.getInstantiationException());
> }
>
> ... which will dump where all the offending connections were
> allocated. Hopefully this will point us at what the problem is. If
> there seems to be no consistency here, I'll have to explore the
> possibility that there are bugs in the connection allocation/free
> code, but we'll see.
>
> Karl
>
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Karl Wright <da...@gmail.com>.
Ok, it is clear from this that most of your threads are waiting to get
a connection, and there are no connections to be found. This is
exactly the problem that Maciej reported, which I created the
CONNECTORS-638 ticket for. There has to be a connection leak
somewhere. Obviously it is not a common situation, or the problem
would arise almost right away; it probably occurs as a result of some
error condition or pathway that is relatively uncommon.
The diagnostic code that is now checked into trunk should work as follows:
(1) First, checkout and build trunk. Since there are schema changes
in trunk vs. older versions of ManifoldCF, you cannot "go backwards"
and run an older version on a particular database instance once you've
run trunk. Keep that in mind.
(2) Add a line to the properties.xml file, as follows:
<property name="org.apache.manifoldcf.database.connectiontracking"
value="true"/>
(3) Start the system up and let it run.
(4) When it fails, you should start to see dumps in the log like this:
Logging.db.warn("Out of db connections, list of
outstanding ones follows.");
for (WrappedConnection c : outstandingConnections)
{
Logging.db.warn("Found a possibly leaked db
connection",c.getInstantiationException());
}
... which will dump where all the offending connections were
allocated. Hopefully this will point us at what the problem is. If
there seems to be no consistency here, I'll have to explore the
possibility that there are bugs in the connection allocation/free
code, but we'll see.
Karl
On Tue, Feb 5, 2013 at 7:38 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
> On 04.02.13 14.40, Karl Wright wrote:
>>
>> There's code now committed to trunk which might help diagnose this
>> problem. It's part of CONNECTORS-638 and is meant to find database
>> connection handle leaks.
>
>
> Thanks! I will deploy trunk on our test server and start a new crawl later
> today.
>
> BTW, here's the thread dump from the Agent - generaed by usint kill -QUIT
> <pid>:
>
> "Thread-5386784" daemon prio=10 tid=0xdbb67000 nid=0x318f runnable
> [0xdc6fe000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at
> com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> at
> com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
> at
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
> - locked <0xe41eeea0> (a java.lang.Object)
> at
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
> at
> com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> - locked <0xe41f0a48> (a
> com.sun.net.ssl.internal.ssl.AppInputStream)
> at
> org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:145)
> at
> org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114)
> at
> org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
> at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:274)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
> - locked <0xe41efe00> (a org.postgresql.core.v3.QueryExecutorImpl)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
> at
> org.apache.manifoldcf.core.database.Database.execute(Database.java:826)
> at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641)
>
> "Thread-5386763" daemon prio=10 tid=0xd8202800 nid=0x3171 runnable
> [0xdc369000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at
> com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> at
> com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
> at
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
> - locked <0xe41dbac8> (a java.lang.Object)
> at
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
> at
> com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> - locked <0xe41dcd78> (a
> com.sun.net.ssl.internal.ssl.AppInputStream)
> at
> org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:145)
> at
> org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114)
> at
> org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
> at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:274)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
> - locked <0xe41dd280> (a org.postgresql.core.v3.QueryExecutorImpl)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)
> at
> org.apache.manifoldcf.core.database.Database.execute(Database.java:840)
> at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641)
>
> "MultiThreadedHttpConnectionManager cleanup" daemon prio=10 tid=0xddc92800
> nid=0x1ca in Object.wait() [0xdccad000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <0xe418ff40> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(Unknown
> Source)
>
> "Idle cleanup thread" daemon prio=10 tid=0xde729800 nid=0x1f56 in
> Object.wait() [0xdd3e8000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:919)
> - locked <0xf272fe90> (a java.lang.Integer)
> at
> org.apache.manifoldcf.crawler.system.IdleCleanupThread.run(IdleCleanupThread.java:70)
>
> "Seeding thread" daemon prio=10 tid=0xde728000 nid=0x1f55 in Object.wait()
> [0xdcfcb000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:250)
> at
> org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:237)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1382)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
> at
> org.apache.manifoldcf.crawler.jobs.JobManager.getJobsReadyForSeeding(JobManager.java:5706)
> at
> org.apache.manifoldcf.crawler.system.SeedingThread.run(SeedingThread.java:94)
>
> "Job reset thread" daemon prio=10 tid=0xde726800 nid=0x1f53 in Object.wait()
> [0xdd06d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
> at
> org.apache.manifoldcf.crawler.jobs.JobManager.finishJobStops(JobManager.java:6673)
> at
> org.apache.manifoldcf.crawler.system.JobResetThread.run(JobResetThread.java:68)
>
> "Document delete thread '9'" daemon prio=10 tid=0xde725000 nid=0x1f52 in
> Object.wait() [0xdd0be000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb3068> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
> - locked <0xe3fb3068> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '8'" daemon prio=10 tid=0xde724000 nid=0x1f51 in
> Object.wait() [0xdd2f5000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb3068> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
> - locked <0xe3fb3068> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '7'" daemon prio=10 tid=0xde723400 nid=0x1f50 in
> Object.wait() [0xdd10f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb3068> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
> - locked <0xe3fb3068> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '6'" daemon prio=10 tid=0xde721c00 nid=0x1f4f in
> Object.wait() [0xdd160000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb3068> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
> - locked <0xe3fb3068> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '5'" daemon prio=10 tid=0xde720400 nid=0x1f4e in
> Object.wait() [0xdd1b1000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb3068> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
> - locked <0xe3fb3068> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '4'" daemon prio=10 tid=0xde71ec00 nid=0x1f4d in
> Object.wait() [0xdd439000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb3068> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
> - locked <0xe3fb3068> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '3'" daemon prio=10 tid=0xde71e000 nid=0x1f4c in
> Object.wait() [0xdd202000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb3068> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
> - locked <0xe3fb3068> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '2'" daemon prio=10 tid=0xde71c800 nid=0x1f4b in
> Object.wait() [0xdd253000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb3068> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
> - locked <0xe3fb3068> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '1'" daemon prio=10 tid=0xde71b000 nid=0x1f48 in
> Object.wait() [0xdd346000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb3068> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
> - locked <0xe3fb3068> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '0'" daemon prio=10 tid=0xde719800 nid=0x1f47 in
> Object.wait() [0xdd397000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb3068> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
> - locked <0xe3fb3068> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete stuffer thread" daemon prio=10 tid=0xde718400 nid=0x1f44 in
> Object.wait() [0xdd48a000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:919)
> - locked <0xf2706a80> (a java.lang.Integer)
> at
> org.apache.manifoldcf.crawler.system.DocumentDeleteStufferThread.run(DocumentDeleteStufferThread.java:112)
>
> "Document cleanup thread '9'" daemon prio=10 tid=0xde717000 nid=0x1f43 in
> Object.wait() [0xdd4db000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb7778> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fb7778> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '8'" daemon prio=10 tid=0xde715c00 nid=0x1f42 in
> Object.wait() [0xdd52c000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb7778> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fb7778> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '7'" daemon prio=10 tid=0xde715400 nid=0x1f41 in
> Object.wait() [0xdde22000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb7778> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fb7778> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '6'" daemon prio=10 tid=0xde713c00 nid=0x1f40 in
> Object.wait() [0xdd57d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb7778> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fb7778> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '5'" daemon prio=10 tid=0xde712400 nid=0x1f3f in
> Object.wait() [0xdd5ce000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb7778> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fb7778> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '4'" daemon prio=10 tid=0xde711000 nid=0x1f3e in
> Object.wait() [0xdd61f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb7778> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fb7778> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '3'" daemon prio=10 tid=0xde70f800 nid=0x1f3d in
> Object.wait() [0xdd670000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb7778> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fb7778> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '2'" daemon prio=10 tid=0xde70e000 nid=0x1f3c in
> Object.wait() [0xdd6c1000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb7778> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fb7778> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '1'" daemon prio=10 tid=0xde70c800 nid=0x1f3b in
> Object.wait() [0xdd712000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb7778> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fb7778> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '0'" daemon prio=10 tid=0xde70b000 nid=0x1f3a in
> Object.wait() [0xdd763000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fb7778> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fb7778> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup stuffer thread" daemon prio=10 tid=0xddc0a400 nid=0x1f39
> in Object.wait() [0xdd7b4000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:919)
> - locked <0xf2640178> (a java.lang.Integer)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupStufferThread.run(DocumentCleanupStufferThread.java:114)
>
> "Expiration thread '9'" daemon prio=10 tid=0xddc09400 nid=0x1f38 in
> Object.wait() [0xdd805000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fc7060> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fc7060> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '8'" daemon prio=10 tid=0xddc07c00 nid=0x1f37 in
> Object.wait() [0xdd856000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fc7060> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fc7060> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '7'" daemon prio=10 tid=0xddc06400 nid=0x1f36 in
> Object.wait() [0xdd8a7000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fc7060> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fc7060> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '6'" daemon prio=10 tid=0xddc05000 nid=0x1f35 in
> Object.wait() [0xdd8f8000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fc7060> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fc7060> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '5'" daemon prio=10 tid=0xddc03c00 nid=0x1f34 in
> Object.wait() [0xdd949000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fc7060> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fc7060> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '4'" daemon prio=10 tid=0xddc02000 nid=0x1f33 in
> Object.wait() [0xdd99a000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fc7060> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fc7060> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '3'" daemon prio=10 tid=0x0a496400 nid=0x1f32 in
> Object.wait() [0xdd9eb000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fc7060> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fc7060> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '2'" daemon prio=10 tid=0x0a494c00 nid=0x1f31 in
> Object.wait() [0xdda3c000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fc7060> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fc7060> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '1'" daemon prio=10 tid=0x0a493800 nid=0x1f30 in
> Object.wait() [0xdda8d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fc7060> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fc7060> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '0'" daemon prio=10 tid=0x0a492400 nid=0x1f2f in
> Object.wait() [0xddade000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe3fc7060> (a java.util.ArrayList)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
> - locked <0xe3fc7060> (a java.util.ArrayList)
> at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Worker thread '24'" daemon prio=10 tid=0x0a56e000 nid=0x1f2e in
> Object.wait() [0xddb2f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '23'" daemon prio=10 tid=0x0a56c400 nid=0x1f2d in
> Object.wait() [0xddb80000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '22'" daemon prio=10 tid=0x0a56ac00 nid=0x1f2c in
> Object.wait() [0xddd80000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '21'" daemon prio=10 tid=0x0a569400 nid=0x1f2b in
> Object.wait() [0xdddd1000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '20'" daemon prio=10 tid=0xde709400 nid=0x1f2a in
> Object.wait() [0xddbd1000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '19'" daemon prio=10 tid=0x0a567400 nid=0x1f28 in
> Object.wait() [0xdde73000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '18'" daemon prio=10 tid=0x0a5a5c00 nid=0x1f27 in
> Object.wait() [0xde5ba000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '17'" daemon prio=10 tid=0xde704800 nid=0x1f26 in
> Object.wait() [0xddec4000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '16'" daemon prio=10 tid=0x0a5a4800 nid=0x1f25 in
> Object.wait() [0xddf15000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '15'" daemon prio=10 tid=0x0a5a3000 nid=0x1f24 in
> Object.wait() [0xddf66000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '14'" daemon prio=10 tid=0x0a5a1800 nid=0x1f23 in
> Object.wait() [0xddfb7000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '13'" daemon prio=10 tid=0x0a59fc00 nid=0x1f22 in
> Object.wait() [0xde008000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '12'" daemon prio=10 tid=0xde703000 nid=0x1f20 in
> Object.wait() [0xde059000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '11'" daemon prio=10 tid=0x0a25c400 nid=0x1f1f in
> Object.wait() [0xde0aa000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '10'" daemon prio=10 tid=0x0a180800 nid=0x1f1e in
> Object.wait() [0xde0fb000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '9'" daemon prio=10 tid=0xde701400 nid=0x1f1d in
> Object.wait() [0xde14c000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '8'" daemon prio=10 tid=0xdecfec00 nid=0x1f1c in
> Object.wait() [0xde19d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1186)
> - locked <0xea62bb98> (a
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
> at java.lang.Thread.join(Thread.java:1239)
> at
> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:673)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
> at
> org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4089)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:1932)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.addDocumentReference(WorkerThread.java:1487)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessActivityLinkHandler.noteDiscoveredLink(WebcrawlerConnector.java:6049)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessActivityHTMLHandler.noteAHREF(WebcrawlerConnector.java:6159)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.LinkParseState.noteNonscriptTag(LinkParseState.java:44)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.FormParseState.noteNonscriptTag(FormParseState.java:52)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ScriptParseState.noteTag(ScriptParseState.java:50)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.BasicParseState.dealWithCharacter(BasicParseState.java:225)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.handleHTML(WebcrawlerConnector.java:7047)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.extractLinks(WebcrawlerConnector.java:6011)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.processDocuments(WebcrawlerConnector.java:1282)
> at
> org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:551)
>
> "Worker thread '7'" daemon prio=10 tid=0xdecfd400 nid=0x1f1b in
> Object.wait() [0xde1ee000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1186)
> - locked <0xea62cb98> (a
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
> at java.lang.Thread.join(Thread.java:1239)
> at
> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:673)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.jobs.IntrinsicLink.recordReferences(IntrinsicLink.java:263)
> at
> org.apache.manifoldcf.crawler.jobs.HopCount.doRecord(HopCount.java:409)
> at
> org.apache.manifoldcf.crawler.jobs.HopCount.recordReferences(HopCount.java:380)
> at
> org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4119)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:1932)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.addDocumentReference(WorkerThread.java:1487)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessActivityLinkHandler.noteDiscoveredLink(WebcrawlerConnector.java:6049)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessActivityHTMLHandler.noteAHREF(WebcrawlerConnector.java:6159)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.LinkParseState.noteNonscriptTag(LinkParseState.java:44)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.FormParseState.noteNonscriptTag(FormParseState.java:52)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ScriptParseState.noteTag(ScriptParseState.java:50)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.BasicParseState.dealWithCharacter(BasicParseState.java:225)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.handleHTML(WebcrawlerConnector.java:7047)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.extractLinks(WebcrawlerConnector.java:6011)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.processDocuments(WebcrawlerConnector.java:1282)
> at
> org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:551)
>
> "Worker thread '6'" daemon prio=10 tid=0xdecfbc00 nid=0x1f1a in
> Object.wait() [0xde23f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '5'" daemon prio=10 tid=0xdecfa400 nid=0x1f19 in
> Object.wait() [0xde290000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '4'" daemon prio=10 tid=0xdecf6c00 nid=0x1f18 in
> Object.wait() [0xde2e1000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
> - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
> at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
> at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
> at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
Re: max_pred_locks_per_transaction
Posted by Erlend Garåsen <e....@usit.uio.no>.
On 04.02.13 14.40, Karl Wright wrote:
> There's code now committed to trunk which might help diagnose this
> problem. It's part of CONNECTORS-638 and is meant to find database
> connection handle leaks.
Thanks! I will deploy trunk on our test server and start a new crawl
later today.
BTW, here's the thread dump from the Agent - generaed by usint kill
-QUIT <pid>:
"Thread-5386784" daemon prio=10 tid=0xdbb67000 nid=0x318f runnable
[0xdc6fe000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
- locked <0xe41eeea0> (a java.lang.Object)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
- locked <0xe41f0a48> (a com.sun.net.ssl.internal.ssl.AppInputStream)
at
org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:145)
at
org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114)
at
org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:274)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
- locked <0xe41efe00> (a org.postgresql.core.v3.QueryExecutorImpl)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
at org.apache.manifoldcf.core.database.Database.execute(Database.java:826)
at
org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641)
"Thread-5386763" daemon prio=10 tid=0xd8202800 nid=0x3171 runnable
[0xdc369000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
- locked <0xe41dbac8> (a java.lang.Object)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
- locked <0xe41dcd78> (a com.sun.net.ssl.internal.ssl.AppInputStream)
at
org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:145)
at
org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114)
at
org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:274)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
- locked <0xe41dd280> (a org.postgresql.core.v3.QueryExecutorImpl)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)
at org.apache.manifoldcf.core.database.Database.execute(Database.java:840)
at
org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641)
"MultiThreadedHttpConnectionManager cleanup" daemon prio=10
tid=0xddc92800 nid=0x1ca in Object.wait() [0xdccad000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0xe418ff40> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(Unknown
Source)
"Idle cleanup thread" daemon prio=10 tid=0xde729800 nid=0x1f56 in
Object.wait() [0xdd3e8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:919)
- locked <0xf272fe90> (a java.lang.Integer)
at
org.apache.manifoldcf.crawler.system.IdleCleanupThread.run(IdleCleanupThread.java:70)
"Seeding thread" daemon prio=10 tid=0xde728000 nid=0x1f55 in
Object.wait() [0xdcfcb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:250)
at
org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:237)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1382)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
at
org.apache.manifoldcf.crawler.jobs.JobManager.getJobsReadyForSeeding(JobManager.java:5706)
at
org.apache.manifoldcf.crawler.system.SeedingThread.run(SeedingThread.java:94)
"Job reset thread" daemon prio=10 tid=0xde726800 nid=0x1f53 in
Object.wait() [0xdd06d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
at
org.apache.manifoldcf.crawler.jobs.JobManager.finishJobStops(JobManager.java:6673)
at
org.apache.manifoldcf.crawler.system.JobResetThread.run(JobResetThread.java:68)
"Document delete thread '9'" daemon prio=10 tid=0xde725000 nid=0x1f52 in
Object.wait() [0xdd0be000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb3068> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0xe3fb3068> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
"Document delete thread '8'" daemon prio=10 tid=0xde724000 nid=0x1f51 in
Object.wait() [0xdd2f5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb3068> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0xe3fb3068> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
"Document delete thread '7'" daemon prio=10 tid=0xde723400 nid=0x1f50 in
Object.wait() [0xdd10f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb3068> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0xe3fb3068> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
"Document delete thread '6'" daemon prio=10 tid=0xde721c00 nid=0x1f4f in
Object.wait() [0xdd160000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb3068> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0xe3fb3068> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
"Document delete thread '5'" daemon prio=10 tid=0xde720400 nid=0x1f4e in
Object.wait() [0xdd1b1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb3068> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0xe3fb3068> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
"Document delete thread '4'" daemon prio=10 tid=0xde71ec00 nid=0x1f4d in
Object.wait() [0xdd439000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb3068> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0xe3fb3068> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
"Document delete thread '3'" daemon prio=10 tid=0xde71e000 nid=0x1f4c in
Object.wait() [0xdd202000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb3068> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0xe3fb3068> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
"Document delete thread '2'" daemon prio=10 tid=0xde71c800 nid=0x1f4b in
Object.wait() [0xdd253000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb3068> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0xe3fb3068> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
"Document delete thread '1'" daemon prio=10 tid=0xde71b000 nid=0x1f48 in
Object.wait() [0xdd346000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb3068> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0xe3fb3068> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
"Document delete thread '0'" daemon prio=10 tid=0xde719800 nid=0x1f47 in
Object.wait() [0xdd397000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb3068> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
- locked <0xe3fb3068> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
"Document delete stuffer thread" daemon prio=10 tid=0xde718400
nid=0x1f44 in Object.wait() [0xdd48a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:919)
- locked <0xf2706a80> (a java.lang.Integer)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteStufferThread.run(DocumentDeleteStufferThread.java:112)
"Document cleanup thread '9'" daemon prio=10 tid=0xde717000 nid=0x1f43
in Object.wait() [0xdd4db000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb7778> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fb7778> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
"Document cleanup thread '8'" daemon prio=10 tid=0xde715c00 nid=0x1f42
in Object.wait() [0xdd52c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb7778> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fb7778> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
"Document cleanup thread '7'" daemon prio=10 tid=0xde715400 nid=0x1f41
in Object.wait() [0xdde22000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb7778> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fb7778> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
"Document cleanup thread '6'" daemon prio=10 tid=0xde713c00 nid=0x1f40
in Object.wait() [0xdd57d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb7778> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fb7778> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
"Document cleanup thread '5'" daemon prio=10 tid=0xde712400 nid=0x1f3f
in Object.wait() [0xdd5ce000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb7778> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fb7778> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
"Document cleanup thread '4'" daemon prio=10 tid=0xde711000 nid=0x1f3e
in Object.wait() [0xdd61f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb7778> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fb7778> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
"Document cleanup thread '3'" daemon prio=10 tid=0xde70f800 nid=0x1f3d
in Object.wait() [0xdd670000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb7778> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fb7778> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
"Document cleanup thread '2'" daemon prio=10 tid=0xde70e000 nid=0x1f3c
in Object.wait() [0xdd6c1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb7778> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fb7778> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
"Document cleanup thread '1'" daemon prio=10 tid=0xde70c800 nid=0x1f3b
in Object.wait() [0xdd712000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb7778> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fb7778> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
"Document cleanup thread '0'" daemon prio=10 tid=0xde70b000 nid=0x1f3a
in Object.wait() [0xdd763000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fb7778> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fb7778> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
"Document cleanup stuffer thread" daemon prio=10 tid=0xddc0a400
nid=0x1f39 in Object.wait() [0xdd7b4000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:919)
- locked <0xf2640178> (a java.lang.Integer)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupStufferThread.run(DocumentCleanupStufferThread.java:114)
"Expiration thread '9'" daemon prio=10 tid=0xddc09400 nid=0x1f38 in
Object.wait() [0xdd805000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fc7060> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fc7060> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
"Expiration thread '8'" daemon prio=10 tid=0xddc07c00 nid=0x1f37 in
Object.wait() [0xdd856000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fc7060> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fc7060> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
"Expiration thread '7'" daemon prio=10 tid=0xddc06400 nid=0x1f36 in
Object.wait() [0xdd8a7000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fc7060> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fc7060> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
"Expiration thread '6'" daemon prio=10 tid=0xddc05000 nid=0x1f35 in
Object.wait() [0xdd8f8000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fc7060> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fc7060> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
"Expiration thread '5'" daemon prio=10 tid=0xddc03c00 nid=0x1f34 in
Object.wait() [0xdd949000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fc7060> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fc7060> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
"Expiration thread '4'" daemon prio=10 tid=0xddc02000 nid=0x1f33 in
Object.wait() [0xdd99a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fc7060> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fc7060> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
"Expiration thread '3'" daemon prio=10 tid=0x0a496400 nid=0x1f32 in
Object.wait() [0xdd9eb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fc7060> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fc7060> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
"Expiration thread '2'" daemon prio=10 tid=0x0a494c00 nid=0x1f31 in
Object.wait() [0xdda3c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fc7060> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fc7060> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
"Expiration thread '1'" daemon prio=10 tid=0x0a493800 nid=0x1f30 in
Object.wait() [0xdda8d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fc7060> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fc7060> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
"Expiration thread '0'" daemon prio=10 tid=0x0a492400 nid=0x1f2f in
Object.wait() [0xddade000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe3fc7060> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
- locked <0xe3fc7060> (a java.util.ArrayList)
at
org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
"Worker thread '24'" daemon prio=10 tid=0x0a56e000 nid=0x1f2e in
Object.wait() [0xddb2f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '23'" daemon prio=10 tid=0x0a56c400 nid=0x1f2d in
Object.wait() [0xddb80000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '22'" daemon prio=10 tid=0x0a56ac00 nid=0x1f2c in
Object.wait() [0xddd80000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '21'" daemon prio=10 tid=0x0a569400 nid=0x1f2b in
Object.wait() [0xdddd1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '20'" daemon prio=10 tid=0xde709400 nid=0x1f2a in
Object.wait() [0xddbd1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '19'" daemon prio=10 tid=0x0a567400 nid=0x1f28 in
Object.wait() [0xdde73000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '18'" daemon prio=10 tid=0x0a5a5c00 nid=0x1f27 in
Object.wait() [0xde5ba000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '17'" daemon prio=10 tid=0xde704800 nid=0x1f26 in
Object.wait() [0xddec4000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '16'" daemon prio=10 tid=0x0a5a4800 nid=0x1f25 in
Object.wait() [0xddf15000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '15'" daemon prio=10 tid=0x0a5a3000 nid=0x1f24 in
Object.wait() [0xddf66000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '14'" daemon prio=10 tid=0x0a5a1800 nid=0x1f23 in
Object.wait() [0xddfb7000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '13'" daemon prio=10 tid=0x0a59fc00 nid=0x1f22 in
Object.wait() [0xde008000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '12'" daemon prio=10 tid=0xde703000 nid=0x1f20 in
Object.wait() [0xde059000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '11'" daemon prio=10 tid=0x0a25c400 nid=0x1f1f in
Object.wait() [0xde0aa000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '10'" daemon prio=10 tid=0x0a180800 nid=0x1f1e in
Object.wait() [0xde0fb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '9'" daemon prio=10 tid=0xde701400 nid=0x1f1d in
Object.wait() [0xde14c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '8'" daemon prio=10 tid=0xdecfec00 nid=0x1f1c in
Object.wait() [0xde19d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1186)
- locked <0xea62bb98> (a
org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
at java.lang.Thread.join(Thread.java:1239)
at
org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:673)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
at
org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4089)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:1932)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.addDocumentReference(WorkerThread.java:1487)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessActivityLinkHandler.noteDiscoveredLink(WebcrawlerConnector.java:6049)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessActivityHTMLHandler.noteAHREF(WebcrawlerConnector.java:6159)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.LinkParseState.noteNonscriptTag(LinkParseState.java:44)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.FormParseState.noteNonscriptTag(FormParseState.java:52)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ScriptParseState.noteTag(ScriptParseState.java:50)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.BasicParseState.dealWithCharacter(BasicParseState.java:225)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.handleHTML(WebcrawlerConnector.java:7047)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.extractLinks(WebcrawlerConnector.java:6011)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.processDocuments(WebcrawlerConnector.java:1282)
at
org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:551)
"Worker thread '7'" daemon prio=10 tid=0xdecfd400 nid=0x1f1b in
Object.wait() [0xde1ee000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1186)
- locked <0xea62cb98> (a
org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
at java.lang.Thread.join(Thread.java:1239)
at
org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:673)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.jobs.IntrinsicLink.recordReferences(IntrinsicLink.java:263)
at org.apache.manifoldcf.crawler.jobs.HopCount.doRecord(HopCount.java:409)
at
org.apache.manifoldcf.crawler.jobs.HopCount.recordReferences(HopCount.java:380)
at
org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4119)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:1932)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.addDocumentReference(WorkerThread.java:1487)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessActivityLinkHandler.noteDiscoveredLink(WebcrawlerConnector.java:6049)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessActivityHTMLHandler.noteAHREF(WebcrawlerConnector.java:6159)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.LinkParseState.noteNonscriptTag(LinkParseState.java:44)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.FormParseState.noteNonscriptTag(FormParseState.java:52)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ScriptParseState.noteTag(ScriptParseState.java:50)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.BasicParseState.dealWithCharacter(BasicParseState.java:225)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.handleHTML(WebcrawlerConnector.java:7047)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.extractLinks(WebcrawlerConnector.java:6011)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.processDocuments(WebcrawlerConnector.java:1282)
at
org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:551)
"Worker thread '6'" daemon prio=10 tid=0xdecfbc00 nid=0x1f1a in
Object.wait() [0xde23f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '5'" daemon prio=10 tid=0xdecfa400 nid=0x1f19 in
Object.wait() [0xde290000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '4'" daemon prio=10 tid=0xdecf6c00 nid=0x1f18 in
Object.wait() [0xde2e1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '3'" daemon prio=10 tid=0xdecf5000 nid=0x1f17 in
Object.wait() [0xde332000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '2'" daemon prio=10 tid=0x0a0f7400 nid=0x1f16 in
Object.wait() [0xde383000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '1'" daemon prio=10 tid=0x0a0f5800 nid=0x1f15 in
Object.wait() [0xde3d4000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Worker thread '0'" daemon prio=10 tid=0x0a0f4000 nid=0x1f14 in
Object.wait() [0xde425000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at
org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
at
org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
at
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
"Set priority thread" daemon prio=10 tid=0x0a0f3000 nid=0x1f13 in
Object.wait() [0xde476000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:919)
- locked <0xeec0af68> (a java.lang.Integer)
at
org.apache.manifoldcf.crawler.system.SetPriorityThread.run(SetPriorityThread.java:148)
"Expire stuffer thread" daemon prio=10 tid=0x0a0bbc00 nid=0x1f12 in
Object.wait() [0xde4c7000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:250)
at
org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:237)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1382)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:827)
at
org.apache.manifoldcf.crawler.jobs.JobManager.getExpiredDocuments(JobManager.java:1698)
at
org.apache.manifoldcf.crawler.system.ExpireStufferThread.run(ExpireStufferThread.java:118)
"Stuffer thread" daemon prio=10 tid=0x0a0d6800 nid=0x1f10 in
Object.wait() [0xde518000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:919)
- locked <0xf1d0dc50> (a java.lang.Integer)
at
org.apache.manifoldcf.crawler.system.StufferThread.run(StufferThread.java:131)
"Job delete thread" daemon prio=10 tid=0x0a0d5000 nid=0x1f0f in
Object.wait() [0xde569000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:250)
at
org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:237)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1382)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
at
org.apache.manifoldcf.crawler.jobs.JobManager.deleteJobsReadyForDelete(JobManager.java:6364)
at
org.apache.manifoldcf.crawler.system.JobDeleteThread.run(JobDeleteThread.java:72)
"Job notification thread" daemon prio=10 tid=0x0a0ba800 nid=0x1f0d in
Object.wait() [0xde60b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:250)
at
org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:237)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1382)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
at
org.apache.manifoldcf.crawler.jobs.JobManager.getJobsReadyForInactivity(JobManager.java:6555)
at
org.apache.manifoldcf.crawler.system.JobNotificationThread.run(JobNotificationThread.java:69)
"Finisher thread" daemon prio=10 tid=0x0a0b9800 nid=0x1f0c in
Object.wait() [0xde65c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:250)
at
org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:237)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1382)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
at
org.apache.manifoldcf.crawler.jobs.JobManager.finishJobs(JobManager.java:6469)
at
org.apache.manifoldcf.crawler.system.FinisherThread.run(FinisherThread.java:64)
"Delete startup thread" daemon prio=10 tid=0x0a0b8800 nid=0x1f0b in
Object.wait() [0xde6ad000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:250)
at
org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:237)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1382)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
at
org.apache.manifoldcf.crawler.jobs.JobManager.getJobsReadyForDelete(JobManager.java:5786)
at
org.apache.manifoldcf.crawler.system.StartDeleteThread.run(StartDeleteThread.java:83)
"Startup thread" daemon prio=10 tid=0x0a0b7800 nid=0x1f0a in
Object.wait() [0xde6fe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:250)
at
org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:237)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1382)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
at
org.apache.manifoldcf.crawler.jobs.JobManager.getJobsReadyForStartup(JobManager.java:5856)
at
org.apache.manifoldcf.crawler.system.StartupThread.run(StartupThread.java:87)
"Job start thread" daemon prio=10 tid=0x0a20d000 nid=0x1f09 in
Object.wait() [0xde860000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:250)
at
org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:237)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1382)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
at
org.apache.manifoldcf.crawler.jobs.JobManager.startJobs(JobManager.java:4711)
at
org.apache.manifoldcf.crawler.system.JobStartThread.run(JobStartThread.java:68)
"Connection pool reaper" daemon prio=10 tid=0xdeee3800 nid=0x1ee2
waiting on condition [0xdedf6000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPoolManager$ConnectionCloserThread.run(ConnectionPoolManager.java:130)
"Low Memory Detector" daemon prio=10 tid=0xdf104800 nid=0x1ee0 runnable
[0x00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0xdf102800 nid=0x1edf waiting on
condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0xdf100800 nid=0x1ede waiting on
condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x09f54800 nid=0x1edd waiting on
condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x09f46000 nid=0x1edc in Object.wait()
[0xdf56d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0xe3fe4ad8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x09f41800 nid=0x1edb in
Object.wait() [0xdf5be000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0xe3fb9178> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x09e01c00 nid=0x1ecc in Object.wait() [0xf7148000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
- locked <0xe3fe96e8> (a
org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
at
org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
at
org.apache.manifoldcf.agents.agentmanager.AgentManager.getAllAgents(AgentManager.java:218)
at
org.apache.manifoldcf.agents.system.ManifoldCF.startAgents(ManifoldCF.java:147)
- locked <0xe3fd4028> (a java.util.HashMap)
at org.apache.manifoldcf.agents.AgentRun.doExecute(AgentRun.java:64)
at
org.apache.manifoldcf.agents.BaseAgentsInitializationCommand.execute(BaseAgentsInitializationCommand.java:37)
at org.apache.manifoldcf.agents.AgentRun.main(AgentRun.java:101)
"VM Thread" prio=10 tid=0x09f3d800 nid=0x1eda runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x09e09000 nid=0x1ecd runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x09e0a800 nid=0x1ece runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x09e0bc00 nid=0x1ecf runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x09e0d400 nid=0x1ed0 runnable
"GC task thread#4 (ParallelGC)" prio=10 tid=0x09e0ec00 nid=0x1ed1 runnable
"GC task thread#5 (ParallelGC)" prio=10 tid=0x09e10000 nid=0x1ed2 runnable
"GC task thread#6 (ParallelGC)" prio=10 tid=0x09e11800 nid=0x1ed3 runnable
"GC task thread#7 (ParallelGC)" prio=10 tid=0x09e12c00 nid=0x1ed4 runnable
"GC task thread#8 (ParallelGC)" prio=10 tid=0x09e14400 nid=0x1ed5 runnable
"GC task thread#9 (ParallelGC)" prio=10 tid=0x09e15800 nid=0x1ed6 runnable
"GC task thread#10 (ParallelGC)" prio=10 tid=0x09e17000 nid=0x1ed7 runnable
"GC task thread#11 (ParallelGC)" prio=10 tid=0x09e18800 nid=0x1ed8 runnable
"GC task thread#12 (ParallelGC)" prio=10 tid=0x09e19c00 nid=0x1ed9 runnable
"VM Periodic Task Thread" prio=10 tid=0xdf116800 nid=0x1ee1 waiting on
condition
JNI global references: 1044
Heap
PSYoungGen total 86656K, used 62881K [0xeea60000, 0xf3fb0000,
0xf3fb0000)
eden space 85952K, 72% used [0xeea60000,0xf2730530,0xf3e50000)
from space 704K, 86% used [0xf3e50000,0xf3ee8000,0xf3f00000)
to space 704K, 0% used [0xf3f00000,0xf3f00000,0xf3fb0000)
PSOldGen total 174784K, used 106478K [0xe3fb0000, 0xeea60000,
0xeea60000)
object space 174784K, 60% used [0xe3fb0000,0xea7abb78,0xeea60000)
PSPermGen total 16384K, used 12215K [0xdffb0000, 0xe0fb0000,
0xe3fb0000)
object space 16384K, 74% used [0xdffb0000,0xe0b9dc00,0xe0fb0000)
Erlend
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Karl Wright <da...@gmail.com>.
There's code now committed to trunk which might help diagnose this
problem. It's part of CONNECTORS-638 and is meant to find database
connection handle leaks.
Karl
On Fri, Feb 1, 2013 at 8:27 AM, Karl Wright <da...@gmail.com> wrote:
> Hi Erlend,
>
> The WAITING in the thread dump has nothing to do with PostgreSQL
> locks. In fact, these thread dumps show absolutely no database
> activity EXCEPT for one thread which is trying to close idle
> connections.
>
> If you are using the multiprocess model, you should get a thread dump
> of the agents process instead; that would be far more meaningful. It
> would help us figure out why this:
>
>>>>>>>
> This is what the Postgres admins told me today. The idle processes
> (idle in transaction) are:
>
> - - SELECT id,status,checktime FROM jobqueue WHERE dochash=$1 AND
> jobid=$2 FOR UPDATE
> - - UPDATE hopcount SET deathmark=$1 WHERE jobid=$2 AND deathmark=$3
> <<<<<<
>
> ... is occurring. I can think of no reason why the above should
> deadlock, which would be one possibility. The other possibility is
> that Postgresql operations are simply timing out on the server side
> and the JDBC client is unaware of that.
>
> In any case, the agents thread dump will help enormously.
>
> Karl
>
>
> On Fri, Feb 1, 2013 at 7:01 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>>
>> Sorry, wrong thread dump. This is the correct one. The previous one was
>> generated after I stopped the services. I guess entries like this one is
>> related to the locks?
>> "resin-118" id=118 WAITING
>>
>> ----8<----
>>
>> [2013-01-31 19:00:00.014] {resin-18} Thread Dump generated Thu Jan 31
>> 18:59:59 CET 2013
>> "Signal Dispatcher" id=4 RUNNABLE
>>
>> "resin-18" id=18 RUNNABLE
>>
>> at
>> sun.management.ThreadImpl.getThreadInfo1 (ThreadImpl.java) (native)
>> at
>> sun.management.ThreadImpl.getThreadInfo (ThreadImpl.java:156)
>> at
>> com.caucho.util.ThreadDump.getThreadDump (ThreadDump.java:121)
>> at
>> com.caucho.admin.action.ThreadDumpAction.execute (ThreadDumpAction.java:52)
>> at
>> com.caucho.health.action.DumpThreads.doActionImpl (DumpThreads.java:99)
>> at
>> com.caucho.health.action.AbstractHealthAction.doAction
>> (AbstractHealthAction.java:90)
>> at
>> com.caucho.env.health.HealthService.executeActions (HealthService.java:722)
>> at
>> com.caucho.env.health.HealthService$ExecuteActionsTask.run
>> (HealthService.java:1030)
>> at
>> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
>> at
>> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>>
>> "resin-148" id=148 RUNNABLE
>> at
>> java.lang.Throwable.fillInStackTrace (Throwable.java) (native)
>> at java.lang.Throwable.<init>
>> (Throwable.java:196)
>> at java.lang.Exception.<init>
>> (Exception.java:41)
>> at
>> javax.management.JMException.<init> (JMException.java:35)
>> at
>> javax.management.OperationsException.<init> (OperationsException.java:36)
>> at
>> javax.management.InstanceNotFoundException.<init>
>> (InstanceNotFoundException.java:34)
>> at
>> com.caucho.jmx.AbstractMBeanServer.getAttribute
>> (AbstractMBeanServer.java:594)
>> at
>> com.caucho.env.meter.JmxExpr$JmxDelta.sample (JmxExpr.java:211)
>> at
>> com.caucho.env.meter.JmxExpr$JmxContainerExpr.sample (JmxExpr.java:94)
>> at
>> com.caucho.env.meter.JmxExpr$JmxContainerExpr.sample (JmxExpr.java:94)
>> at
>> com.caucho.env.meter.JmxCalculationMeterImpl.sample
>> (JmxCalculationMeterImpl.java:62)
>> at
>> com.caucho.server.admin.StatSystem$Sample.sample (StatSystem.java:244)
>> at
>> com.caucho.server.admin.ProStatSystem.sampleData (ProStatSystem.java:582)
>> at
>> com.caucho.server.admin.ProStatSystem.sample (ProStatSystem.java:558)
>>
>> at
>> com.caucho.server.admin.ProStatSystem.handleAlarm (ProStatSystem.java:646)
>> at com.caucho.util.Alarm.handleAlarm
>> (Alarm.java:523)
>> at com.caucho.util.Alarm.run
>> (Alarm.java:495)
>> at
>> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
>> at
>> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>>
>> "resin-125" id=125 RUNNABLE
>> at
>> com.caucho.vfs.JniMemoryMappedFile.nativeRead (JniMemoryMappedFile.java)
>> (native)
>> at
>> com.caucho.vfs.JniMemoryMappedFile.read (JniMemoryMappedFile.java:136)
>> at
>> com.caucho.db.block.BlockReadWrite.readBlockImpl (BlockReadWrite.java:210)
>> at
>> com.caucho.db.block.BlockReadWrite.readBlock (BlockReadWrite.java:177)
>> at com.caucho.db.block.Block.read
>> (Block.java:259)
>> at
>> com.caucho.db.block.BlockStore.readBlock (BlockStore.java:569)
>> at
>> com.caucho.db.xa.DbTransaction.readBlock (DbTransaction.java:332)
>> at
>> com.caucho.db.table.TableIterator.nextBlock (TableIterator.java:276)
>> at
>> com.caucho.db.sql.RowIterateExpr.nextBlock (RowIterateExpr.java:87)
>> at com.caucho.db.sql.Query.nextBlock
>> (Query.java:788)
>> at com.caucho.db.sql.Query.start
>> (Query.java:726)
>> at
>> com.caucho.db.sql.DeleteQuery.execute (DeleteQuery.java:74)
>> at
>> com.caucho.db.jdbc.PreparedStatementImpl.execute
>> (PreparedStatementImpl.java:365)
>> at
>> com.caucho.db.jdbc.PreparedStatementImpl.executeUpdate
>> (PreparedStatementImpl.java:335)
>> at
>> com.caucho.server.admin.StatDatabase$ReaperTask.handleAlarm
>> (StatDatabase.java:941)
>>
>> at com.caucho.util.Alarm.handleAlarm
>> (Alarm.java:523)
>> at com.caucho.util.Alarm.run
>> (Alarm.java:495)
>> at
>> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
>> at
>> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>>
>> "resin-port-127.0.0.1:6945-16" id=16
>> RUNNABLE (in native)
>>
>> "resin-port-127.0.0.1:6945-17" id=17
>> RUNNABLE (in native)
>>
>> "resin-port-127.0.0.1:6945-19" id=19
>> RUNNABLE (in native)
>>
>>
>> "resin-port-127.0.0.1:6945-31" id=31
>> RUNNABLE (in native)
>>
>> "resin-port-127.0.0.1:6945-37" id=37
>> RUNNABLE (in native)
>> at
>> com.caucho.vfs.JniSocketImpl.nativeAccept (JniSocketImpl.java) (native)
>> at
>> com.caucho.vfs.JniSocketImpl.accept (JniSocketImpl.java:141)
>> at
>> com.caucho.vfs.JniServerSocketImpl.accept (JniServerSocketImpl.java:254)
>> at
>> com.caucho.network.listen.TcpSocketLinkListener.accept
>> (TcpSocketLinkListener.java:1372)
>> at
>> com.caucho.network.listen.TcpSocketLink.accept (TcpSocketLink.java:978)
>> at
>> com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl
>> (TcpSocketLink.java:928)
>> at
>> com.caucho.network.listen.ConnectionTask.runThread (ConnectionTask.java:117)
>> at
>> com.caucho.network.listen.ConnectionTask.run (ConnectionTask.java:93)
>> at
>> com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks
>> (SocketLinkThreadLauncher.java:169)
>> at
>> com.caucho.network.listen.TcpSocketAcceptThread.run
>> (TcpSocketAcceptThread.java:61)
>> at
>> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
>> at
>> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>>
>> "resin-port-127.0.0.1:6945-13" id=13
>> RUNNABLE (in native)
>>
>> "resin-port-127.0.0.1:6945-14" id=14
>> RUNNABLE (in native)
>>
>> "resin-port-127.0.0.1:6945-15" id=15
>> RUNNABLE (in native)
>>
>> "resin-port-127.0.0.1:6945-22" id=22
>> RUNNABLE (in native)
>>
>> "resin-port-127.0.0.1:6945-24" id=24
>> RUNNABLE (in native)
>>
>> "resin-port-127.0.0.1:6945-26" id=26
>> RUNNABLE (in native)
>>
>> "resin-port-127.0.0.1:6945-30" id=30
>> RUNNABLE (in native)
>>
>> "resin-port-127.0.0.1:6945-41" id=41
>> RUNNABLE (in native)
>>
>> "resin-port-127.0.0.1:6945-88" id=88
>> RUNNABLE (in native)
>>
>> "resin-port-127.0.0.1:6945-90" id=90
>> RUNNABLE (in native)
>>
>> "resin-port-127.0.0.1:6945-98" id=98
>> RUNNABLE (in native)
>> at
>> com.caucho.vfs.JniSocketImpl.nativeAccept (JniSocketImpl.java) (native)
>> at
>> com.caucho.vfs.JniSocketImpl.accept (JniSocketImpl.java:141)
>> at
>> com.caucho.vfs.JniServerSocketImpl.accept (JniServerSocketImpl.java:254)
>> at
>> com.caucho.network.listen.TcpSocketLinkListener.accept
>> (TcpSocketLinkListener.java:1372)
>> at
>> com.caucho.network.listen.TcpSocketLink.accept (TcpSocketLink.java:978)
>> at
>> com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl
>> (TcpSocketLink.java:928)
>> at
>> com.caucho.network.listen.TcpSocketLink.handleAcceptTask
>> (TcpSocketLink.java:902)
>> at
>> com.caucho.network.listen.ConnectionTask.runThread (ConnectionTask.java:114)
>> at
>> com.caucho.network.listen.ConnectionTask.run (ConnectionTask.java:93)
>> at
>> com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks
>> (SocketLinkThreadLauncher.java:169)
>> at
>> com.caucho.network.listen.TcpSocketResumeThread.run
>> (TcpSocketResumeThread.java:62)
>> at
>> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
>> at
>> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>>
>> "resin-main-link" id=33 RUNNABLE (in
>> native)
>> at
>> java.net.SocketInputStream.socketRead0 (SocketInputStream.java) (native)
>> at java.net.SocketInputStream.read
>> (SocketInputStream.java:129)
>> at java.net.SocketInputStream.read
>> (SocketInputStream.java:182)
>> at
>> com.caucho.remote.websocket.UnmaskedFrameInputStream.readFrameHeaderImpl
>> (UnmaskedFrameInputStream.java:142)
>> at
>> com.caucho.remote.websocket.FrameInputStream.readFrameHeader
>> (FrameInputStream.java:97)
>> at
>> com.caucho.remote.websocket.WebSocketInputStream.startBinaryMessage
>> (WebSocketInputStream.java:78)
>> at
>> com.caucho.hmtp.HmtpWebSocketReader.readPacket (HmtpWebSocketReader.java:70)
>> at com.caucho.hmtp.HmtpLinkWorker.run
>> (HmtpLinkWorker.java:114)
>> at
>> com.caucho.server.resin.ResinLink.run (ResinLink.java:71)
>> at
>> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
>> at
>> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>>
>> "resin-select-manager-0" id=36 RUNNABLE
>> (in native)
>> at
>> com.caucho.network.listen.JniSelectManager.selectNative
>> (JniSelectManager.java) (native)
>> at
>> com.caucho.network.listen.JniSelectManager.runImpl
>> (JniSelectManager.java:392)
>> at
>> com.caucho.network.listen.JniSelectManager.run (JniSelectManager.java:359)
>> at java.lang.Thread.run
>> (Thread.java:662)
>>
>> "resin-fail-safe-halt" id=34 WAITING
>> at sun.misc.Unsafe.park (Unsafe.java)
>> (native)
>> at
>> java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
>> at
>> com.caucho.env.shutdown.ShutdownSystem$FailSafeHaltThread.run
>> (ShutdownSystem.java:499)
>>
>> "resin-shutdown" id=35 WAITING
>> [2013-01-31 19:00:00.014] {resin-18} at sun.misc.Unsafe.park (Unsafe.java)
>> (native)
>>
>> at
>> java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
>> at
>> com.caucho.env.shutdown.ShutdownSystem$ShutdownThread.run
>> (ShutdownSystem.java:446)
>>
>> "resin-118" id=118 WAITING
>>
>> "resin-134" id=134 WAITING
>>
>> "resin-141" id=141 WAITING
>>
>> "resin-144" id=144 WAITING
>>
>> "resin-154" id=154 WAITING
>>
>> "resin-156" id=156 WAITING
>>
>> "resin-163" id=163 WAITING
>>
>> "resin-167" id=167 WAITING
>>
>> "resin-169" id=169 WAITING
>>
>> "resin-170" id=170 WAITING
>>
>> "resin-171" id=171 WAITING
>>
>> "resin-174" id=174 WAITING
>>
>> "resin-175" id=175 WAITING
>>
>> "resin-176" id=176 WAITING
>>
>> "resin-177" id=177 WAITING
>>
>> "resin-178" id=178 WAITING
>>
>> "resin-179" id=179 WAITING
>>
>> "resin-47" id=47 WAITING
>>
>> at sun.misc.Unsafe.park (Unsafe.java)
>> (native)
>> at
>> java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
>> at
>> com.caucho.env.thread2.ResinThread2.park (ResinThread2.java:196)
>> at
>> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:147)
>> at
>> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>>
>> "Finalizer" id=3 WAITING
>> waiting on
>> java.lang.ref.ReferenceQueue$Lock@4ecd51ad
>> at java.lang.Object.wait
>> (Object.java) (native)
>> at
>> java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:118)
>> at
>> java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:134)
>> at
>> java.lang.ref.Finalizer$FinalizerThread.run (Finalizer.java:159)
>>
>> "Reference Handler" id=2 WAITING
>> waiting on
>> java.lang.ref.Reference$Lock@9ad5d46
>> at java.lang.Object.wait
>> (Object.java) (native)
>> at java.lang.Object.wait
>> (Object.java:485)
>> at
>> java.lang.ref.Reference$ReferenceHandler.run (Reference.java:116)
>>
>>
>> "BlockWriter[Table[log_data:7]]-13" id=38 TIMED_WAITING
>>
>>
>> "BlockWriter[Table[stat_data_mcf_1:5]]-11" id=136 TIMED_WAITING
>>
>> "CoordinatorThread[]-3" id=12
>> TIMED_WAITING
>>
>> "FlushWorker[]-5" id=168 TIMED_WAITING
>>
>> "HealthSystemHealthCheckWorker" id=28
>> TIMED_WAITING
>>
>>
>> "resin-port-127.0.0.1:6945-launcher" id=172 TIMED_WAITING
>>
>> at sun.misc.Unsafe.park (Unsafe.java)
>> (native)
>> at
>> java.util.concurrent.locks.LockSupport.parkUntil (LockSupport.java:351)
>> at
>> com.caucho.env.thread2.AbstractTaskWorker2.run
>> (AbstractTaskWorker2.java:216)
>> at
>> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
>> at
>> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>>
>> "main" id=1 TIMED_WAITING
>> waiting on
>> com.caucho.server.resin.ResinWaitForExitService@7d2f117
>> at java.lang.Object.wait
>> (Object.java) (native)
>> at
>> com.caucho.server.resin.ResinWaitForExitService.waitForExit
>> (ResinWaitForExitService.java:135)
>> at
>> com.caucho.server.resin.Resin.waitForExit (Resin.java:1190)
>> at com.caucho.server.resin.Resin.main
>> (Resin.java:1239)
>>
>> "resin-timer" id=9 TIMED_WAITING
>> at java.lang.Thread.sleep
>> (Thread.java) (native)
>> at
>> com.caucho.util.CurrentTime$CurrentTimeThread.run (CurrentTime.java:228)
>>
>>
>> "ThreadLauncher2[ThreadPool[system]]-1" id=10 TIMED_WAITING
>>
>> "ThreadScheduleWorker[]-2" id=29
>> TIMED_WAITING
>> at sun.misc.Unsafe.park (Unsafe.java)
>> (native)
>> at
>> java.util.concurrent.locks.LockSupport.parkUntil (LockSupport.java:351)
>> at
>> com.caucho.env.thread2.AbstractTaskWorker2.run
>> (AbstractTaskWorker2.java:216)
>> at java.lang.Thread.run
>> (Thread.java:662)
>>
>> "Connection pool reaper" id=49
>> TIMED_WAITING
>> at java.lang.Thread.sleep
>> (Thread.java) (native)
>> at
>> org.apache.manifoldcf.core.jdbcpool.ConnectionPoolManager$ConnectionCloserThread.run
>> (ConnectionPoolManager.java:130)
>>
>> [2013-01-31 19:03:00.003] {resin-154} AnomalyAnalyzer JVM|Thread|JVM
>> Runnable Count WARNING
>> 3.000 sample is 147.49% of 2.034 avg,
>> 5.327 std deviations (std=0.181, n=1029.0)
>> [2013-01-31 19:09:00.002] {resin-168} AnomalyAnalyzer JVM|Thread|JVM
>> Runnable Count WARNING
>> 3.000 sample is 147.44% of 2.035 avg,
>> 5.269 std deviations (std=0.183, n=1065.0)
>> [2013-01-31 19:14:00.002] {resin-118} AnomalyAnalyzer JVM|Thread|JVM
>> Runnable Count WARNING
>> 3.000 sample is 147.43% of 2.035 avg,
>> 5.257 std deviations (std=0.184, n=1089.0)
>> [2013-02-01 11:05:00.003] {resin-449} AnomalyAnalyzer OS|Process|File
>> Descriptor Count WARNING
>> 229.000 sample is 101.32% of 226.010
>> avg, 8.674 std deviations (std=0.345, n=8554.0)
>> [2013-02-01 11:06:00.005] {resin-449} AnomalyAnalyzer JVM|Thread|JVM Waiting
>> Count WARNING
>> 37.000 sample is 114.35% of 32.357
>> avg, 5.171 std deviations (std=0.898, n=8559.0)
>> [2013-02-01 11:06:00.005] {resin-449} AnomalyAnalyzer Resin|Thread|Thread
>> Active Count WARNING
>> 18.000 sample is 81.99% of 21.953
>> avg, 6.191 std deviations (std=0.639, n=8559.0)
>> [2013-02-01 11:06:00.005] {resin-421} Health action DumpThreads[] executing
>> [2013-02-01 11:06:00.008] {resin-425} Health action DumpThreads[] executing
Re: max_pred_locks_per_transaction
Posted by Karl Wright <da...@gmail.com>.
Hi Erlend,
The WAITING in the thread dump has nothing to do with PostgreSQL
locks. In fact, these thread dumps show absolutely no database
activity EXCEPT for one thread which is trying to close idle
connections.
If you are using the multiprocess model, you should get a thread dump
of the agents process instead; that would be far more meaningful. It
would help us figure out why this:
>>>>>>
This is what the Postgres admins told me today. The idle processes
(idle in transaction) are:
- - SELECT id,status,checktime FROM jobqueue WHERE dochash=$1 AND
jobid=$2 FOR UPDATE
- - UPDATE hopcount SET deathmark=$1 WHERE jobid=$2 AND deathmark=$3
<<<<<<
... is occurring. I can think of no reason why the above should
deadlock, which would be one possibility. The other possibility is
that Postgresql operations are simply timing out on the server side
and the JDBC client is unaware of that.
In any case, the agents thread dump will help enormously.
Karl
On Fri, Feb 1, 2013 at 7:01 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> Sorry, wrong thread dump. This is the correct one. The previous one was
> generated after I stopped the services. I guess entries like this one is
> related to the locks?
> "resin-118" id=118 WAITING
>
> ----8<----
>
> [2013-01-31 19:00:00.014] {resin-18} Thread Dump generated Thu Jan 31
> 18:59:59 CET 2013
> "Signal Dispatcher" id=4 RUNNABLE
>
> "resin-18" id=18 RUNNABLE
>
> at
> sun.management.ThreadImpl.getThreadInfo1 (ThreadImpl.java) (native)
> at
> sun.management.ThreadImpl.getThreadInfo (ThreadImpl.java:156)
> at
> com.caucho.util.ThreadDump.getThreadDump (ThreadDump.java:121)
> at
> com.caucho.admin.action.ThreadDumpAction.execute (ThreadDumpAction.java:52)
> at
> com.caucho.health.action.DumpThreads.doActionImpl (DumpThreads.java:99)
> at
> com.caucho.health.action.AbstractHealthAction.doAction
> (AbstractHealthAction.java:90)
> at
> com.caucho.env.health.HealthService.executeActions (HealthService.java:722)
> at
> com.caucho.env.health.HealthService$ExecuteActionsTask.run
> (HealthService.java:1030)
> at
> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
> at
> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>
> "resin-148" id=148 RUNNABLE
> at
> java.lang.Throwable.fillInStackTrace (Throwable.java) (native)
> at java.lang.Throwable.<init>
> (Throwable.java:196)
> at java.lang.Exception.<init>
> (Exception.java:41)
> at
> javax.management.JMException.<init> (JMException.java:35)
> at
> javax.management.OperationsException.<init> (OperationsException.java:36)
> at
> javax.management.InstanceNotFoundException.<init>
> (InstanceNotFoundException.java:34)
> at
> com.caucho.jmx.AbstractMBeanServer.getAttribute
> (AbstractMBeanServer.java:594)
> at
> com.caucho.env.meter.JmxExpr$JmxDelta.sample (JmxExpr.java:211)
> at
> com.caucho.env.meter.JmxExpr$JmxContainerExpr.sample (JmxExpr.java:94)
> at
> com.caucho.env.meter.JmxExpr$JmxContainerExpr.sample (JmxExpr.java:94)
> at
> com.caucho.env.meter.JmxCalculationMeterImpl.sample
> (JmxCalculationMeterImpl.java:62)
> at
> com.caucho.server.admin.StatSystem$Sample.sample (StatSystem.java:244)
> at
> com.caucho.server.admin.ProStatSystem.sampleData (ProStatSystem.java:582)
> at
> com.caucho.server.admin.ProStatSystem.sample (ProStatSystem.java:558)
>
> at
> com.caucho.server.admin.ProStatSystem.handleAlarm (ProStatSystem.java:646)
> at com.caucho.util.Alarm.handleAlarm
> (Alarm.java:523)
> at com.caucho.util.Alarm.run
> (Alarm.java:495)
> at
> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
> at
> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>
> "resin-125" id=125 RUNNABLE
> at
> com.caucho.vfs.JniMemoryMappedFile.nativeRead (JniMemoryMappedFile.java)
> (native)
> at
> com.caucho.vfs.JniMemoryMappedFile.read (JniMemoryMappedFile.java:136)
> at
> com.caucho.db.block.BlockReadWrite.readBlockImpl (BlockReadWrite.java:210)
> at
> com.caucho.db.block.BlockReadWrite.readBlock (BlockReadWrite.java:177)
> at com.caucho.db.block.Block.read
> (Block.java:259)
> at
> com.caucho.db.block.BlockStore.readBlock (BlockStore.java:569)
> at
> com.caucho.db.xa.DbTransaction.readBlock (DbTransaction.java:332)
> at
> com.caucho.db.table.TableIterator.nextBlock (TableIterator.java:276)
> at
> com.caucho.db.sql.RowIterateExpr.nextBlock (RowIterateExpr.java:87)
> at com.caucho.db.sql.Query.nextBlock
> (Query.java:788)
> at com.caucho.db.sql.Query.start
> (Query.java:726)
> at
> com.caucho.db.sql.DeleteQuery.execute (DeleteQuery.java:74)
> at
> com.caucho.db.jdbc.PreparedStatementImpl.execute
> (PreparedStatementImpl.java:365)
> at
> com.caucho.db.jdbc.PreparedStatementImpl.executeUpdate
> (PreparedStatementImpl.java:335)
> at
> com.caucho.server.admin.StatDatabase$ReaperTask.handleAlarm
> (StatDatabase.java:941)
>
> at com.caucho.util.Alarm.handleAlarm
> (Alarm.java:523)
> at com.caucho.util.Alarm.run
> (Alarm.java:495)
> at
> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
> at
> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>
> "resin-port-127.0.0.1:6945-16" id=16
> RUNNABLE (in native)
>
> "resin-port-127.0.0.1:6945-17" id=17
> RUNNABLE (in native)
>
> "resin-port-127.0.0.1:6945-19" id=19
> RUNNABLE (in native)
>
>
> "resin-port-127.0.0.1:6945-31" id=31
> RUNNABLE (in native)
>
> "resin-port-127.0.0.1:6945-37" id=37
> RUNNABLE (in native)
> at
> com.caucho.vfs.JniSocketImpl.nativeAccept (JniSocketImpl.java) (native)
> at
> com.caucho.vfs.JniSocketImpl.accept (JniSocketImpl.java:141)
> at
> com.caucho.vfs.JniServerSocketImpl.accept (JniServerSocketImpl.java:254)
> at
> com.caucho.network.listen.TcpSocketLinkListener.accept
> (TcpSocketLinkListener.java:1372)
> at
> com.caucho.network.listen.TcpSocketLink.accept (TcpSocketLink.java:978)
> at
> com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl
> (TcpSocketLink.java:928)
> at
> com.caucho.network.listen.ConnectionTask.runThread (ConnectionTask.java:117)
> at
> com.caucho.network.listen.ConnectionTask.run (ConnectionTask.java:93)
> at
> com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks
> (SocketLinkThreadLauncher.java:169)
> at
> com.caucho.network.listen.TcpSocketAcceptThread.run
> (TcpSocketAcceptThread.java:61)
> at
> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
> at
> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>
> "resin-port-127.0.0.1:6945-13" id=13
> RUNNABLE (in native)
>
> "resin-port-127.0.0.1:6945-14" id=14
> RUNNABLE (in native)
>
> "resin-port-127.0.0.1:6945-15" id=15
> RUNNABLE (in native)
>
> "resin-port-127.0.0.1:6945-22" id=22
> RUNNABLE (in native)
>
> "resin-port-127.0.0.1:6945-24" id=24
> RUNNABLE (in native)
>
> "resin-port-127.0.0.1:6945-26" id=26
> RUNNABLE (in native)
>
> "resin-port-127.0.0.1:6945-30" id=30
> RUNNABLE (in native)
>
> "resin-port-127.0.0.1:6945-41" id=41
> RUNNABLE (in native)
>
> "resin-port-127.0.0.1:6945-88" id=88
> RUNNABLE (in native)
>
> "resin-port-127.0.0.1:6945-90" id=90
> RUNNABLE (in native)
>
> "resin-port-127.0.0.1:6945-98" id=98
> RUNNABLE (in native)
> at
> com.caucho.vfs.JniSocketImpl.nativeAccept (JniSocketImpl.java) (native)
> at
> com.caucho.vfs.JniSocketImpl.accept (JniSocketImpl.java:141)
> at
> com.caucho.vfs.JniServerSocketImpl.accept (JniServerSocketImpl.java:254)
> at
> com.caucho.network.listen.TcpSocketLinkListener.accept
> (TcpSocketLinkListener.java:1372)
> at
> com.caucho.network.listen.TcpSocketLink.accept (TcpSocketLink.java:978)
> at
> com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl
> (TcpSocketLink.java:928)
> at
> com.caucho.network.listen.TcpSocketLink.handleAcceptTask
> (TcpSocketLink.java:902)
> at
> com.caucho.network.listen.ConnectionTask.runThread (ConnectionTask.java:114)
> at
> com.caucho.network.listen.ConnectionTask.run (ConnectionTask.java:93)
> at
> com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks
> (SocketLinkThreadLauncher.java:169)
> at
> com.caucho.network.listen.TcpSocketResumeThread.run
> (TcpSocketResumeThread.java:62)
> at
> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
> at
> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>
> "resin-main-link" id=33 RUNNABLE (in
> native)
> at
> java.net.SocketInputStream.socketRead0 (SocketInputStream.java) (native)
> at java.net.SocketInputStream.read
> (SocketInputStream.java:129)
> at java.net.SocketInputStream.read
> (SocketInputStream.java:182)
> at
> com.caucho.remote.websocket.UnmaskedFrameInputStream.readFrameHeaderImpl
> (UnmaskedFrameInputStream.java:142)
> at
> com.caucho.remote.websocket.FrameInputStream.readFrameHeader
> (FrameInputStream.java:97)
> at
> com.caucho.remote.websocket.WebSocketInputStream.startBinaryMessage
> (WebSocketInputStream.java:78)
> at
> com.caucho.hmtp.HmtpWebSocketReader.readPacket (HmtpWebSocketReader.java:70)
> at com.caucho.hmtp.HmtpLinkWorker.run
> (HmtpLinkWorker.java:114)
> at
> com.caucho.server.resin.ResinLink.run (ResinLink.java:71)
> at
> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
> at
> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>
> "resin-select-manager-0" id=36 RUNNABLE
> (in native)
> at
> com.caucho.network.listen.JniSelectManager.selectNative
> (JniSelectManager.java) (native)
> at
> com.caucho.network.listen.JniSelectManager.runImpl
> (JniSelectManager.java:392)
> at
> com.caucho.network.listen.JniSelectManager.run (JniSelectManager.java:359)
> at java.lang.Thread.run
> (Thread.java:662)
>
> "resin-fail-safe-halt" id=34 WAITING
> at sun.misc.Unsafe.park (Unsafe.java)
> (native)
> at
> java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
> at
> com.caucho.env.shutdown.ShutdownSystem$FailSafeHaltThread.run
> (ShutdownSystem.java:499)
>
> "resin-shutdown" id=35 WAITING
> [2013-01-31 19:00:00.014] {resin-18} at sun.misc.Unsafe.park (Unsafe.java)
> (native)
>
> at
> java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
> at
> com.caucho.env.shutdown.ShutdownSystem$ShutdownThread.run
> (ShutdownSystem.java:446)
>
> "resin-118" id=118 WAITING
>
> "resin-134" id=134 WAITING
>
> "resin-141" id=141 WAITING
>
> "resin-144" id=144 WAITING
>
> "resin-154" id=154 WAITING
>
> "resin-156" id=156 WAITING
>
> "resin-163" id=163 WAITING
>
> "resin-167" id=167 WAITING
>
> "resin-169" id=169 WAITING
>
> "resin-170" id=170 WAITING
>
> "resin-171" id=171 WAITING
>
> "resin-174" id=174 WAITING
>
> "resin-175" id=175 WAITING
>
> "resin-176" id=176 WAITING
>
> "resin-177" id=177 WAITING
>
> "resin-178" id=178 WAITING
>
> "resin-179" id=179 WAITING
>
> "resin-47" id=47 WAITING
>
> at sun.misc.Unsafe.park (Unsafe.java)
> (native)
> at
> java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
> at
> com.caucho.env.thread2.ResinThread2.park (ResinThread2.java:196)
> at
> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:147)
> at
> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>
> "Finalizer" id=3 WAITING
> waiting on
> java.lang.ref.ReferenceQueue$Lock@4ecd51ad
> at java.lang.Object.wait
> (Object.java) (native)
> at
> java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:118)
> at
> java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:134)
> at
> java.lang.ref.Finalizer$FinalizerThread.run (Finalizer.java:159)
>
> "Reference Handler" id=2 WAITING
> waiting on
> java.lang.ref.Reference$Lock@9ad5d46
> at java.lang.Object.wait
> (Object.java) (native)
> at java.lang.Object.wait
> (Object.java:485)
> at
> java.lang.ref.Reference$ReferenceHandler.run (Reference.java:116)
>
>
> "BlockWriter[Table[log_data:7]]-13" id=38 TIMED_WAITING
>
>
> "BlockWriter[Table[stat_data_mcf_1:5]]-11" id=136 TIMED_WAITING
>
> "CoordinatorThread[]-3" id=12
> TIMED_WAITING
>
> "FlushWorker[]-5" id=168 TIMED_WAITING
>
> "HealthSystemHealthCheckWorker" id=28
> TIMED_WAITING
>
>
> "resin-port-127.0.0.1:6945-launcher" id=172 TIMED_WAITING
>
> at sun.misc.Unsafe.park (Unsafe.java)
> (native)
> at
> java.util.concurrent.locks.LockSupport.parkUntil (LockSupport.java:351)
> at
> com.caucho.env.thread2.AbstractTaskWorker2.run
> (AbstractTaskWorker2.java:216)
> at
> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
> at
> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>
> "main" id=1 TIMED_WAITING
> waiting on
> com.caucho.server.resin.ResinWaitForExitService@7d2f117
> at java.lang.Object.wait
> (Object.java) (native)
> at
> com.caucho.server.resin.ResinWaitForExitService.waitForExit
> (ResinWaitForExitService.java:135)
> at
> com.caucho.server.resin.Resin.waitForExit (Resin.java:1190)
> at com.caucho.server.resin.Resin.main
> (Resin.java:1239)
>
> "resin-timer" id=9 TIMED_WAITING
> at java.lang.Thread.sleep
> (Thread.java) (native)
> at
> com.caucho.util.CurrentTime$CurrentTimeThread.run (CurrentTime.java:228)
>
>
> "ThreadLauncher2[ThreadPool[system]]-1" id=10 TIMED_WAITING
>
> "ThreadScheduleWorker[]-2" id=29
> TIMED_WAITING
> at sun.misc.Unsafe.park (Unsafe.java)
> (native)
> at
> java.util.concurrent.locks.LockSupport.parkUntil (LockSupport.java:351)
> at
> com.caucho.env.thread2.AbstractTaskWorker2.run
> (AbstractTaskWorker2.java:216)
> at java.lang.Thread.run
> (Thread.java:662)
>
> "Connection pool reaper" id=49
> TIMED_WAITING
> at java.lang.Thread.sleep
> (Thread.java) (native)
> at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPoolManager$ConnectionCloserThread.run
> (ConnectionPoolManager.java:130)
>
> [2013-01-31 19:03:00.003] {resin-154} AnomalyAnalyzer JVM|Thread|JVM
> Runnable Count WARNING
> 3.000 sample is 147.49% of 2.034 avg,
> 5.327 std deviations (std=0.181, n=1029.0)
> [2013-01-31 19:09:00.002] {resin-168} AnomalyAnalyzer JVM|Thread|JVM
> Runnable Count WARNING
> 3.000 sample is 147.44% of 2.035 avg,
> 5.269 std deviations (std=0.183, n=1065.0)
> [2013-01-31 19:14:00.002] {resin-118} AnomalyAnalyzer JVM|Thread|JVM
> Runnable Count WARNING
> 3.000 sample is 147.43% of 2.035 avg,
> 5.257 std deviations (std=0.184, n=1089.0)
> [2013-02-01 11:05:00.003] {resin-449} AnomalyAnalyzer OS|Process|File
> Descriptor Count WARNING
> 229.000 sample is 101.32% of 226.010
> avg, 8.674 std deviations (std=0.345, n=8554.0)
> [2013-02-01 11:06:00.005] {resin-449} AnomalyAnalyzer JVM|Thread|JVM Waiting
> Count WARNING
> 37.000 sample is 114.35% of 32.357
> avg, 5.171 std deviations (std=0.898, n=8559.0)
> [2013-02-01 11:06:00.005] {resin-449} AnomalyAnalyzer Resin|Thread|Thread
> Active Count WARNING
> 18.000 sample is 81.99% of 21.953
> avg, 6.191 std deviations (std=0.639, n=8559.0)
> [2013-02-01 11:06:00.005] {resin-421} Health action DumpThreads[] executing
> [2013-02-01 11:06:00.008] {resin-425} Health action DumpThreads[] executing
Re: max_pred_locks_per_transaction
Posted by Erlend Garåsen <e....@usit.uio.no>.
Sorry, wrong thread dump. This is the correct one. The previous one was
generated after I stopped the services. I guess entries like this one is
related to the locks?
"resin-118" id=118 WAITING
----8<----
[2013-01-31 19:00:00.014] {resin-18} Thread Dump generated Thu Jan 31
18:59:59 CET 2013
"Signal Dispatcher" id=4 RUNNABLE
"resin-18" id=18 RUNNABLE
at
sun.management.ThreadImpl.getThreadInfo1 (ThreadImpl.java) (native)
at
sun.management.ThreadImpl.getThreadInfo (ThreadImpl.java:156)
at
com.caucho.util.ThreadDump.getThreadDump (ThreadDump.java:121)
at
com.caucho.admin.action.ThreadDumpAction.execute (ThreadDumpAction.java:52)
at
com.caucho.health.action.DumpThreads.doActionImpl (DumpThreads.java:99)
at
com.caucho.health.action.AbstractHealthAction.doAction
(AbstractHealthAction.java:90)
at
com.caucho.env.health.HealthService.executeActions (HealthService.java:722)
at
com.caucho.env.health.HealthService$ExecuteActionsTask.run
(HealthService.java:1030)
at
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
at
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
"resin-148" id=148 RUNNABLE
at
java.lang.Throwable.fillInStackTrace (Throwable.java) (native)
at java.lang.Throwable.<init>
(Throwable.java:196)
at java.lang.Exception.<init>
(Exception.java:41)
at
javax.management.JMException.<init> (JMException.java:35)
at
javax.management.OperationsException.<init> (OperationsException.java:36)
at
javax.management.InstanceNotFoundException.<init>
(InstanceNotFoundException.java:34)
at
com.caucho.jmx.AbstractMBeanServer.getAttribute
(AbstractMBeanServer.java:594)
at
com.caucho.env.meter.JmxExpr$JmxDelta.sample (JmxExpr.java:211)
at
com.caucho.env.meter.JmxExpr$JmxContainerExpr.sample (JmxExpr.java:94)
at
com.caucho.env.meter.JmxExpr$JmxContainerExpr.sample (JmxExpr.java:94)
at
com.caucho.env.meter.JmxCalculationMeterImpl.sample
(JmxCalculationMeterImpl.java:62)
at
com.caucho.server.admin.StatSystem$Sample.sample (StatSystem.java:244)
at
com.caucho.server.admin.ProStatSystem.sampleData (ProStatSystem.java:582)
at
com.caucho.server.admin.ProStatSystem.sample (ProStatSystem.java:558)
at
com.caucho.server.admin.ProStatSystem.handleAlarm (ProStatSystem.java:646)
at
com.caucho.util.Alarm.handleAlarm (Alarm.java:523)
at com.caucho.util.Alarm.run
(Alarm.java:495)
at
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
at
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
"resin-125" id=125 RUNNABLE
at
com.caucho.vfs.JniMemoryMappedFile.nativeRead (JniMemoryMappedFile.java)
(native)
at
com.caucho.vfs.JniMemoryMappedFile.read (JniMemoryMappedFile.java:136)
at
com.caucho.db.block.BlockReadWrite.readBlockImpl (BlockReadWrite.java:210)
at
com.caucho.db.block.BlockReadWrite.readBlock (BlockReadWrite.java:177)
at
com.caucho.db.block.Block.read (Block.java:259)
at
com.caucho.db.block.BlockStore.readBlock (BlockStore.java:569)
at
com.caucho.db.xa.DbTransaction.readBlock (DbTransaction.java:332)
at
com.caucho.db.table.TableIterator.nextBlock (TableIterator.java:276)
at
com.caucho.db.sql.RowIterateExpr.nextBlock (RowIterateExpr.java:87)
at
com.caucho.db.sql.Query.nextBlock (Query.java:788)
at com.caucho.db.sql.Query.start
(Query.java:726)
at
com.caucho.db.sql.DeleteQuery.execute (DeleteQuery.java:74)
at
com.caucho.db.jdbc.PreparedStatementImpl.execute
(PreparedStatementImpl.java:365)
at
com.caucho.db.jdbc.PreparedStatementImpl.executeUpdate
(PreparedStatementImpl.java:335)
at
com.caucho.server.admin.StatDatabase$ReaperTask.handleAlarm
(StatDatabase.java:941)
at
com.caucho.util.Alarm.handleAlarm (Alarm.java:523)
at com.caucho.util.Alarm.run
(Alarm.java:495)
at
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
at
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
"resin-port-127.0.0.1:6945-16"
id=16 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-17"
id=17 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-19"
id=19 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-31"
id=31 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-37"
id=37 RUNNABLE (in native)
at
com.caucho.vfs.JniSocketImpl.nativeAccept (JniSocketImpl.java) (native)
at
com.caucho.vfs.JniSocketImpl.accept (JniSocketImpl.java:141)
at
com.caucho.vfs.JniServerSocketImpl.accept (JniServerSocketImpl.java:254)
at
com.caucho.network.listen.TcpSocketLinkListener.accept
(TcpSocketLinkListener.java:1372)
at
com.caucho.network.listen.TcpSocketLink.accept (TcpSocketLink.java:978)
at
com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl
(TcpSocketLink.java:928)
at
com.caucho.network.listen.ConnectionTask.runThread (ConnectionTask.java:117)
at
com.caucho.network.listen.ConnectionTask.run (ConnectionTask.java:93)
at
com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks
(SocketLinkThreadLauncher.java:169)
at
com.caucho.network.listen.TcpSocketAcceptThread.run
(TcpSocketAcceptThread.java:61)
at
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
at
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
"resin-port-127.0.0.1:6945-13"
id=13 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-14"
id=14 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-15"
id=15 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-22"
id=22 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-24"
id=24 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-26"
id=26 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-30"
id=30 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-41"
id=41 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-88"
id=88 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-90"
id=90 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-98"
id=98 RUNNABLE (in native)
at
com.caucho.vfs.JniSocketImpl.nativeAccept (JniSocketImpl.java) (native)
at
com.caucho.vfs.JniSocketImpl.accept (JniSocketImpl.java:141)
at
com.caucho.vfs.JniServerSocketImpl.accept (JniServerSocketImpl.java:254)
at
com.caucho.network.listen.TcpSocketLinkListener.accept
(TcpSocketLinkListener.java:1372)
at
com.caucho.network.listen.TcpSocketLink.accept (TcpSocketLink.java:978)
at
com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl
(TcpSocketLink.java:928)
at
com.caucho.network.listen.TcpSocketLink.handleAcceptTask
(TcpSocketLink.java:902)
at
com.caucho.network.listen.ConnectionTask.runThread (ConnectionTask.java:114)
at
com.caucho.network.listen.ConnectionTask.run (ConnectionTask.java:93)
at
com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks
(SocketLinkThreadLauncher.java:169)
at
com.caucho.network.listen.TcpSocketResumeThread.run
(TcpSocketResumeThread.java:62)
at
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
at
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
"resin-main-link" id=33 RUNNABLE
(in native)
at
java.net.SocketInputStream.socketRead0 (SocketInputStream.java) (native)
at
java.net.SocketInputStream.read (SocketInputStream.java:129)
at
java.net.SocketInputStream.read (SocketInputStream.java:182)
at
com.caucho.remote.websocket.UnmaskedFrameInputStream.readFrameHeaderImpl
(UnmaskedFrameInputStream.java:142)
at
com.caucho.remote.websocket.FrameInputStream.readFrameHeader
(FrameInputStream.java:97)
at
com.caucho.remote.websocket.WebSocketInputStream.startBinaryMessage
(WebSocketInputStream.java:78)
at
com.caucho.hmtp.HmtpWebSocketReader.readPacket (HmtpWebSocketReader.java:70)
at
com.caucho.hmtp.HmtpLinkWorker.run (HmtpLinkWorker.java:114)
at
com.caucho.server.resin.ResinLink.run (ResinLink.java:71)
at
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
at
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
"resin-select-manager-0" id=36
RUNNABLE (in native)
at
com.caucho.network.listen.JniSelectManager.selectNative
(JniSelectManager.java) (native)
at
com.caucho.network.listen.JniSelectManager.runImpl
(JniSelectManager.java:392)
at
com.caucho.network.listen.JniSelectManager.run (JniSelectManager.java:359)
at java.lang.Thread.run
(Thread.java:662)
"resin-fail-safe-halt" id=34 WAITING
at sun.misc.Unsafe.park
(Unsafe.java) (native)
at
java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
at
com.caucho.env.shutdown.ShutdownSystem$FailSafeHaltThread.run
(ShutdownSystem.java:499)
"resin-shutdown" id=35 WAITING
[2013-01-31 19:00:00.014] {resin-18} at sun.misc.Unsafe.park
(Unsafe.java) (native)
at
java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
at
com.caucho.env.shutdown.ShutdownSystem$ShutdownThread.run
(ShutdownSystem.java:446)
"resin-118" id=118 WAITING
"resin-134" id=134 WAITING
"resin-141" id=141 WAITING
"resin-144" id=144 WAITING
"resin-154" id=154 WAITING
"resin-156" id=156 WAITING
"resin-163" id=163 WAITING
"resin-167" id=167 WAITING
"resin-169" id=169 WAITING
"resin-170" id=170 WAITING
"resin-171" id=171 WAITING
"resin-174" id=174 WAITING
"resin-175" id=175 WAITING
"resin-176" id=176 WAITING
"resin-177" id=177 WAITING
"resin-178" id=178 WAITING
"resin-179" id=179 WAITING
"resin-47" id=47 WAITING
at sun.misc.Unsafe.park
(Unsafe.java) (native)
at
java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
at
com.caucho.env.thread2.ResinThread2.park (ResinThread2.java:196)
at
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:147)
at
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
"Finalizer" id=3 WAITING
waiting on
java.lang.ref.ReferenceQueue$Lock@4ecd51ad
at java.lang.Object.wait
(Object.java) (native)
at
java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:118)
at
java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:134)
at
java.lang.ref.Finalizer$FinalizerThread.run (Finalizer.java:159)
"Reference Handler" id=2 WAITING
waiting on
java.lang.ref.Reference$Lock@9ad5d46
at java.lang.Object.wait
(Object.java) (native)
at java.lang.Object.wait
(Object.java:485)
at
java.lang.ref.Reference$ReferenceHandler.run (Reference.java:116)
"BlockWriter[Table[log_data:7]]-13" id=38 TIMED_WAITING
"BlockWriter[Table[stat_data_mcf_1:5]]-11" id=136 TIMED_WAITING
"CoordinatorThread[]-3" id=12
TIMED_WAITING
"FlushWorker[]-5" id=168 TIMED_WAITING
"HealthSystemHealthCheckWorker"
id=28 TIMED_WAITING
"resin-port-127.0.0.1:6945-launcher" id=172 TIMED_WAITING
at sun.misc.Unsafe.park
(Unsafe.java) (native)
at
java.util.concurrent.locks.LockSupport.parkUntil (LockSupport.java:351)
at
com.caucho.env.thread2.AbstractTaskWorker2.run
(AbstractTaskWorker2.java:216)
at
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
at
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
"main" id=1 TIMED_WAITING
waiting on
com.caucho.server.resin.ResinWaitForExitService@7d2f117
at java.lang.Object.wait
(Object.java) (native)
at
com.caucho.server.resin.ResinWaitForExitService.waitForExit
(ResinWaitForExitService.java:135)
at
com.caucho.server.resin.Resin.waitForExit (Resin.java:1190)
at
com.caucho.server.resin.Resin.main (Resin.java:1239)
"resin-timer" id=9 TIMED_WAITING
at java.lang.Thread.sleep
(Thread.java) (native)
at
com.caucho.util.CurrentTime$CurrentTimeThread.run (CurrentTime.java:228)
"ThreadLauncher2[ThreadPool[system]]-1" id=10 TIMED_WAITING
"ThreadScheduleWorker[]-2" id=29
TIMED_WAITING
at sun.misc.Unsafe.park
(Unsafe.java) (native)
at
java.util.concurrent.locks.LockSupport.parkUntil (LockSupport.java:351)
at
com.caucho.env.thread2.AbstractTaskWorker2.run
(AbstractTaskWorker2.java:216)
at java.lang.Thread.run
(Thread.java:662)
"Connection pool reaper" id=49
TIMED_WAITING
at java.lang.Thread.sleep
(Thread.java) (native)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPoolManager$ConnectionCloserThread.run
(ConnectionPoolManager.java:130)
[2013-01-31 19:03:00.003] {resin-154} AnomalyAnalyzer JVM|Thread|JVM
Runnable Count WARNING
3.000 sample is 147.49% of 2.034
avg, 5.327 std deviations (std=0.181, n=1029.0)
[2013-01-31 19:09:00.002] {resin-168} AnomalyAnalyzer JVM|Thread|JVM
Runnable Count WARNING
3.000 sample is 147.44% of 2.035
avg, 5.269 std deviations (std=0.183, n=1065.0)
[2013-01-31 19:14:00.002] {resin-118} AnomalyAnalyzer JVM|Thread|JVM
Runnable Count WARNING
3.000 sample is 147.43% of 2.035
avg, 5.257 std deviations (std=0.184, n=1089.0)
[2013-02-01 11:05:00.003] {resin-449} AnomalyAnalyzer OS|Process|File
Descriptor Count WARNING
229.000 sample is 101.32% of
226.010 avg, 8.674 std deviations (std=0.345, n=8554.0)
[2013-02-01 11:06:00.005] {resin-449} AnomalyAnalyzer JVM|Thread|JVM
Waiting Count WARNING
37.000 sample is 114.35% of
32.357 avg, 5.171 std deviations (std=0.898, n=8559.0)
[2013-02-01 11:06:00.005] {resin-449} AnomalyAnalyzer
Resin|Thread|Thread Active Count WARNING
18.000 sample is 81.99% of
21.953 avg, 6.191 std deviations (std=0.639, n=8559.0)
[2013-02-01 11:06:00.005] {resin-421} Health action DumpThreads[] executing
[2013-02-01 11:06:00.008] {resin-425} Health action DumpThreads[] executing
Re: max_pred_locks_per_transaction
Posted by Erlend Garåsen <e....@usit.uio.no>.
On 25.01.13 12.24, Karl Wright wrote:
> Leaving logging at the default values would have shown the ERROR
> message you have below. So the cause for the pause must have been
> something else.
Agree.
> When ManifoldCF seems to make no progress, the first thing to do is
> look at the simple history and see if it is retrying on something for
> some reason. If that is not helpful, get a thread dump. You can use
> jstack for that purpose.
Resin has some health check tools I can try to configure to give me such
information. I have included a thread dump in the end of this email.
The problem also occurs by running RC8/1.1. It even occurs on our
production system running MCF 1.0.1. I created a test job which crawls
the same documents, but instead of posting to Solr, it just dumps to
NullOutput, which means it is not related to the Solr output connector.
We have never discovered such problems before, and the only difference
is that I have added a couple of new hosts. These hosts do have a lot of
excludes. We have began to use the "exclude from index" functionality
which do have some complex regexps, so maybe this might be the source of
the problem (a possible bug in MCF):
---8<--
# Exclude some journal resources from the index:
https://www\.journals\.uio\.no/index\.php/.*/(search|issue|information|announcement|current|user|login|about|notification).*
https://www\.journals\.uio\.no/index\.php/index/(search|user|login).*
https://www\.journals\.uio\.no/index\.php/.*/user$
# Exclude some DOU resources from the index:
# - where the url contains characters after the digits (.+):
https://www\.duo\.uio\.no/handle/\d{9}/\d+/.+
# - if they are navigation pages:
https://www\.duo\.uio\.no/(community-list|browse|password-login|inn|discover).*
---8<--
This is what the Postgres admins told me today. The idle processes (idle
in transaction) are:
- - SELECT id,status,checktime FROM jobqueue WHERE dochash=$1 AND
jobid=$2 FOR UPDATE
- - UPDATE hopcount SET deathmark=$1 WHERE jobid=$2 AND deathmark=$3
I have started the crawl again with the following params set to DEBUG.
In addition I have notified the server admins and asked them to adjust
Resin's health check to give us even more detailed output. I will get
back to this issue as soon as I have found out more.
<property name="org.apache.manifoldcf.lock" value="DEBUG"/>
<property name="org.apache.manifoldcf.agents" value="DEBUG"/>
<property name="org.apache.manifoldcf.crawlerthreads" value="DEBUG"/>
<property name="org.apache.manifoldcf.jobs" value="DEBUG"/>
<property name="org.apache.manifoldcf.connectors" value="DEBUG"/>
Here's what Resin have detected regarding these locks:
[2013-02-01 11:06:00.019] {resin-421} Thread Dump generated Fri Feb 01
11:05:59 CET 2013
"Signal Dispatcher" id=4 RUNNABLE
"resin-421" id=421 RUNNABLE
at
sun.management.ThreadImpl.getThreadInfo1 (ThreadImpl.java) (native)
at
sun.management.ThreadImpl.getThreadInfo (ThreadImpl.java:156)
at
com.caucho.util.ThreadDump.getThreadDump (ThreadDump.java:121)
at
com.caucho.admin.action.ThreadDumpAction.execute (ThreadDumpAction.java:52)
at
com.caucho.health.action.DumpThreads.doActionImpl (DumpThreads.java:99)
at
com.caucho.health.action.AbstractHealthAction.doAction
(AbstractHealthAction.java:90)
at
com.caucho.env.health.HealthService.executeActions (HealthService.java:722)
at
com.caucho.env.health.HealthService$ExecuteActionsTask.run
(HealthService.java:1030)
at
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
at
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
"resin-449" id=449 RUNNABLE
at
com.caucho.env.health.HealthService.getCurrent (HealthService.java:100)
at
com.caucho.env.health.ProHealthSystemFacade.fireEventImpl
(ProHealthSystemFacade.java:61)
at
com.caucho.env.health.HealthSystemFacade.fireEvent
(HealthSystemFacade.java:65)
at
com.caucho.health.analysis.AnomalyAnalyzer.analyze
(AnomalyAnalyzer.java:178)
at
com.caucho.server.admin.ProStatSystem.analyze (ProStatSystem.java:628)
at
com.caucho.server.admin.ProStatSystem.sample (ProStatSystem.java:560)
at
com.caucho.server.admin.ProStatSystem.handleAlarm (ProStatSystem.java:646)
at
com.caucho.util.Alarm.handleAlarm (Alarm.java:523)
at com.caucho.util.Alarm.run
(Alarm.java:495)
at
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
at
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
"resin-port-127.0.0.1:6945-31"
id=31 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-37"
id=37 RUNNABLE (in native)
at
com.caucho.vfs.JniSocketImpl.readNative (JniSocketImpl.java) (native)
at
com.caucho.vfs.JniSocketImpl.read (JniSocketImpl.java:438)
at
com.caucho.vfs.JniStream.readTimeout (JniStream.java:102)
at
com.caucho.vfs.ReadStream.fillWithTimeout (ReadStream.java:1145)
at
com.caucho.network.listen.TcpSocketLinkListener.keepaliveThreadRead
(TcpSocketLinkListener.java:1546)
at
com.caucho.network.listen.TcpSocketLink.processKeepalive
(TcpSocketLink.java:1348)
at
com.caucho.network.listen.TcpSocketLink.handleRequestsImpl
(TcpSocketLink.java:1233)
at
com.caucho.network.listen.TcpSocketLink.handleRequests
(TcpSocketLink.java:1149)
at
com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl
(TcpSocketLink.java:950)
at
com.caucho.network.listen.ConnectionTask.runThread (ConnectionTask.java:117)
at
com.caucho.network.listen.ConnectionTask.run (ConnectionTask.java:93)
at
com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks
(SocketLinkThreadLauncher.java:169)
at
com.caucho.network.listen.TcpSocketAcceptThread.run
(TcpSocketAcceptThread.java:61)
at
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
at
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
"resin-port-127.0.0.1:6945-14"
id=14 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-15"
id=15 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-22"
id=22 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-24"
id=24 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-26"
id=26 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-30"
id=30 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-41"
id=41 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-88"
id=88 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-90"
id=90 RUNNABLE (in native)
"resin-port-127.0.0.1:6945-98"
id=98 RUNNABLE (in native)
at
com.caucho.vfs.JniSocketImpl.nativeAccept (JniSocketImpl.java) (native)
at
com.caucho.vfs.JniSocketImpl.accept (JniSocketImpl.java:141)
at
com.caucho.vfs.JniServerSocketImpl.accept (JniServerSocketImpl.java:254)
at
com.caucho.network.listen.TcpSocketLinkListener.accept
(TcpSocketLinkListener.java:1372)
at
com.caucho.network.listen.TcpSocketLink.accept (TcpSocketLink.java:978)
at
com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl
(TcpSocketLink.java:928)
at
com.caucho.network.listen.TcpSocketLink.handleAcceptTask
(TcpSocketLink.java:902)
at
com.caucho.network.listen.ConnectionTask.runThread (ConnectionTask.java:114)
at
com.caucho.network.listen.ConnectionTask.run (ConnectionTask.java:93)
at
com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks
(SocketLinkThreadLauncher.java:169)
at
com.caucho.network.listen.TcpSocketResumeThread.run
(TcpSocketResumeThread.java:62)
at
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
at
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
"resin-port-127.0.0.1:6945-13"
id=13 RUNNABLE (in native)
at
com.caucho.vfs.JniSocketImpl.readNative (JniSocketImpl.java) (native)
at
com.caucho.vfs.JniSocketImpl.read (JniSocketImpl.java:438)
at
com.caucho.vfs.JniStream.readTimeout (JniStream.java:102)
at
com.caucho.vfs.ReadStream.fillWithTimeout (ReadStream.java:1145)
at
com.caucho.network.listen.TcpSocketLinkListener.keepaliveThreadRead
(TcpSocketLinkListener.java:1546)
at
com.caucho.network.listen.TcpSocketLink.processKeepalive
(TcpSocketLink.java:1348)
at
com.caucho.network.listen.TcpSocketLink.handleRequestsImpl
(TcpSocketLink.java:1233)
at
com.caucho.network.listen.TcpSocketLink.handleRequests
(TcpSocketLink.java:1149)
at
com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl
(TcpSocketLink.java:950)
at
com.caucho.network.listen.TcpSocketLink.handleAcceptTask
(TcpSocketLink.java:902)
at
com.caucho.network.listen.ConnectionTask.runThread (ConnectionTask.java:114)
at
com.caucho.network.listen.ConnectionTask.run (ConnectionTask.java:93)
at
com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks
(SocketLinkThreadLauncher.java:169)
at
com.caucho.network.listen.TcpSocketResumeThread.run
(TcpSocketResumeThread.java:62)
at
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
at
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
"resin-main-link" id=33 RUNNABLE
(in native)
at
java.net.SocketInputStream.socketRead0 (SocketInputStream.java) (native)
at
java.net.SocketInputStream.read (SocketInputStream.java:129)
at
java.net.SocketInputStream.read (SocketInputStream.java:182)
at
com.caucho.remote.websocket.UnmaskedFrameInputStream.readFrameHeaderImpl
(UnmaskedFrameInputStream.java:142)
at
com.caucho.remote.websocket.FrameInputStream.readFrameHeader
(FrameInputStream.java:97)
at
com.caucho.remote.websocket.WebSocketInputStream.startBinaryMessage
(WebSocketInputStream.java:78)
at
com.caucho.hmtp.HmtpWebSocketReader.readPacket (HmtpWebSocketReader.java:70)
at
com.caucho.hmtp.HmtpLinkWorker.run (HmtpLinkWorker.java:114)
at
com.caucho.server.resin.ResinLink.run (ResinLink.java:71)
at
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
at
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
"resin-select-manager-0" id=36
RUNNABLE (in native)
at
com.caucho.network.listen.JniSelectManager.selectNative
(JniSelectManager.java) (native)
at
com.caucho.network.listen.JniSelectManager.runImpl
(JniSelectManager.java:392)
at
com.caucho.network.listen.JniSelectManager.run (JniSelectManager.java:359)
at java.lang.Thread.run
(Thread.java:662)
"resin-fail-safe-halt" id=34 WAITING
at sun.misc.Unsafe.park
(Unsafe.java) (native)
at
java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
at
com.caucho.env.shutdown.ShutdownSystem$FailSafeHaltThread.run
(ShutdownSystem.java:499)
"resin-shutdown" id=35 WAITING
at sun.misc.Unsafe.park
(Unsafe.java) (native)
at
java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
at
com.caucho.env.shutdown.ShutdownSystem$ShutdownThread.run
(ShutdownSystem.java:446)
"resin-16" id=16 WAITING
"resin-17" id=17 WAITING
"resin-19" id=19 WAITING
"resin-397" id=397 WAITING
"resin-403" id=403 WAITING
"resin-412" id=412 WAITING
"resin-425" id=425 WAITING
"resin-426" id=426 WAITING
"resin-428" id=428 WAITING
"resin-432" id=432 WAITING
"resin-433" id=433 WAITING
"resin-434" id=434 WAITING
[2013-02-01 11:06:00.019] {resin-421}
"resin-435" id=435 WAITING
"resin-436" id=436 WAITING
"resin-438" id=438 WAITING
"resin-439" id=439 WAITING
"resin-440" id=440 WAITING
"resin-441" id=441 WAITING
"resin-442" id=442 WAITING
"resin-444" id=444 WAITING
"resin-451" id=451 WAITING
at sun.misc.Unsafe.park
(Unsafe.java) (native)
at
java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
at
com.caucho.env.thread2.ResinThread2.park (ResinThread2.java:196)
at
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:147)
at
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
"Finalizer" id=3 WAITING
waiting on
java.lang.ref.ReferenceQueue$Lock@4ecd51ad
at java.lang.Object.wait
(Object.java) (native)
at
java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:118)
at
java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:134)
at
java.lang.ref.Finalizer$FinalizerThread.run (Finalizer.java:159)
"Reference Handler" id=2 WAITING
waiting on
java.lang.ref.Reference$Lock@9ad5d46
at java.lang.Object.wait
(Object.java) (native)
at java.lang.Object.wait
(Object.java:485)
at
java.lang.ref.Reference$ReferenceHandler.run (Reference.java:116)
"BlockWriter[Table[log_data:7]]-13" id=450 TIMED_WAITING
"BlockWriter[Table[stat_data_mcf_1:5]]-11" id=437 TIMED_WAITING
"CoordinatorThread[]-3" id=12
TIMED_WAITING
"FlushWorker[]-5" id=448
TIMED_WAITING
"HealthSystemHealthCheckWorker"
id=28 TIMED_WAITING
"resin-port-127.0.0.1:6945-launcher" id=447 TIMED_WAITING
at sun.misc.Unsafe.park
(Unsafe.java) (native)
at
java.util.concurrent.locks.LockSupport.parkUntil (LockSupport.java:351)
at
com.caucho.env.thread2.AbstractTaskWorker2.run
(AbstractTaskWorker2.java:216)
at
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
at
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
"main" id=1 TIMED_WAITING
waiting on
com.caucho.server.resin.ResinWaitForExitService@7d2f117
at java.lang.Object.wait
(Object.java) (native)
at
com.caucho.server.resin.ResinWaitForExitService.waitForExit
(ResinWaitForExitService.java:135)
at
com.caucho.server.resin.Resin.waitForExit (Resin.java:1190)
at
com.caucho.server.resin.Resin.main (Resin.java:1239)
"resin-timer" id=9 TIMED_WAITING
at java.lang.Thread.sleep
(Thread.java) (native)
at
com.caucho.util.CurrentTime$CurrentTimeThread.run (CurrentTime.java:228)
"ThreadLauncher2[ThreadPool[system]]-1" id=10 TIMED_WAITING
"ThreadScheduleWorker[]-2" id=29
TIMED_WAITING
at sun.misc.Unsafe.park
(Unsafe.java) (native)
at
java.util.concurrent.locks.LockSupport.parkUntil (LockSupport.java:351)
at
com.caucho.env.thread2.AbstractTaskWorker2.run
(AbstractTaskWorker2.java:216)
at java.lang.Thread.run
(Thread.java:662)
"Connection pool reaper" id=49
TIMED_WAITING
at java.lang.Thread.sleep
(Thread.java) (native)
at
org.apache.manifoldcf.core.jdbcpool.ConnectionPoolManager$ConnectionCloserThread.run
(ConnectionPoolManager.java:130)
Erlend
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: max_pred_locks_per_transaction
Posted by Karl Wright <da...@gmail.com>.
Hi Erlend,
Leaving logging at the default values would have shown the ERROR
message you have below. So the cause for the pause must have been
something else.
When ManifoldCF seems to make no progress, the first thing to do is
look at the simple history and see if it is retrying on something for
some reason. If that is not helpful, get a thread dump. You can use
jstack for that purpose.
As for the PostgreSQL parameters, max_pred_locks_per_transaction seems
to be PostgreSQL 9 black magic. Here's the documentation:
http://www.postgresql.org/docs/9.1/static/runtime-config-locks.html
Default is 64, but they don't say how it is allocated. I'd guess
therefore that you should try 50% higher and see if that works, e.g.
96. I guess the limit is the amount of shared memory your OS allows
you to allocate.
Karl
On Fri, Jan 25, 2013 at 5:26 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> After we started to crawl journals, the crawler just stopped after a couple
> of hours. Running MCF in debug mode gave me the stack trace shown below. I
> thing we need to adjust some PG parameters, perhaps
> max_pred_locks_per_transaction. The database admins are now asking me about
> which value to set. They have increased it, but I don't know whether it is
> sufficient.
>
> Another thing. I did not see this message until I changed the log level to
> debug, but the log4j should catch this error messages with warn level
> enabled. So maybe it is a dead end, i.e. a totally different cause and that
> this just occurred as a coincidence.
>
> ERROR 2013-01-19 03:47:15,049 (Worker thread '49') - Worker thread
> aborting and restarting due to database connection reset: Database
> exception: Exception doing query: ERROR: out of shared memory
> Hint: You might need to increase max_pred_locks_per_transaction.
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
> exception: Exception doing query: ERROR: out of shared memory
> Hint: You might need to increase max_pred_locks_per_transaction.
> at
> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:681)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
> at
> org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4089)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:1932)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.flush(WorkerThread.java:1863)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:554)
> Caused by: org.postgresql.util.PSQLException: ERROR: out of shared memory
> Hint: You might need to increase max_pred_locks_per_transaction.
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
> at org.apache.manifoldcf.core.database.Database.execute(Database.java:826)
> at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641)
> DEBUG 2013-01-19 03:47:23,386 (Idle cleanup thread) - Checking for
> connections, idleTimeout: 1358563583386
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050