You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by "Karl Wright (JIRA)" <ji...@apache.org> on 2011/02/27 17:36:39 UTC

[jira] Created: (CONNECTORS-166) Crawl seizes up when running Derby

Crawl seizes up when running Derby
----------------------------------

                 Key: CONNECTORS-166
                 URL: https://issues.apache.org/jira/browse/CONNECTORS-166
             Project: ManifoldCF
          Issue Type: Bug
          Components: Framework crawler agent
    Affects Versions: ManifoldCF 0.1, ManifoldCF next
            Reporter: Karl Wright
            Assignee: Karl Wright


A crawl using multiple worker threads with Derby eventually hangs, because threads get deadlocked dealing with carrydown information.  At the time of hang, a thread dump yields:

"Worker thread '5'" daemon prio=6 tid=0x02fc7800 nid=0xd78 in Object.wait() [0x0465f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x2858b720> (a org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
        at java.lang.Thread.join(Unknown Source)
        - locked <0x2858b720> (a org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
        at java.lang.Thread.join(Unknown Source)
        at org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:453)
        at org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:489)
        at org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1131)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
        at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168)
        at org.apache.manifoldcf.core.database.DBInterfaceDerby.performQuery(DBInterfaceDerby.java:785)
        at org.apache.manifoldcf.crawler.jobs.JobManager.processDeleteHashSet(JobManager.java:2592)
        at org.apache.manifoldcf.crawler.jobs.JobManager.calculateAffectedDeleteCarrydownChildren(JobManager.java:2565)
        at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentDeletedMultiple(JobManager.java:2494)
        at org.apache.manifoldcf.crawler.system.WorkerThread.processDeleteLists(WorkerThread.java:1077)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:544)

... for at least two threads.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Resolved: (CONNECTORS-166) Crawl seizes up when running Derby

Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CONNECTORS-166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Karl Wright resolved CONNECTORS-166.
------------------------------------

       Resolution: Fixed
    Fix Version/s: ManifoldCF 0.2

r1082140.


> Crawl seizes up when running Derby
> ----------------------------------
>
>                 Key: CONNECTORS-166
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-166
>             Project: ManifoldCF
>          Issue Type: Bug
>          Components: Framework crawler agent
>    Affects Versions: ManifoldCF 0.1, ManifoldCF 0.2
>            Reporter: Karl Wright
>            Assignee: Karl Wright
>             Fix For: ManifoldCF 0.2
>
>
> A crawl using multiple worker threads with Derby eventually hangs, because threads get deadlocked dealing with carrydown information.  At the time of hang, a thread dump yields:
> "Worker thread '5'" daemon prio=6 tid=0x02fc7800 nid=0xd78 in Object.wait() [0x0465f000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x2858b720> (a org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
>         at java.lang.Thread.join(Unknown Source)
>         - locked <0x2858b720> (a org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
>         at java.lang.Thread.join(Unknown Source)
>         at org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:453)
>         at org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:489)
>         at org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1131)
>         at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168)
>         at org.apache.manifoldcf.core.database.DBInterfaceDerby.performQuery(DBInterfaceDerby.java:785)
>         at org.apache.manifoldcf.crawler.jobs.JobManager.processDeleteHashSet(JobManager.java:2592)
>         at org.apache.manifoldcf.crawler.jobs.JobManager.calculateAffectedDeleteCarrydownChildren(JobManager.java:2565)
>         at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentDeletedMultiple(JobManager.java:2494)
>         at org.apache.manifoldcf.crawler.system.WorkerThread.processDeleteLists(WorkerThread.java:1077)
>         at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:544)
> ... for at least two threads.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CONNECTORS-166) Crawl seizes up when running Derby

Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CONNECTORS-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006583#comment-13006583 ] 

Karl Wright commented on CONNECTORS-166:
----------------------------------------

According to the Derby team, Derby trunk fixes this problem.  I've therefore build trunk and checked it in.
r1081520.


> Crawl seizes up when running Derby
> ----------------------------------
>
>                 Key: CONNECTORS-166
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-166
>             Project: ManifoldCF
>          Issue Type: Bug
>          Components: Framework crawler agent
>    Affects Versions: ManifoldCF 0.1, ManifoldCF 0.2
>            Reporter: Karl Wright
>            Assignee: Karl Wright
>
> A crawl using multiple worker threads with Derby eventually hangs, because threads get deadlocked dealing with carrydown information.  At the time of hang, a thread dump yields:
> "Worker thread '5'" daemon prio=6 tid=0x02fc7800 nid=0xd78 in Object.wait() [0x0465f000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x2858b720> (a org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
>         at java.lang.Thread.join(Unknown Source)
>         - locked <0x2858b720> (a org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
>         at java.lang.Thread.join(Unknown Source)
>         at org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:453)
>         at org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:489)
>         at org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1131)
>         at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168)
>         at org.apache.manifoldcf.core.database.DBInterfaceDerby.performQuery(DBInterfaceDerby.java:785)
>         at org.apache.manifoldcf.crawler.jobs.JobManager.processDeleteHashSet(JobManager.java:2592)
>         at org.apache.manifoldcf.crawler.jobs.JobManager.calculateAffectedDeleteCarrydownChildren(JobManager.java:2565)
>         at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentDeletedMultiple(JobManager.java:2494)
>         at org.apache.manifoldcf.crawler.system.WorkerThread.processDeleteLists(WorkerThread.java:1077)
>         at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:544)
> ... for at least two threads.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CONNECTORS-166) Crawl seizes up when running Derby

Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CONNECTORS-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007401#comment-13007401 ] 

Karl Wright commented on CONNECTORS-166:
----------------------------------------

Oleg reports that the test seems to pass.  The only remaining issue is that the version of Derby built from trunk has upgrade blocked.  I will therefore need to build a version of Derby based on the latest release plus the patch instead.


> Crawl seizes up when running Derby
> ----------------------------------
>
>                 Key: CONNECTORS-166
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-166
>             Project: ManifoldCF
>          Issue Type: Bug
>          Components: Framework crawler agent
>    Affects Versions: ManifoldCF 0.1, ManifoldCF 0.2
>            Reporter: Karl Wright
>            Assignee: Karl Wright
>
> A crawl using multiple worker threads with Derby eventually hangs, because threads get deadlocked dealing with carrydown information.  At the time of hang, a thread dump yields:
> "Worker thread '5'" daemon prio=6 tid=0x02fc7800 nid=0xd78 in Object.wait() [0x0465f000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x2858b720> (a org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
>         at java.lang.Thread.join(Unknown Source)
>         - locked <0x2858b720> (a org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
>         at java.lang.Thread.join(Unknown Source)
>         at org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:453)
>         at org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:489)
>         at org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1131)
>         at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168)
>         at org.apache.manifoldcf.core.database.DBInterfaceDerby.performQuery(DBInterfaceDerby.java:785)
>         at org.apache.manifoldcf.crawler.jobs.JobManager.processDeleteHashSet(JobManager.java:2592)
>         at org.apache.manifoldcf.crawler.jobs.JobManager.calculateAffectedDeleteCarrydownChildren(JobManager.java:2565)
>         at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentDeletedMultiple(JobManager.java:2494)
>         at org.apache.manifoldcf.crawler.system.WorkerThread.processDeleteLists(WorkerThread.java:1077)
>         at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:544)
> ... for at least two threads.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira