You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by "Karl Wright (JIRA)" <ji...@apache.org> on 2012/08/13 03:18:37 UTC
[jira] [Created] (CONNECTORS-502) Crawling with MySQL as a backend
shows signs of transactional integrity problems
Karl Wright created CONNECTORS-502:
--------------------------------------
Summary: Crawling with MySQL as a backend shows signs of transactional integrity problems
Key: CONNECTORS-502
URL: https://issues.apache.org/jira/browse/CONNECTORS-502
Project: ManifoldCF
Issue Type: Bug
Components: Framework crawler agent
Affects Versions: ManifoldCF 0.6
Environment: MySQL 5.1
Reporter: Karl Wright
When running the webcrawler-loadtests-mysql target, I got the following error:
{code}
ERROR 2012-08-12 20:59:35,664 (Worker thread '10') - Exception tossed: Unexpected jobqueue status - record id 1344819115577, expecting active status, saw 0
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected jobqueue status - record id 1344819115577, expecting active status, saw 0
at org.apache.manifoldcf.crawler.jobs.JobQueue.updateOrHopcountRemoveRecord(JobQueue.java:800)
at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentHopcountRemovalMultiple(JobManager.java:2583)
at org.apache.manifoldcf.crawler.system.WorkerThread.processJobQueueHopcountRemovals(WorkerThread.java:1100)
at org.apache.manifoldcf.crawler.system.WorkerThread.processHopcountRemovalLists(WorkerThread.java:952)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:790)
{code}
This indicates that although the stuffer thread thought it had put the document into the ACTIVE state, it had either never made it to the ACTIVE state or had been overridden by some other code. Either way, MySQL transactions do not seem to be working properly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-502) Crawling with MySQL as a
backend shows signs of transactional integrity problems
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433770#comment-13433770 ]
Karl Wright commented on CONNECTORS-502:
----------------------------------------
Fixed the problem with revised code for transaction types in the MySQL database class.
> Crawling with MySQL as a backend shows signs of transactional integrity problems
> --------------------------------------------------------------------------------
>
> Key: CONNECTORS-502
> URL: https://issues.apache.org/jira/browse/CONNECTORS-502
> Project: ManifoldCF
> Issue Type: Bug
> Components: Framework crawler agent
> Affects Versions: ManifoldCF 0.6
> Environment: MySQL 5.1
> Reporter: Karl Wright
> Fix For: ManifoldCF 0.7
>
>
> When running the webcrawler-loadtests-mysql target, I got the following error:
> {code}
> ERROR 2012-08-12 20:59:35,664 (Worker thread '10') - Exception tossed: Unexpected jobqueue status - record id 1344819115577, expecting active status, saw 0
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected jobqueue status - record id 1344819115577, expecting active status, saw 0
> at org.apache.manifoldcf.crawler.jobs.JobQueue.updateOrHopcountRemoveRecord(JobQueue.java:800)
> at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentHopcountRemovalMultiple(JobManager.java:2583)
> at org.apache.manifoldcf.crawler.system.WorkerThread.processJobQueueHopcountRemovals(WorkerThread.java:1100)
> at org.apache.manifoldcf.crawler.system.WorkerThread.processHopcountRemovalLists(WorkerThread.java:952)
> at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:790)
> {code}
> This indicates that although the stuffer thread thought it had put the document into the ACTIVE state, it had either never made it to the ACTIVE state or had been overridden by some other code. Either way, MySQL transactions do not seem to be working properly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CONNECTORS-502) Crawling with MySQL as a backend
shows signs of transactional integrity problems
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karl Wright updated CONNECTORS-502:
-----------------------------------
Fix Version/s: ManifoldCF 0.7
> Crawling with MySQL as a backend shows signs of transactional integrity problems
> --------------------------------------------------------------------------------
>
> Key: CONNECTORS-502
> URL: https://issues.apache.org/jira/browse/CONNECTORS-502
> Project: ManifoldCF
> Issue Type: Bug
> Components: Framework crawler agent
> Affects Versions: ManifoldCF 0.6
> Environment: MySQL 5.1
> Reporter: Karl Wright
> Fix For: ManifoldCF 0.7
>
>
> When running the webcrawler-loadtests-mysql target, I got the following error:
> {code}
> ERROR 2012-08-12 20:59:35,664 (Worker thread '10') - Exception tossed: Unexpected jobqueue status - record id 1344819115577, expecting active status, saw 0
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected jobqueue status - record id 1344819115577, expecting active status, saw 0
> at org.apache.manifoldcf.crawler.jobs.JobQueue.updateOrHopcountRemoveRecord(JobQueue.java:800)
> at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentHopcountRemovalMultiple(JobManager.java:2583)
> at org.apache.manifoldcf.crawler.system.WorkerThread.processJobQueueHopcountRemovals(WorkerThread.java:1100)
> at org.apache.manifoldcf.crawler.system.WorkerThread.processHopcountRemovalLists(WorkerThread.java:952)
> at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:790)
> {code}
> This indicates that although the stuffer thread thought it had put the document into the ACTIVE state, it had either never made it to the ACTIVE state or had been overridden by some other code. Either way, MySQL transactions do not seem to be working properly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-502) Crawling with MySQL as a
backend shows signs of transactional integrity problems
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13432885#comment-13432885 ]
Karl Wright commented on CONNECTORS-502:
----------------------------------------
On Ubuntu running MySQL 5.5, I don't get the same error, but it does not find all the documents (only 760 or so), and it hangs at a certain point and does not recover.
> Crawling with MySQL as a backend shows signs of transactional integrity problems
> --------------------------------------------------------------------------------
>
> Key: CONNECTORS-502
> URL: https://issues.apache.org/jira/browse/CONNECTORS-502
> Project: ManifoldCF
> Issue Type: Bug
> Components: Framework crawler agent
> Affects Versions: ManifoldCF 0.6
> Environment: MySQL 5.1
> Reporter: Karl Wright
>
> When running the webcrawler-loadtests-mysql target, I got the following error:
> {code}
> ERROR 2012-08-12 20:59:35,664 (Worker thread '10') - Exception tossed: Unexpected jobqueue status - record id 1344819115577, expecting active status, saw 0
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected jobqueue status - record id 1344819115577, expecting active status, saw 0
> at org.apache.manifoldcf.crawler.jobs.JobQueue.updateOrHopcountRemoveRecord(JobQueue.java:800)
> at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentHopcountRemovalMultiple(JobManager.java:2583)
> at org.apache.manifoldcf.crawler.system.WorkerThread.processJobQueueHopcountRemovals(WorkerThread.java:1100)
> at org.apache.manifoldcf.crawler.system.WorkerThread.processHopcountRemovalLists(WorkerThread.java:952)
> at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:790)
> {code}
> This indicates that although the stuffer thread thought it had put the document into the ACTIVE state, it had either never made it to the ACTIVE state or had been overridden by some other code. Either way, MySQL transactions do not seem to be working properly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CONNECTORS-502) Crawling with MySQL as a backend
shows signs of transactional integrity problems
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karl Wright resolved CONNECTORS-502.
------------------------------------
Resolution: Fixed
Assignee: Karl Wright
> Crawling with MySQL as a backend shows signs of transactional integrity problems
> --------------------------------------------------------------------------------
>
> Key: CONNECTORS-502
> URL: https://issues.apache.org/jira/browse/CONNECTORS-502
> Project: ManifoldCF
> Issue Type: Bug
> Components: Framework crawler agent
> Affects Versions: ManifoldCF 0.6
> Environment: MySQL 5.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.7
>
>
> When running the webcrawler-loadtests-mysql target, I got the following error:
> {code}
> ERROR 2012-08-12 20:59:35,664 (Worker thread '10') - Exception tossed: Unexpected jobqueue status - record id 1344819115577, expecting active status, saw 0
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected jobqueue status - record id 1344819115577, expecting active status, saw 0
> at org.apache.manifoldcf.crawler.jobs.JobQueue.updateOrHopcountRemoveRecord(JobQueue.java:800)
> at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentHopcountRemovalMultiple(JobManager.java:2583)
> at org.apache.manifoldcf.crawler.system.WorkerThread.processJobQueueHopcountRemovals(WorkerThread.java:1100)
> at org.apache.manifoldcf.crawler.system.WorkerThread.processHopcountRemovalLists(WorkerThread.java:952)
> at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:790)
> {code}
> This indicates that although the stuffer thread thought it had put the document into the ACTIVE state, it had either never made it to the ACTIVE state or had been overridden by some other code. Either way, MySQL transactions do not seem to be working properly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira