You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by "Karl Wright (JIRA)" <ji...@apache.org> on 2013/11/12 09:29:19 UTC
[jira] [Commented] (CONNECTORS-13) We should augment file-based
synchronization to allow a process/service-based synchronization as well
[ https://issues.apache.org/jira/browse/CONNECTORS-13?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13819921#comment-13819921 ]
Karl Wright commented on CONNECTORS-13:
---------------------------------------
Still need:
- Basic testing and debugging - would be great if we came up with a unit test that exercised the functionality in a multithreaded way
- Code review for how zookeeper error conditions are being handled
- A utility for loading a properties file into zookeeper configuration
- Example code/scripts (probably will call it cluster-example)
> We should augment file-based synchronization to allow a process/service-based synchronization as well
> -----------------------------------------------------------------------------------------------------
>
> Key: CONNECTORS-13
> URL: https://issues.apache.org/jira/browse/CONNECTORS-13
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Framework core
> Affects Versions: ManifoldCF 0.1, ManifoldCF 0.2
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 1.5
>
>
> The current implementation relies on the file system to synchronize activity between various LCF processes. This has several downsides: first, it is possible to get the file system into a state that is corrupted (by killing processes); second, this limits the future ability to spread crawler workload over multiple machines.
> It should be reasonably straightforward, and probably more resilient, to introduce a "synchronization process", which all other LCF processes talk to in order to manage locks, shared data, and other synchronization activities.
--
This message was sent by Atlassian JIRA
(v6.1#6144)