[jira] [Commented] (CONNECTORS-13) We should move to eliminate process synchronization via shared file system, and use a process/service instead

["Jan Høydahl (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CONNECTORS-13?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13771119#comment-13771119 ] 

Jan Høydahl commented on CONNECTORS-13:
---------------------------------------

Have a look at how Solr has solved this. Would be sweet if MCF used ZK for locks as well as config, then it could share the same ZK ensemble as Solr.

I know Solr elects one of the nodes as Overseer, which then is in charge of updating the cluster state, cleaning stale locks etc. It has a work-queue right in ZK and if the overseer dies, a new is elected.
                
> We should move to eliminate process synchronization via shared file system, and use a process/service instead
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: CONNECTORS-13
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-13
>             Project: ManifoldCF
>          Issue Type: Improvement
>          Components: Framework core
>    Affects Versions: ManifoldCF 0.1, ManifoldCF 0.2
>            Reporter: Karl Wright
>             Fix For: ManifoldCF next
>
>
> The current implementation relies on the file system to synchronize activity between various LCF processes.  This has several downsides: first, it is possible to get the file system into a state that is corrupted (by killing processes); second, this limits the future ability to spread crawler workload over multiple machines.
> It should be reasonably straightforward, and probably more resilient, to introduce a "synchronization process", which all other LCF processes talk to in order to manage locks, shared data, and other synchronization activities.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira