You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by "Ralph Benjamin Ruijs (JIRA)" <ji...@apache.org> on 2010/04/16 17:22:25 UTC
[jira] Created: (CONNECTORS-27) Add support for observation to the
crawler agent
Add support for observation to the crawler agent
------------------------------------------------
Key: CONNECTORS-27
URL: https://issues.apache.org/jira/browse/CONNECTORS-27
Project: Lucene Connector Framework
Issue Type: New Feature
Components: Framework crawler agent
Reporter: Ralph Benjamin Ruijs
Priority: Minor
When crawling a large repository, it could take a lot of time before changes are propagated to Solr. You can add an event listener to the repository, and be notified about changes. The crawler will ensure you have a complete copy in case of missed events.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CONNECTORS-27) Add support for observation to the
crawler agent
Posted by "Ralph Benjamin Ruijs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-27?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ralph Benjamin Ruijs updated CONNECTORS-27:
-------------------------------------------
Attachment: Added_observation_logic_to_the_crawler.patch
This patch adds support for observation to the crawler. It creates a new thread in the crawler agent which polls the active jobs. A connector needs to implement the actual observation and gets a reserved connection to listen on.
> Add support for observation to the crawler agent
> ------------------------------------------------
>
> Key: CONNECTORS-27
> URL: https://issues.apache.org/jira/browse/CONNECTORS-27
> Project: Lucene Connector Framework
> Issue Type: New Feature
> Components: Framework crawler agent
> Reporter: Ralph Benjamin Ruijs
> Priority: Minor
> Attachments: Added_observation_logic_to_the_crawler.patch
>
>
> When crawling a large repository, it could take a lot of time before changes are propagated to Solr. You can add an event listener to the repository, and be notified about changes. The crawler will ensure you have a complete copy in case of missed events.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CONNECTORS-27) Add support for observation to
the crawler agent
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857861#action_12857861 ]
Karl Wright commented on CONNECTORS-27:
---------------------------------------
I understand what your proposed infrastructure does. What I don't understand is the use case. It seems to me like all you are doing is adding a poll method to a repository connector. But there already is one. Can you provide a case which demonstrates the need for this infrastructure?
> Add support for observation to the crawler agent
> ------------------------------------------------
>
> Key: CONNECTORS-27
> URL: https://issues.apache.org/jira/browse/CONNECTORS-27
> Project: Lucene Connector Framework
> Issue Type: New Feature
> Components: Framework crawler agent
> Reporter: Ralph Benjamin Ruijs
> Priority: Minor
> Attachments: Added_observation_logic_to_the_crawler.patch
>
>
> When crawling a large repository, it could take a lot of time before changes are propagated to Solr. You can add an event listener to the repository, and be notified about changes. The crawler will ensure you have a complete copy in case of missed events.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira