You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by "Karl Wright (JIRA)" <ji...@apache.org> on 2010/06/02 16:33:46 UTC
[jira] Created: (CONNECTORS-41) Add hooks to output connectors for
receiving event notifications, specifically job start, job end, etc.
Add hooks to output connectors for receiving event notifications, specifically job start, job end, etc.
-------------------------------------------------------------------------------------------------------
Key: CONNECTORS-41
URL: https://issues.apache.org/jira/browse/CONNECTORS-41
Project: Lucene Connector Framework
Issue Type: Improvement
Components: Framework core
Reporter: Karl Wright
Priority: Minor
Currently there is no logic that informs an output connection of a job start, end, deletion, or other activity. While this would seem to have little to do with an output connector, this feature has been requested by Jack Krupansky as a potential way of deciding when to tell Solr to commit documents, rather than leave it up to Solr's configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-41) Add hooks to output connectors
for receiving event notifications, specifically job start, job end, etc.
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904622#action_12904622 ]
Karl Wright commented on CONNECTORS-41:
---------------------------------------
I think we're discussing two entirely distinct features here.
Feature 1: Let the output connector know that a job is finished, so that it can flush whatever internal buffering etc. it has been doing (e.g. tell solr to commit).
Feature 2: Provide some general way of monitoring the progress of jobs etc.
Feature 2 is already met by the API, in my opinion. It's a polling-style fulfillment of the requirement, but it does exist. There doesn't seem to me to yet be a requirement that a notification-style API be provided also, despite the stated use case.
Feature 1 is what I consider to be the use case for this current ticket.
> Add hooks to output connectors for receiving event notifications, specifically job start, job end, etc.
> -------------------------------------------------------------------------------------------------------
>
> Key: CONNECTORS-41
> URL: https://issues.apache.org/jira/browse/CONNECTORS-41
> Project: Apache Connectors Framework
> Issue Type: Improvement
> Components: Framework core
> Reporter: Karl Wright
> Priority: Minor
>
> Currently there is no logic that informs an output connection of a job start, end, deletion, or other activity. While this would seem to have little to do with an output connector, this feature has been requested by Jack Krupansky as a potential way of deciding when to tell Solr to commit documents, rather than leave it up to Solr's configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-41) Add hooks to output connectors
for receiving event notifications, specifically job start, job end, etc.
Posted by "Jack Krupansky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904610#action_12904610 ]
Jack Krupansky commented on CONNECTORS-41:
------------------------------------------
To be clear about the use case, the notification is not to an output connector or output connection per se, but to an external process that is trying to monitor the job status. Kind of a reverse API. The URL that job status notifications should be sent to might be in the same process as Solr or another process that is monitoring Solr. Further, this feature should be of value for any type of output connector, although Solr is my current main interest.
> Add hooks to output connectors for receiving event notifications, specifically job start, job end, etc.
> -------------------------------------------------------------------------------------------------------
>
> Key: CONNECTORS-41
> URL: https://issues.apache.org/jira/browse/CONNECTORS-41
> Project: Apache Connectors Framework
> Issue Type: Improvement
> Components: Framework core
> Reporter: Karl Wright
> Priority: Minor
>
> Currently there is no logic that informs an output connection of a job start, end, deletion, or other activity. While this would seem to have little to do with an output connector, this feature has been requested by Jack Krupansky as a potential way of deciding when to tell Solr to commit documents, rather than leave it up to Solr's configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-41) Add hooks to output connectors
for receiving event notifications, specifically job start, job end, etc.
Posted by "Jack Krupansky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904618#action_12904618 ]
Jack Krupansky commented on CONNECTORS-41:
------------------------------------------
The notification is certainly "associated" with the output connection... or is it really associated with the job?
Originally, my thinking had been that the notification URL would be specified as part of the output connection, but maybe it is an output-specific parameter that gets specified for the job. It could be either.
OTOH, maybe a "notification connector" makes more sense and is more general rather than just something to use for Solr. Also, that might provide the way to implement an optional commit for Solr, as a simple notification connection, so that ACF core itself doesn't know or care about Solr or commits or any of that. I think the concept of a notification connector makes sense, but is not essential for release 0.1 or 0.5.
I'm open to suggestions. We can do it real simple as a parameter for the Solr output connector or the job, or we could be more general. Tough call. If you feel up to doing the more general feature, fine, but the simple notification URL feature is all that is essential.
Also, to be clear about the use case, it is not just Solr commit, but some external app might just want to notify a user that their job has finished and to do whatever other (beyond Solr commit) processing may be needed upon job completion.
> Add hooks to output connectors for receiving event notifications, specifically job start, job end, etc.
> -------------------------------------------------------------------------------------------------------
>
> Key: CONNECTORS-41
> URL: https://issues.apache.org/jira/browse/CONNECTORS-41
> Project: Apache Connectors Framework
> Issue Type: Improvement
> Components: Framework core
> Reporter: Karl Wright
> Priority: Minor
>
> Currently there is no logic that informs an output connection of a job start, end, deletion, or other activity. While this would seem to have little to do with an output connector, this feature has been requested by Jack Krupansky as a potential way of deciding when to tell Solr to commit documents, rather than leave it up to Solr's configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CONNECTORS-41) Add hooks to output connectors for
receiving event notifications, specifically job start, job end, etc.
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-41?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karl Wright resolved CONNECTORS-41.
-----------------------------------
Fix Version/s: LCF Release 0.5
Resolution: Fixed
Fix checked in, r991295.
> Add hooks to output connectors for receiving event notifications, specifically job start, job end, etc.
> -------------------------------------------------------------------------------------------------------
>
> Key: CONNECTORS-41
> URL: https://issues.apache.org/jira/browse/CONNECTORS-41
> Project: Apache Connectors Framework
> Issue Type: Improvement
> Components: Framework core
> Reporter: Karl Wright
> Assignee: Karl Wright
> Priority: Minor
> Fix For: LCF Release 0.5
>
>
> Currently there is no logic that informs an output connection of a job start, end, deletion, or other activity. While this would seem to have little to do with an output connector, this feature has been requested by Jack Krupansky as a potential way of deciding when to tell Solr to commit documents, rather than leave it up to Solr's configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CONNECTORS-41) Add hooks to output connectors for
receiving event notifications, specifically job start, job end, etc.
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-41?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karl Wright reassigned CONNECTORS-41:
-------------------------------------
Assignee: Karl Wright
> Add hooks to output connectors for receiving event notifications, specifically job start, job end, etc.
> -------------------------------------------------------------------------------------------------------
>
> Key: CONNECTORS-41
> URL: https://issues.apache.org/jira/browse/CONNECTORS-41
> Project: Apache Connectors Framework
> Issue Type: Improvement
> Components: Framework core
> Reporter: Karl Wright
> Assignee: Karl Wright
> Priority: Minor
>
> Currently there is no logic that informs an output connection of a job start, end, deletion, or other activity. While this would seem to have little to do with an output connector, this feature has been requested by Jack Krupansky as a potential way of deciding when to tell Solr to commit documents, rather than leave it up to Solr's configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-41) Add hooks to output connectors
for receiving event notifications, specifically job start, job end, etc.
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904582#action_12904582 ]
Karl Wright commented on CONNECTORS-41:
---------------------------------------
I looked at this in some detail yesterday. The prime implementation option is to add notification methods to IOutputConnector, so that job events get reported to the connector when the job is being terminated. The issue in this case is going to be how exactly to handle ServiceInterruption exceptions that occur at the time of the notification into the connector. This is not hypothetical because in the Solr case a notification may well fail, or it may take a very long time (many minutes). Usually when there is a possibility of extended interaction it argues for an additional state in the database.
It looks like it will not be possible to delay the change of the job status, since that takes place in a transaction. If the notification fails, the job could otherwise be left in the "running" state, and a retry would naturally occur until the commit succeeded. But that doesn't look possible given the transaction structure.
An alternative (non-notification) method of handling a commit request would require the commit to take place as part of the output connector's poll() method. This is a little better to work with because the poll() method will naturally retry in any case. The issue here is that there would be no *guarantee* of a commit taking place at all, since it isn't part of the connector contract that the connection must continue to exist for any period of time, which I think would violate the spirit of this ticket.
If explicit notification takes place, we could just report any error, and forget about it, rather than keeping the job alive for a retry. That, too, would mean that a commit was not guaranteed to occur during the job's lifecycle.
The final alternative, which would seemingly work, would involve there being two job shutdown states - one prior to notification, and the second after notification. The first state would be entered based on the current shutdown logic. The second state would be entered only after the notification had been successful. Thus, the notification *could* be called more than once, if there were errors, or if the crawler were shut down and restarted before the state transition was completed. The extra state would also allow the job's pre-notification status to be noted in the crawler ui.
Because of the potential time delay of a commit, it is probably best for the first to second shutdown state transition to be handled by a separate thread, or family of threads.
> Add hooks to output connectors for receiving event notifications, specifically job start, job end, etc.
> -------------------------------------------------------------------------------------------------------
>
> Key: CONNECTORS-41
> URL: https://issues.apache.org/jira/browse/CONNECTORS-41
> Project: Apache Connectors Framework
> Issue Type: Improvement
> Components: Framework core
> Reporter: Karl Wright
> Priority: Minor
>
> Currently there is no logic that informs an output connection of a job start, end, deletion, or other activity. While this would seem to have little to do with an output connector, this feature has been requested by Jack Krupansky as a potential way of deciding when to tell Solr to commit documents, rather than leave it up to Solr's configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-41) Add hooks to output connectors
for receiving event notifications, specifically job start, job end, etc.
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904611#action_12904611 ]
Karl Wright commented on CONNECTORS-41:
---------------------------------------
Does it makes no sense to create an entirely new kind of connector just for notifications of this sort? So when you create a job you select THREE different kinds of connection (repository, output, and notification)? That seems like supreme overkill to me, and I can well argue that this kind of notification really is only useful to an output connection in any case.
> Add hooks to output connectors for receiving event notifications, specifically job start, job end, etc.
> -------------------------------------------------------------------------------------------------------
>
> Key: CONNECTORS-41
> URL: https://issues.apache.org/jira/browse/CONNECTORS-41
> Project: Apache Connectors Framework
> Issue Type: Improvement
> Components: Framework core
> Reporter: Karl Wright
> Priority: Minor
>
> Currently there is no logic that informs an output connection of a job start, end, deletion, or other activity. While this would seem to have little to do with an output connector, this feature has been requested by Jack Krupansky as a potential way of deciding when to tell Solr to commit documents, rather than leave it up to Solr's configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-41) Add hooks to output connectors
for receiving event notifications, specifically job start, job end, etc.
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904614#action_12904614 ]
Karl Wright commented on CONNECTORS-41:
---------------------------------------
I also claim that even if a wholly-as-yet-to-be-cooked-up external process is notified, you will have (a) the possibility of error, and (b) the probability of delay, because you are going to want this behavior to be synchronous. So all my original reasoning still applies. I would furthermore argue that we really want the Solr output connector to work against the existing Solr artifact. So if you have something special in mind, please either offer it as a patch to Solr, or write your own dedicated output connector.
> Add hooks to output connectors for receiving event notifications, specifically job start, job end, etc.
> -------------------------------------------------------------------------------------------------------
>
> Key: CONNECTORS-41
> URL: https://issues.apache.org/jira/browse/CONNECTORS-41
> Project: Apache Connectors Framework
> Issue Type: Improvement
> Components: Framework core
> Reporter: Karl Wright
> Priority: Minor
>
> Currently there is no logic that informs an output connection of a job start, end, deletion, or other activity. While this would seem to have little to do with an output connector, this feature has been requested by Jack Krupansky as a potential way of deciding when to tell Solr to commit documents, rather than leave it up to Solr's configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.