[jira] Updated: (CONNECTORS-145) The logic for dealing with a downed Search Engine in ExpireThread is not optimal

["Karl Wright (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CONNECTORS-145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Karl Wright updated CONNECTORS-145:
-----------------------------------

    Component/s: Framework crawler agent
    Description: 
The expire stuffer thread puts documents from both the PENDING and PENDINGPURGATORY states into the ACTIVE and ACTIVEPURGATORY states.  The expire threads should deal with errors by moving the candidate document back into its original state for a later expiration attempt, but right now the Expire Thread simply blocks and retries, essentially using up an expire thread for the duration of the outage.
In a time when there was only one output connection, this was acceptable logic, but since there are multiple such connections possible now, it is a potential liability.

Thus, logic must change in the Expire Thread to perform the appropriate error recovery.

The other thread family that has this problem is the Document Delete Thread family.  These threads will require some thought to fix because there is currently no deletion scheduling field in the jobqueue database table, and yet we'd need one if we were going to fix this problem in an appropriate manner.  However, we can probably reuse the checktime field for this purpose if we are clever.


  was:
The expire stuffer thread puts documents from both the PENDING and PENDINGPURGATORY states into the BEINGDELETED state.  But this state is insufficient to handle proper error recovery in the case of a down search engine blocking expiration.  What should happen is that the error from the failed expiration attempt should move the candidate document back into its original state for a later expiration attempt, but right now the Expire Thread simply blocks and retries, essentially using up an expire thread for the duration of the outage.
In a time when there was only one output connection, this was acceptable logic, but since there are multiple such connections possible now, it is a potential liability.

The way forward is to introduce a new document state, BEINGDELETEDPURGATORY, and insure that the mapping between PENDING->BEINGDELETED and PENDINGPURGATORY->BEINGDELETEDPURGATORY is preserved.  Then, logic must change in the Expire Thread to perform the appropriate error recovery.

The other thread family that has this problem is the Document Delete Thread family.  These threads will require some thought to fix because there is currently no deletion scheduling field in the jobqueue database table, and yet we'd need one if we were going to fix this problem in an appropriate manner.  However, we can probably reuse the checktime field for this purpose if we are clever.


        Summary: The logic for dealing with a downed Search Engine in ExpireThread is not optimal  (was: The BEINGDELETED document state is insufficient to allow proper error recovery from a search engine that's down)

> The logic for dealing with a downed Search Engine in ExpireThread is not optimal
> --------------------------------------------------------------------------------
>
>                 Key: CONNECTORS-145
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-145
>             Project: ManifoldCF
>          Issue Type: Bug
>          Components: Framework crawler agent
>            Reporter: Karl Wright
>
> The expire stuffer thread puts documents from both the PENDING and PENDINGPURGATORY states into the ACTIVE and ACTIVEPURGATORY states.  The expire threads should deal with errors by moving the candidate document back into its original state for a later expiration attempt, but right now the Expire Thread simply blocks and retries, essentially using up an expire thread for the duration of the outage.
> In a time when there was only one output connection, this was acceptable logic, but since there are multiple such connections possible now, it is a potential liability.
> Thus, logic must change in the Expire Thread to perform the appropriate error recovery.
> The other thread family that has this problem is the Document Delete Thread family.  These threads will require some thought to fix because there is currently no deletion scheduling field in the jobqueue database table, and yet we'd need one if we were going to fix this problem in an appropriate manner.  However, we can probably reuse the checktime field for this purpose if we are clever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.