Retry mechanism authority connectors

[ju...@francelabs.com]

Hi,

I noticed that with authority connectors that perform API calls, seldom
issues like network failures lead to DEAD_AUTHORITY. Assuming one wants to
have a retry/fail mechanism, would you suggest that each connector should
implement its own, or would you rather suggest - as it is the case with
other connectors - to throw a specific exception (like the
ServiceInterruption) that triggers an already implemented retry/fail
mechanism ?

Regards,
Julien Massiera



 

RE: Retry mechanism authority connectors

[ju...@francelabs.com]

Yes, makes sense, thanks for the answer.

-----Message d'origine-----
De : Karl Wright <da...@gmail.com> 
Envoyé : lundi 11 janvier 2021 18:58
À : dev <de...@manifoldcf.apache.org>
Objet : Re: Retry mechanism authority connectors

When I was designing this, I thought long and hard about this question.
Since it is security related, in the end I decided to let the application who called the authority service have whatever retry logic made sense.

I don't see any objection to retrying certain errors at the connector level, but for framework handling, remember that ServiceInterruption on the crawler side is hooked into the framework in a way that allows queue-based retries over a long period of time.  ServiceInterruption errors thrown can specify a specific retry schedule which the framework performs.  That works only because, if a single document fails, there are plenty of other documents to process.  For the authority service, the back-off would be just dead wait time, and there's no integration with anything else in the framework at all.  So it really doesn't matter where retries happen.

Karl


On Mon, Jan 11, 2021 at 11:10 AM <ju...@francelabs.com> wrote:

> Hi,
>
> I noticed that with authority connectors that perform API calls, 
> seldom issues like network failures lead to DEAD_AUTHORITY. Assuming 
> one wants to have a retry/fail mechanism, would you suggest that each 
> connector should implement its own, or would you rather suggest - as 
> it is the case with other connectors - to throw a specific exception 
> (like the
> ServiceInterruption) that triggers an already implemented retry/fail 
> mechanism ?
>
> Regards,
> Julien Massiera
>
>
>
>
>
>

Re: Retry mechanism authority connectors

[Karl Wright <da...@gmail.com>]

When I was designing this, I thought long and hard about this question.
Since it is security related, in the end I decided to let the application
who called the authority service have whatever retry logic made sense.

I don't see any objection to retrying certain errors at the connector
level, but for framework handling, remember that ServiceInterruption on the
crawler side is hooked into the framework in a way that allows queue-based
retries over a long period of time.  ServiceInterruption errors thrown can
specify a specific retry schedule which the framework performs.  That works
only because, if a single document fails, there are plenty of other
documents to process.  For the authority service, the back-off would be
just dead wait time, and there's no integration with anything else in the
framework at all.  So it really doesn't matter where retries happen.

Karl


On Mon, Jan 11, 2021 at 11:10 AM <ju...@francelabs.com> wrote:

> Hi,
>
> I noticed that with authority connectors that perform API calls, seldom
> issues like network failures lead to DEAD_AUTHORITY. Assuming one wants to
> have a retry/fail mechanism, would you suggest that each connector should
> implement its own, or would you rather suggest - as it is the case with
> other connectors - to throw a specific exception (like the
> ServiceInterruption) that triggers an already implemented retry/fail
> mechanism ?
>
> Regards,
> Julien Massiera
>
>
>
>
>
>