[jira] [Closed] (ARTEMIS-2941) Improve JDBC HA connection resiliency

["Clebert Suconic (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARTEMIS-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Clebert Suconic closed ARTEMIS-2941.
------------------------------------
    Fix Version/s: 2.16.0
       Resolution: Fixed

> Improve JDBC HA connection resiliency
> -------------------------------------
>
>                 Key: ARTEMIS-2941
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-2941
>             Project: ActiveMQ Artemis
>          Issue Type: Improvement
>          Components: Broker
>    Affects Versions: 2.15.0
>            Reporter: Francesco Nigro
>            Assignee: Francesco Nigro
>            Priority: Major
>             Fix For: 2.16.0
>
>          Time Spent: 2h
>  Remaining Estimate: 0h
>
> This is aiming to replace the restart enhancement feature of https://issues.apache.org/jira/browse/ARTEMIS-2918 because this last one is too dangerous due to the numerous potential leaks that a server in production could hit by allowing it to restart while keeping the Java process around. 
> Currently, JDBC HA uses an expiration time on locks that mark the time by which a server instance is allowed to keep a specific role, dependent by the owned lock (live or backup).
> Right now, the first failed attempt to renew such expiration time force a broker to shutdown immediately, while it could be more "relaxed" and just keep retry until the very end ie when the expiration time is approaching to end.
>  
> The only concern of this feature is related to the relation between the broker wall-clock time and the DBMS one, that's used to set the expiration time and that should be within certain margins.
> For this last part I'm aware that classic ActiveMQ lease locks use some configuration parameter to set the magnitude of the allowed difference (and to compute some base offset too).
>  
> Right now this feature seems more risk-free and appealing then  https://issues.apache.org/jira/browse/ARTEMIS-2918, given it narrows the scope of it to what's the very core issue ie a more resilient behaviour on JDBC lost connectivity.
>  
> To understand the implications of such change, consider a shared store HA pair with configured 60 seconds of expiration time:
>  # DBMS goes down
>  # an in-flight persistent operation on the live data store cause the live broker to kill itself immediately, because no reliable storage is connected
>  # backup is unable to renew its backup lease lock
>  # DBMS goes up in time, before the backup lock local expiration time is ended
>  # backup is able to renew its backup lease lock and retrieve the very last state of live (that was live) and, if no script is configured to restart the live, to failover and take its role
>  # backup is now live and able to serve clients
>  
>  
> There are 2 legit questions re potential improvements on this:
>  # why the live cannot keep re-trying I/O (on the journal, paging or large messages) until its local expiration time end? 
>  # why the live isn't just returning back an I/O error to the clients?
>  
> The former is complex: the main problem I see is from the resource utilization point of view; keeping an accumulating backlog of pending requests, blocked awaiting the last one for an arbitrary long time will probably cause the broker memory to blown up, to not mention that clients will timed out too.
> The latter seems more appealing, because will allow clients to fail fast, but it would affect the current semantic we use on the broker storage operations and I need more investigation to understand how to implement it.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)