Disabling lease check for read repository

[yogesh upadhyay <up...@gmail.com>]

Hello,

We are using Oak with Mysql. Our app is deployed in different colos across
north America and Asia. Currently our write DB lives in north america and
read DB across all other Colos. We are constantly getting lease failure
error from Asia colo *"**This oak instance failed to update the lease in
time and can therefore no longer access this DocumentNodeStore."* Looking
at
https://github.com/apache/jackrabbit-oak/blob/b5155082fef948fb004a3fb9caaaf79e3d5052d7/oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/ClusterNodeInfo.java
it's
not clear

   - what's the role of lease check if you are using Mysql.
   - Can we disable leasecheck for read repository ?
   - What is the consequence of disabling lease check in prod ? (repository
   corruption ?)


Any insight will be useful.

Yogesh

Re: Disabling lease check for read repository

[yogesh upadhyay <up...@gmail.com>]

Thank You Marcel.

Yogesh

On Mon, Oct 7, 2019, 1:55 AM Marcel Reutegger <mr...@adobe.com.invalid>
wrote:

> Hi Yogesh,
>
> On 04.10.19, 00:48, "yogesh upadhyay" <up...@gmail.com> wrote:
> > - what's the role of lease check if you are using Mysql.
>
> The lease check is part of the clusterId mechanism that gives each cluster
> node
> a unique id. This is not specific to the DocumentStore backend in use and
> is
> needed for any backend.
>
> See also
> https://jackrabbit.apache.org/oak/docs/nodestore/documentmk.html#cluster-node-metadata
>
> > - Can we disable leasecheck for read repository ?
>
> This is already the case when you construct a read-only DocumentNodeStore
> [0].
>
> > - What is the consequence of disabling lease check in prod ? (repository
> > corruption ?)
>
> One of the problems that can occur are described here:
> https://issues.apache.org/jira/browse/OAK-3883
>
> Consistency can be at risk when lease checks are disabled and recovery is
> performed
> by another cluster node even though the (temporarily?) unresponsive
> cluster node
> is still alive and then continues to write.
>
> There are also implications on the Sling Topology when lease checks are
> disabled.
> The system may end up in a situation where multiple leaders exist.
>
> Regards
>  Marcel
>
> [0]
> https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.18.0/oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreBuilder.java#L267
>
>

Re: Disabling lease check for read repository

[Marcel Reutegger <mr...@adobe.com.INVALID>]

Hi Yogesh,

On 04.10.19, 00:48, "yogesh upadhyay" <up...@gmail.com> wrote:
> - what's the role of lease check if you are using Mysql.

The lease check is part of the clusterId mechanism that gives each cluster node
a unique id. This is not specific to the DocumentStore backend in use and is
needed for any backend.

See also https://jackrabbit.apache.org/oak/docs/nodestore/documentmk.html#cluster-node-metadata

> - Can we disable leasecheck for read repository ?

This is already the case when you construct a read-only DocumentNodeStore [0].

> - What is the consequence of disabling lease check in prod ? (repository
> corruption ?)

One of the problems that can occur are described here:
https://issues.apache.org/jira/browse/OAK-3883

Consistency can be at risk when lease checks are disabled and recovery is performed
by another cluster node even though the (temporarily?) unresponsive cluster node
is still alive and then continues to write.

There are also implications on the Sling Topology when lease checks are disabled.
The system may end up in a situation where multiple leaders exist.

Regards
 Marcel

[0] https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.18.0/oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreBuilder.java#L267