[jira] [Updated] (IGNITE-19227) Wait for schema awailability out of JRaft threads

["Roman Puchkovskiy (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IGNITE-19227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Roman Puchkovskiy updated IGNITE-19227:
---------------------------------------
    Description: 
According to [https://cwiki.apache.org/confluence/display/IGNITE/IEP-98%3A+Schema+Synchronization#IEP98:SchemaSynchronization-Waitingforsafetimeinthepast] , we might need to wait for schema availability when fetching a schema. If such waits happen inside a PartitionListener, JRaft threads might be blocked for a noticeable amount of time (maybe even seconds). We should avoid this.
h3. In RW transactions

When a primary node is going to process a request, it waits till it has all the schema versions for the corresponding timestamp (beginTs or commitTs) Top (i.e. that MS SafeTime >= Top). {*}The wait happens outside of JRaft threads{*}. Then it obtains the global schema revision SR of the latest schema update that is not later than the corresponding timestamp. It then builds a command (putting that SR inside) and submits it to RAFT.

When an AppendEntriesRequest is built, Replicator inspects all the entries it includes in it, extracts SRs from each of them, takes max of them (as MSR, for ‘max schema revision’) and puts it in the AppendEntriesRequest.

When the request is processed by a follower/learner, it compares the MSR from the request with its locally known MSR (in the Catalog). If the request’s MSR > local MSR, then the request is rejected (with reason EBUSY). It will be retried by the leader after some time. As an optimization, we might wait for some time in hope that the local MSR catches up with the request’s MSR.

As we need an additional field in AppendEntriesRequest that will only be used by partition groups, we could add a generic container for properties to this interface, like Map<String, Object> extras().

To extract the SR from a command, we might just deserialize it completely, but this requires a lot of work that is not necessary. We might serialize commands having SR in a special way (putting SR in the very first bytes of the message) to make its retrieval effective.

As the primary has already made sure that it has the schema versions needed to execute the command, no waits will be needed on the primary node while executing the RAFT command.

As secondaries/learners refuse AppendEntries which they cannot execute waitless, they will not have to wait at all in JRaft threads.

A case when the RAFT leader is not collocated with the primary is possible. We can add the same validation for ActionRequests: pass the required SR inside an ActionRequest, validate it in ActionRequestProcessor and reject requests having SR above the local MSR.
h3. In RO transactions

When processing an RO transaction, we just wait for MS SafeTime. This is made out of RAFT, so no special measures are needed.

  was:
According to [https://cwiki.apache.org/confluence/display/IGNITE/IEP-98%3A+Schema+Synchronization#IEP98:SchemaSynchronization-Waitingforsafetimeinthepast] , we might need to wait for schema availability when fetching a schema. If such waits happen inside a PartitionListener, JRaft threads might be blocked for a noticeable amount of time (maybe even seconds). We should avoid this.

For RW transactions, we can fetch the schema needed by the operation on the primary replica before submitting a RAFT command to RAFT, so that the possible wait happen in a user's thread.

For RO transactions, this is not a problem because we don't use RAFT for RO transactions.


> Wait for schema awailability out of JRaft threads
> -------------------------------------------------
>
>                 Key: IGNITE-19227
>                 URL: https://issues.apache.org/jira/browse/IGNITE-19227
>             Project: Ignite
>          Issue Type: Improvement
>            Reporter: Roman Puchkovskiy
>            Priority: Major
>              Labels: ignite-3
>
> According to [https://cwiki.apache.org/confluence/display/IGNITE/IEP-98%3A+Schema+Synchronization#IEP98:SchemaSynchronization-Waitingforsafetimeinthepast] , we might need to wait for schema availability when fetching a schema. If such waits happen inside a PartitionListener, JRaft threads might be blocked for a noticeable amount of time (maybe even seconds). We should avoid this.
> h3. In RW transactions
> When a primary node is going to process a request, it waits till it has all the schema versions for the corresponding timestamp (beginTs or commitTs) Top (i.e. that MS SafeTime >= Top). {*}The wait happens outside of JRaft threads{*}. Then it obtains the global schema revision SR of the latest schema update that is not later than the corresponding timestamp. It then builds a command (putting that SR inside) and submits it to RAFT.
> When an AppendEntriesRequest is built, Replicator inspects all the entries it includes in it, extracts SRs from each of them, takes max of them (as MSR, for ‘max schema revision’) and puts it in the AppendEntriesRequest.
> When the request is processed by a follower/learner, it compares the MSR from the request with its locally known MSR (in the Catalog). If the request’s MSR > local MSR, then the request is rejected (with reason EBUSY). It will be retried by the leader after some time. As an optimization, we might wait for some time in hope that the local MSR catches up with the request’s MSR.
> As we need an additional field in AppendEntriesRequest that will only be used by partition groups, we could add a generic container for properties to this interface, like Map<String, Object> extras().
> To extract the SR from a command, we might just deserialize it completely, but this requires a lot of work that is not necessary. We might serialize commands having SR in a special way (putting SR in the very first bytes of the message) to make its retrieval effective.
> As the primary has already made sure that it has the schema versions needed to execute the command, no waits will be needed on the primary node while executing the RAFT command.
> As secondaries/learners refuse AppendEntries which they cannot execute waitless, they will not have to wait at all in JRaft threads.
> A case when the RAFT leader is not collocated with the primary is possible. We can add the same validation for ActionRequests: pass the required SR inside an ActionRequest, validate it in ActionRequestProcessor and reject requests having SR above the local MSR.
> h3. In RO transactions
> When processing an RO transaction, we just wait for MS SafeTime. This is made out of RAFT, so no special measures are needed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)