[jira] [Updated] (IGNITE-18539) Implement build procedure for new indexes

["Kirill Tkalenko (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IGNITE-18539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kirill Tkalenko updated IGNITE-18539:
-------------------------------------
    Fix Version/s: 3.0.0-beta2

> Implement build procedure for new indexes
> -----------------------------------------
>
>                 Key: IGNITE-18539
>                 URL: https://issues.apache.org/jira/browse/IGNITE-18539
>             Project: Ignite
>          Issue Type: Improvement
>            Reporter: Ivan Bessonov
>            Assignee: Kirill Tkalenko
>            Priority: Major
>              Labels: ignite-3
>             Fix For: 3.0.0-beta2
>
>
> First of all, "build" is the process of moving the index from "WO" to "RW" state in schema definitions. Finished build creates a new schema version. This means that new transactions, started after a specific timestamp, will be able to use this index in their plans.
> Now, when index is created, we also create a table somewhere that looks similar to this:
>  
> {code:java}
> indexComplation.<indexId>.0 = false
> ...
> indexComplation.<indexId>.N = false {code}
> It is used to track completion of index build for each individual partition. Once it's built bor each partition, WO -> RW state transition may occur.
>  
> For each individual partition, leader/coordinator/leaseholder/whatever node starts iterating all rowIds in the partition and replicates the "index build command" for each row (or using batches). This procedure can be described with this pseudocode:
> {code:java}
> RowId curRowId = minRowId();
> // Get existing row id higher or equal than the parameter.
> curRowId = partition.findRowId(curRowId);
> while (curRowId != null) {
>     // Put that into a state machine.
>     replicateIndexRowRebuild(indexId, curRowId);
>     // Compute the next row id candidate.
>     curRowId = increment(curRowId);
>     // Get the real next row id value.
>     if (curRowId != null) {
>         curRowId = partition.findRowId(curRowId);
>     }
> }
> indexBuildCompleted(indexId, partitionId);{code}
> Such replication doesn't require too much traffic, and it guarantees consistency between replicas, which is crucial.
> Once index is created and WO, all insertions should update it "unconditionally", independently from the build process. This gives us cooperation and simplifies the way we guarantee consistency between row store and index.
> Easy optimization: during regular load, we can avoid updating WO indexes if rowId > lastRowId for that particular index, because it will be updated later anyway.
>  
> About that: we should store "lastUpdatedRowId" for every index that's not yet completed its build. We need it for 2 reasons:
>  * leader might be changed
>  * cluster may be restarted
> In both cases we simply start where we stopped last time, instead of starting from scratch, like it was in Ignite 2.x.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)