[jira] [Updated] (IGNITE-17700) Realtime CDC

["Nikita Amelchev (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IGNITE-17700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nikita Amelchev updated IGNITE-17700:
-------------------------------------
    Fix Version/s:     (was: 2.16)

> Realtime CDC
> ------------
>
>                 Key: IGNITE-17700
>                 URL: https://issues.apache.org/jira/browse/IGNITE-17700
>             Project: Ignite
>          Issue Type: Improvement
>            Reporter: Nikolay Izhikov
>            Assignee: Maksim Timonin
>            Priority: Major
>              Labels: IEP-104, IEP-59, important, ise
>
> Right now, timeout between CDC event happens and consumer notified about it is relatively big.
> It sums from the following:
>  * {{{}DataStorageConfiguration#walForceArchiveTimeout{}}}.
>  * segment archivation time.
>  * {{CdcConfiguration#checkFrequence}}
>  * time to consume previous events.
> Advantages of this way described in [IEP-59|https://cwiki.apache.org/confluence/display/IGNITE/IEP-59+Change+Data+Capture]
> So if user wants to minimize this timeouts we can implement the following:
>  * Collects cdc event inside fixed-size buffer in node.
>  * Consume events from buffer by CDCConsumer inside node process.
>  * Store WALPointer of last event added to buffer.
>  * On buffer overflow stop push events to it.
>  ** Fallback to WAL segments iteration
>  ** When iteration catch up WAL tail switch back to the buffer.
>  * On node failover recover consumer with the cdc state as in ignite-cdc.
> With this approach most of the time CDCConsumer will consume events in near realtime fashion.
> Description of the feature:
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-104+CDC%3A+Realtime+mode



--
This message was sent by Atlassian Jira
(v8.20.10#820010)