[jira] [Updated] (IGNITE-17700) Online CDC

["Nikolay Izhikov (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IGNITE-17700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nikolay Izhikov updated IGNITE-17700:
-------------------------------------
    Labels: IEP-59 ise  (was: IEP-59)

> Online CDC
> ----------
>
>                 Key: IGNITE-17700
>                 URL: https://issues.apache.org/jira/browse/IGNITE-17700
>             Project: Ignite
>          Issue Type: Improvement
>            Reporter: Nikolay Izhikov
>            Priority: Major
>              Labels: IEP-59, ise
>
> Right now, timeout between CDC event happens and consumer notified about it is relatively big.
> It sums from the following: 
>   * {{DataStorageConfiguration#walForceArchiveTimeout}}.
>   * segment archivation time.
>   * {{CdcConfiguration#checkFrequence}}
>   * time to consume previous events.
> Advantages of this way described in [IEP-59|https://cwiki.apache.org/confluence/display/IGNITE/IEP-59+Change+Data+Capture]
> So if user wants to minimize this timeouts we can implement the following:
>   * Collects cdc event inside fixed-size buffer in node.
>   * Consume events from buffer by CDCConsumer inside node process.
>   * Store WALPointer of last event added to buffer.
>   * On buffer overflow stop push events to it.
>   ** Fallback to WAL segments iteration
>   ** When iteration catch up WAL tail switch back to the buffer. 
>   * On node failover recover consumer with the cdc state as in ignite-cdc.
> With this approach most of the time CDCConsumer will consume events in near realtime fashion.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)