[jira] [Comment Edited] (IGNITE-18209) Reduce binary metadata synchronization time for CDC through Kafka

["Ilya Shishkov (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IGNITE-18209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17637760#comment-17637760 ] 

Ilya Shishkov edited comment on IGNITE-18209 at 12/5/22 12:56 PM:
------------------------------------------------------------------

Tests of [PR #187|https://github.com/apache/ignite-extensions/pull/187] passed without timeout:
# [TC1|https://ci.ignite.apache.org/buildConfiguration/IgniteExtensions_Tests_Cdc/6924088?buildTab=overview&hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandBuildChangesSection=true]
# [TC2|https://ci2.ignite.apache.org/buildConfiguration/IgniteExtensions_Tests_Cdc/6922844?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandBuildChangesSection=true]


was (Author: shishkovilja):
Tests of PoC PR passed without timeout:
# [TC1|https://ci.ignite.apache.org/buildConfiguration/IgniteExtensions_Tests_Cdc/6924088?buildTab=overview&hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandBuildChangesSection=true]
# [TC2|https://ci2.ignite.apache.org/buildConfiguration/IgniteExtensions_Tests_Cdc/6922844?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandBuildChangesSection=true]

> Reduce binary metadata synchronization time for CDC through Kafka
> -----------------------------------------------------------------
>
>                 Key: IGNITE-18209
>                 URL: https://issues.apache.org/jira/browse/IGNITE-18209
>             Project: Ignite
>          Issue Type: Improvement
>          Components: extensions
>            Reporter: Ilya Shishkov
>            Assignee: Ilya Shishkov
>            Priority: Major
>              Labels: IEP-59, ise
>
> Currently, there is a bottleneck in synchronized method {{KafkaToIgniteMetadataUpdater#updateMetadata}}:
> # {{KafkaToIgniteCdcStreamer}} contains multiple {{KafkaToIgniteCdcStreamerApplier}} which shares _single_ {{KafkaToIgniteMetadataUpdater}}.
> # All appliers handle corrsponding partitions consequently.
> # {{META_UPDATE_MARKER}} is sent twice to each Kafka partition of event topic: firstly, in case of type mappings updates, secondly, in case of binary types update.
> # When first {{KafkaToIgniteCdcStreamerApplier}} meets {{META_UPDATE_MARKER}} it calls {{KafkaToIgniteMetadataUpdater#updateMetadata}} which in turn calls {{KafkaConsumer#poll}}, which returns immediately [1] when data is present in metadata topic. If there are few binary types and mappings to update, some {{KafkaToIgniteCdcStreamerApplier}} thread will consume all entries from metadata topic.
> # All other threads of all {{KafkaToIgniteCdcStreamerApplier}} will call {{KafkaConsumer#poll}} from empty metadata topic, which will remain blocked until new data becomes available or request timeout occurs [1].
> # Because of {{synchronized}} access to {{KafkaToIgniteMetadataUpdater#updateMetadata}} all threads of all {{KafkaToIgniteCdcStreamerApplier}} will form a sequence of calls. Each call will block remaining appliers threads for {{kafkaReqTimeout}} period (if metadata topic remains empty).
> # The last call, i.e. last Kafka partition polling in this chain will happen at least after {{(partitionsCount x 2 - 1) x kafkaReqTimeout}} period. For example for default timeout and 16 Kafka partitions _last partition will be consumed after 1.5 minutes_. 
> # Amount of threads in {{KafkaToIgniteCdcStreamer}} does not make sence.
> # Data updates are blocked for Kafka partitions with unhandled update markers.
> As I understand possible solutions are:
> # Hold information about replicated types or get it from {{BinaryContext}}. Information about type can be sent with {{META_UPDATE_MARKER}}: see [PR#187|https://github.com/apache/ignite-extensions/pull/187]
> # Completely remove metadata topic, and send metadata merged with marker directly into event topic: see [PR#196|https://github.com/apache/ignite-extensions/pull/196]
> # Any other ways to sync appliers?
> As a PoC of approach with {{BinaryContext}} I have prepared PR [2].
> Links:
> # https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#poll-java.time.Duration-
> # https://github.com/apache/ignite-extensions/pull/187
> # https://github.com/apache/ignite-extensions/pull/196



--
This message was sent by Atlassian Jira
(v8.20.10#820010)