[jira] [Resolved] (KAFKA-5330) Use per-task converters in Connect

["Ewen Cheslack-Postava (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ewen Cheslack-Postava resolved KAFKA-5330.
------------------------------------------
       Resolution: Fixed
    Fix Version/s: 1.0.0

Issue resolved by pull request 3196
[https://github.com/apache/kafka/pull/3196]

> Use per-task converters in Connect
> ----------------------------------
>
>                 Key: KAFKA-5330
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5330
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>    Affects Versions: 0.11.0.0
>            Reporter: Ewen Cheslack-Postava
>             Fix For: 1.0.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Because Connect started with a worker-wide model of data formats, we currently allocate a single Converter per worker and only allocate an independent one when the user overrides the converter.
> This can lead to performance problems when the worker-level default converter is used by a large number of tasks because converters need to be threadsafe to support this model and they may spend a lot of time just on synchronization.
> We could, instead, simply allocate one converter per task. There is some overhead involved, but generally it shouldn't be that large. For example, Confluent's Avro converters will each have their own schema cache and have to make their on calls to the schema registry API, but these are relatively small, likely inconsequential compared to any normal overhead we would already have for creating and managing each task. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)