You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "Qingsheng Ren (Jira)" <ji...@apache.org> on 2021/12/01 09:58:00 UTC

[jira] [Created] (FLINK-25132) KafkaSource cannot work with object-reusing DeserializationSchema

Qingsheng Ren created FLINK-25132:
-------------------------------------

             Summary: KafkaSource cannot work with object-reusing DeserializationSchema
                 Key: FLINK-25132
                 URL: https://issues.apache.org/jira/browse/FLINK-25132
             Project: Flink
          Issue Type: Bug
          Components: Connectors / Kafka
    Affects Versions: 1.14.0
            Reporter: Qingsheng Ren
             Fix For: 1.14.1


Currently Kafka source deserializes ConsumerRecords in split reader and puts them into the elementQueue, then task's main thread polls these records from the queue asynchronously. This mechanism cannot cooperate with DeserializationSchemas with object reuse: all records staying in the element queue points to the same object.

A solution would be moving deserialization to RecordEmitter, which works in the task's main thread. 

Notes that this issue actually effects all sources which do deserialization in split reader. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)