You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "DDDxiyesi (via GitHub)" <gi...@apache.org> on 2025/07/23 09:10:01 UTC
[I] [Bug] Failed to deserialize consumer record [paimon]
DDDxiyesi opened a new issue, #5945:
URL: https://github.com/apache/paimon/issues/5945
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
paimon-flink-1.20-1.2.0
paimon-flink-action-1.2.0
### Compute Engine
flink 1.20.1
flink-sql-connector-kafka-3.4.0-1.20
### Minimal reproduce step
```
kafka_sync_database \
--warehouse hdfs://nameservice1/paimon/databases \
--database ods_db \
--table_prefix ods_computer_ \
--kafka_conf properties.bootstrap.servers=xxxxxx:9092 \
--kafka_conf topic=TOPIC \
--kafka_conf properties.group.id=gtoup_2 \
--kafka_conf scan.startup.mode=earliest-offset \
--kafka_conf value.format=canal-json \
--kafka_conf key.format=raw \
--catalog_conf metastore=hive \
--catalog_conf uri=thrift://xxxxxx:9083 \
--catalog_conf hive-conf-dir=/opt/datasophon/hive/conf/ \
--table_conf bucket=8 \
--table_conf changelog-producer=input \
--table_conf metastore.partitioned-table=true \
--table_conf consumer.expiration-time=30d \
--multiple_table_partition_keys t=id,id_1,id_2
```
### What doesn't meet your expectations?
java.io.IOException: Failed to deserialize consumer record due to
at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter.emitRecord(KafkaRecordEmitter.java:56) ~[flink-sql-connector-kafka-3.4.0-1.20.jar:3.4.0-1.20]
at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter.emitRecord(KafkaRecordEmitter.java:33) ~[flink-sql-connector-kafka-3.4.0-1.20.jar:3.4.0-1.20]
at org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:203) ~[flink-connector-files-1.20.1.jar:1.20.1]
at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:443) ~[flink-streaming-java-1.20.1.jar:1.20.1]
at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68) ~[flink-streaming-java-1.20.1.jar:1.20.1]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-streaming-java-1.20.1.jar:1.20.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:638) ~[flink-streaming-java-1.20.1.jar:1.20.1]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) ~[flink-streaming-java-1.20.1.jar:1.20.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:973) ~[flink-streaming-java-1.20.1.jar:1.20.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:917) ~[flink-streaming-java-1.20.1.jar:1.20.1]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970) ~[flink-dist-1.20.1.jar:1.20.1]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:949) [flink-dist-1.20.1.jar:1.20.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763) [flink-dist-1.20.1.jar:1.20.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) [flink-dist-1.20.1.jar:1.20.1]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
Caused by: java.io.IOException: Failed to deserialize consumer record ConsumerRecord(topic = XXXXX, partition = 6, leaderEpoch = 0, offset = 0, CreateTime = 1753250107345, serialized key size = 43, serialized value size = 2909, headers = RecordHeaders(headers = [], isReadOnly = false), key = [B@63786e81, value = [B@3267a0bf).
at org.apache.flink.connector.kafka.source.reader.deserializer.KafkaDeserializationSchemaWrapper.deserialize(KafkaDeserializationSchemaWrapper.java:59) ~[flink-sql-connector-kafka-3.4.0-1.20.jar:3.4.0-1.20]
at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter.emitRecord(KafkaRecordEmitter.java:53) ~[flink-sql-connector-kafka-3.4.0-1.20.jar:3.4.0-1.20]
... 14 more
Caused by: org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.JsonParseException: Unexpected character ('|' (code 124)): Expected space separating root-level values
at [Source: (byte[])"2025|41001|2025166"; line: 1, column: 31]
at org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2418) ~[paimon-flink-1.20-1.2.0.jar:1.2.0]
at org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:749) ~[paimon-flink-1.20-1.2.0.jar:1.2.0]
at org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:673) ~[paimon-flink-1.20-1.2.0.jar:1.2.0]
at org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportMissingRootWS(ParserMinimalBase.java:721) ~[paimon-flink-1.20-1.2.0.jar:1.2.0]
at org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._verifyRootSpace(UTF8StreamJsonParser.java:1781) ~[paimon-flink-1.20-1.2.0.jar:1.2.0]
at org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseUnsignedNumber(UTF8StreamJsonParser.java:1510) ~[paimon-flink-1.20-1.2.0.jar:1.2.0]
at org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:905) ~[paimon-flink-1.20-1.2.0.jar:1.2.0]
at org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:793) ~[paimon-flink-1.20-1.2.0.jar:1.2.0]
at org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4817) ~[paimon-flink-1.20-1.2.0.jar:1.2.0]
at org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4723) ~[paimon-flink-1.20-1.2.0.jar:1.2.0]
at org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3738) ~[paimon-flink-1.20-1.2.0.jar:1.2.0]
at org.apache.paimon.flink.action.cdc.kafka.KafkaDebeziumJsonDeserializationSchema.deserialize(KafkaDebeziumJsonDeserializationSchema.java:70) ~[paimon-flink-1.20-1.2.0.jar:1.2.0]
at org.apache.paimon.flink.action.cdc.kafka.KafkaDebeziumJsonDeserializationSchema.deserialize(KafkaDebeziumJsonDeserializationSchema.java:39) ~[paimon-flink-1.20-1.2.0.jar:1.2.0]
at org.apache.flink.streaming.connectors.kafka.KafkaDeserializationSchema.deserialize(KafkaDeserializationSchema.java:81) ~[flink-sql-connector-kafka-3.4.0-1.20.jar:3.4.0-1.20]
at org.apache.flink.connector.kafka.source.reader.deserializer.KafkaDeserializationSchemaWrapper.deserialize(KafkaDeserializationSchemaWrapper.java:56) ~[flink-sql-connector-kafka-3.4.0-1.20.jar:3.4.0-1.20]
at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter.emitRecord(KafkaRecordEmitter.java:53) ~[flink-sql-connector-kafka-3.4.0-1.20.jar:3.4.0-1.20]
... 14 more
### Anything else?
Paimon CDC 0.8 works well. When upgrading to Paimon 1.2.0, "deserialize" error appears.
Seems like the key of a message must be in valid JSON format when KafkaDebeziumJsonDeserializationSchema deserializing the key.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Failed to deserialize consumer record [paimon]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on issue #5945:
URL: https://github.com/apache/paimon/issues/5945#issuecomment-3209886288
I think we can for now ignore non-JSON behaviour which will maintain the existing behaviour, but prevent de-serialization errors. This should fix the error, I'll raise a patch, maybe long term we can handle the non-json values separately
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Failed to deserialize consumer record [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #5945: [Bug] Failed to deserialize consumer record
URL: https://github.com/apache/paimon/issues/5945
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org