You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Bobby Richard <bo...@broadcom.com> on 2022/10/07 14:20:32 UTC

Debezium format with MongoDB

Is it possible to use the Flink Debezium format with the changelogs
generated by the Debezium MongoDB Connector? I have tried multiple
configurations (with and without json schema included), and I always
receive a java.io.IOException: Corrupt Debezium JSON message. Could it be
related to the fact that the MongoDB connector sends the record as a JSON
string instead of regular JSON?

Example mongodb debezium changelog (notice the json string for
payload.after):
{ "schema": { ... }, "payload": { "after": "{\"_id\" : {\"$numberLong\" :
\"1004\"},\"first_name\" : \"Anne\",\"last_name\" : \"Kretchmar\",\"email\"
: \"annek@noanswer.org\"}", "patch": null, "source": { "version":
"1.9.6.Final", "connector": "mongodb", "name": "fulfillment", "ts_ms":
1558965508000, "snapshot": false, "db": "inventory", "rs": "rs0",
"collection": "customers", "ord": 31, "h": 1546547425148721999 }, "op": "c",
"ts_ms": 1558965515240 } }

Example Postgres Debezium changelog:
{
"schema": { ... },
"payload": { "before": null, "after": { "id": 1, "first_name": "Anne",
"last_name": "Kretchmar", "email": "annek@noanswer.org" }, "source": {
"version": "1.9.6.Final", "connector": "postgresql", "name":
"PostgreSQL_server", "ts_ms": 1559033904863, "snapshot": true, "db":
"postgres", "sequence": "[\"24023119\",\"24023128\"]" "schema": "public",
"table": "customers", "txId": 555, "lsn": 24023128, "xmin": null }, "op":
"c", "ts_ms": 1559033904863 } }

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.