You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "yjsdev (via GitHub)" <gi...@apache.org> on 2026/06/04 03:41:38 UTC

[I] [Bug] [MongoDB-CDC] Bug DataException: operationType is not a valid field name [seatunnel]

yjsdev opened a new issue, #11008:
URL: https://github.com/apache/seatunnel/issues/11008

   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   2.3.12版本、2.3.13版本均出现Mongo-CDC同步时,Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.kafka.connect.errors.DataException: operationType is not a valid field name
   
   例如test-db to test-db2,表t_sys_user,
   数据示例:
   db.t_sys_user.insertOne({
     "_id":"12",
     phone: "47873454385",
     name: "直接写入测试8"
   })
   env {
     job.mode = "STREAMING"
     checkpoint.interval = 10000
     parallelism = 1
   }
   
   
   ### SeaTunnel Version
   
   2.3.12版本、2.3.13版本
   
   ### SeaTunnel Config
   
   ```conf
   conf示例:
   source {
     MongoDB-CDC {
       # ✅ 建议:在hosts和connection.options中都进行配置,以确保正确连接到副本集
       hosts = "127.0.0.1:27017"
       # 在URI中指定副本集名称和读偏好,强制从Primary读取,确保能获取完整的Change Stream数据
       connection.options = "replicaSet=rs0&readPreference=primary"
   
       database = ["test-db"]
       collection = ["test-db.t_sys_user"]
   
       # ✅ 关键优化:设置一个大于0的心跳间隔,防止resume token过期
       # 官方建议当集合变更慢时强烈建议设置此参数 [citation:3]
       heartbeat.interval.ms = 10000
       schema = {
         table = "test-db.t_sys_user"
         fields {
           "_id" : string
           "phone" : string,
           "name" : string
         }
       }
     }
   }
   
   sink {
     MongoDB {
       uri = "mongodb://127.0.0.1:27017"
       database = "test-db2"
       collection = "t_sys_user"
       # 🔑 关键修改:使用 distributed_id 作为主键
       primary-key = ["_id"]
       # 兼容旧版本的写法
       #upsert-key = ["_id"]
       upsert-enable = true
       buffer-flush.max-rows = 1000
       buffer-flush.interval = 10000
     }
   }
   ```
   
   ### Running Command
   
   ```shell
   String configurePath = args.length > 0 ? args[0] : "/examples/mongoA_to_mongoB.conf";
   ```
   
   ### Error Exception
   
   ```log
   ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
   	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228)
   	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
   	at org.apache.seatunnel.example.engine.SeaTunnelEngineLocalExample.main(SeaTunnelEngineLocalExample.java:49)
   Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.kafka.connect.errors.DataException: operationType is not a valid field name
   	at org.apache.kafka.connect.data.Struct.lookupField(Struct.java:254)
   	at org.apache.kafka.connect.data.Struct.getCheckType(Struct.java:261)
   	at org.apache.kafka.connect.data.Struct.getString(Struct.java:158)
   	at org.apache.seatunnel.connectors.seatunnel.cdc.mongodb.sender.MongoDBConnectorDeserializationSchema.operationTypeFor(MongoDBConnectorDeserializationSchema.java:183)
   	at org.apache.seatunnel.connectors.seatunnel.cdc.mongodb.sender.MongoDBConnectorDeserializationSchema.deserialize(MongoDBConnectorDeserializationSchema.java:107)
   	at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceRecordEmitter.emitElement(IncrementalSourceRecordEmitter.java:201)
   	at org.apache.seatunnel.connectors.seatunnel.cdc.mongodb.source.MongoDBRecordEmitter.processElement(MongoDBRecordEmitter.java:79)
   	at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceRecordEmitter.emitRecord(IncrementalSourceRecordEmitter.java:102)
   	at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceRecordEmitter.emitRecord(IncrementalSourceRecordEmitter.java:62)
   	at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:110)
   	at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:119)
   	at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:114)
   	at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:119)
   	at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:159)
   	at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:127)
   	at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:165)
   	at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:132)
   	at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:679)
   	at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1008)
   	at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43)
   	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
   	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
   	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
   	at java.base/java.lang.Thread.run(Thread.java:842)
   
   	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:220)
   	... 2 more
   ```
   
   ### Zeta or Flink or Spark Version
   
   _No response_
   
   ### Java or Scala Version
   
   _No response_
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [MongoDB-CDC] Bug DataException: operationType is not a valid field name [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11008:
URL: https://github.com/apache/seatunnel/issues/11008#issuecomment-4627269736

   I checked the current MongoDB CDC path, and this does look like a real connector bug direction rather than just a sink-side configuration problem.
   
   The key point is that the MongoDB CDC deserialization path currently reads `operationType` directly from the change-event `Struct` without first checking whether that field is actually present. In the current code this happens in both the MongoDB CDC deserialization path and the output-buffer rewrite path. If a record shape reaches SeaTunnel without `operationType`, you get exactly the `DataException` shown in this issue.
   
   So the thread is worth keeping open as a bug.
   
   To narrow down which MongoDB record shape is missing that field, we still need three details:
   1. the MongoDB server version and deployment mode (standalone / replica set / sharded)
   2. the full stack trace from the first failing record, including the lines immediately before the `DataException`
   3. whether this happens during the initial snapshot stage, or only after the first change-stream event arrives
   
   Also, because Apache SeaTunnel is an international community, please keep the follow-up details in English if possible so more contributors can help review the case.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org