You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "letong520 (via GitHub)" <gi...@apache.org> on 2026/01/26 07:27:35 UTC
[I] [Bug] [seatunnel-engin] MongoDB CDC 同步数据时报错 [seatunnel]
letong520 opened a new issue, #10398:
URL: https://github.com/apache/seatunnel/issues/10398
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
同步mongodb数据到doris中,在同步全量数时,大概三百万同步了100万左右时,有个报错导致数据一直失败,检查mongo的数据是无任何问题的,从日志是拆分时发生java.lang.NullPointerException,要如何解决
### SeaTunnel Version
2.3.12
### SeaTunnel Config
```conf
env{
parallelism = 1
job.name = "mongodb-cdc"
job.mode = "STREAMING"
checkpoint.interval = 60000
}
source {
MongoDB-CDC {
hosts = "${MONGODB_HOSTS}"
database = ["ivcp_m_user"]
collection = ["ivcp_m_user.user_user_group_rela_info"]
username = "${MONGODB_USER}"
password = "${MONGODB_PWD}"
startup.mode = "initial"
schema = {
table = "ivcp_m_user.user_user_group_rela_info"
primaryKey {
name = "pk_id"
columnNames = [_id]
}
columns = [
{
name = _id
type = string
},{
name = userId
type = bigint
},{
name = groupId
type = bigint
},{
name = matchType
type = "array<string>"
},{
name = groupMatchRule
type = string
},{
name = haveRunTimeCondition
type = string
},{
name = nonRuntimeConditionMatchResult
type = string
},{
name = opId
type = bigint
},{
name = createdTime
type = timestamp
},{
name = updatedBy
type = bigint
},{
name = updatedTime
type = timestamp
}
]
}
}
}
sink {
Doris {
fenodes = "${DORIS_HOST}:${DORIS_PORT}"
query-port = 9030
username = ${DORIS_USER}
password = "${DORIS_PWD}"
schema_save_mode = "ERROR_WHEN_SCHEMA_NOT_EXIST"
data_save_mode = "APPEND_DATA"
sink.enable-2pc = "true"
sink.enable-delete = "true"
sink.label-prefix = "doris_mongodb_cdc"
database = "ivcp_bo_report"
table = "ods_${table_name}"
doris.config {
format = "json"
read_json_by_line = "true"
}
}
}
```
### Running Command
```shell
/opt/seatunnel/bin/seatunnel.sh \
-e local \
--config /opt/seatunnel/jobs/mongodb-cdc.yaml \
--variable "DORIS_HOST=${DORIS_HOST}" \
--variable "DORIS_PORT=${DORIS_PORT}" \
--variable "DORIS_USER=${DORIS_USER}" \
--variable "DORIS_PWD=${DORIS_PWD}" \
--variable "MONGODB_HOSTS=${MONGODB_HOSTS}" \
--variable "MONGODB_USER=${MONGODB_USER}" \
--variable "MONGODB_PWD=${MONGODB_PWD}"
```
### Error Exception
```log
2026-01-26 15:17:09,647 ERROR [o.a.s.e.s.d.p.PhysicalVertex ] [hz.main.seaTunnel.task.thread-7] - Job (1068062844672540673), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MongoDB-CDC]-SourceTask (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=2}] end with state FAILED and Exception: java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106)
at java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.finishCurrentFetch(SourceReaderBase.java:210)
at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.moveToNextSplit(SourceReaderBase.java:188)
at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169)
at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:93)
at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:119)
at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:159)
2026-01-26 15:17:09,647 INFO [o.a.s.e.s.d.p.SubPlan ] [seatunnel-coordinator-service-9] - Job (1068062844672540673), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MongoDB-CDC]-SourceTask (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=2}] future complete with state FAILED
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:127)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:165)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:132)
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:679)
at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1008)
at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
2026-01-26 15:17:09,647 ERROR [o.a.s.e.s.d.p.SubPlan ] [seatunnel-coordinator-service-9] - Task TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=2} Failed in Job mongodb-cdc (1068062844672540673), Pipeline: [(1/1)], Begin to cancel other tasks in this pipeline.
2026-01-26 15:17:09,647 INFO [o.a.s.e.s.m.JobMaster ] [hz.main.seaTunnel.task.thread-7] - release the task group resource TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=2}
2026-01-26 15:17:09,647 INFO [a.s.e.s.s.s.DefaultSlotService] [hz.main.generic-operation.thread-9] - received slot release request, jobID: 1068062844672540673, slot: SlotProfile{worker=[localhost]:5801, slotID=8, ownerJobID=1068062844672540673, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='e7983909-b91d-4837-9f17-84ea965c7e73'}
2026-01-26 15:17:09,647 INFO [o.a.s.e.s.d.p.SubPlan ] [seatunnel-coordinator-service-9] - Job mongodb-cdc (1068062844672540673), Pipeline: [(1/1)] turned from state RUNNING to FAILING.
2026-01-26 15:17:09,647 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [seatunnel-coordinator-service-9] - Job (1068062844672540673), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MongoDB-CDC]-SplitEnumerator (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=1}] state process is start
2026-01-26 15:17:09,648 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [seatunnel-coordinator-service-9] - Job (1068062844672540673), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MongoDB-CDC]-SplitEnumerator (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=1}] turned from state RUNNING to CANCELING.
2026-01-26 15:17:09,648 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [seatunnel-coordinator-service-9] - Send cancel Job (1068062844672540673), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MongoDB-CDC]-SplitEnumerator (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=1}] operator to member [localhost]:5801
2026-01-26 15:17:09,648 INFO [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-9] - [localhost]:5801 [seatunnel-248379] [5.1] Task (TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=1}) need cancel.
2026-01-26 15:17:09,648 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [seatunnel-coordinator-service-9] - Job (1068062844672540673), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MongoDB-CDC]-SourceTask (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=2}] state process is start
2026-01-26 15:17:09,648 WARN [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=1}] - [localhost]:5801 [seatunnel-248379] [5.1] Interrupted task 1000100000000 - org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask@920df4
2026-01-26 15:17:09,648 INFO [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=1}] - [localhost]:5801 [seatunnel-248379] [5.1] taskDone, taskId = 1000100000000, taskGroup = TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=1}
2026-01-26 15:17:09,650 INFO [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=1}] - [localhost]:5801 [seatunnel-248379] [5.1] taskGroup TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=1} complete with CANCELED
2026-01-26 15:17:09,650 INFO [.e.IncrementalSourceEnumerator] [BlockingWorker-TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=1}] - Closing enumerator...
2026-01-26 15:17:09,650 INFO [o.a.s.e.s.TaskExecutionService] [hz.main.seaTunnel.task.thread-7] - [localhost]:5801 [seatunnel-248379] [5.1] Task TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=1} complete with state CANCELED
2026-01-26 15:17:09,650 INFO [o.a.s.a.e.LoggingEventHandler ] [hz.main.generic-operation.thread-8] - log event: EnumeratorCloseEvent(createdTime=1769411829650, jobId=1068062844672540673, eventType=LIFECYCLE_ENUMERATOR_CLOSE)
2026-01-26 15:17:09,650 INFO [o.a.s.e.s.CoordinatorService ] [hz.main.seaTunnel.task.thread-7] - [localhost]:5801 [seatunnel-248379] [5.1] Received task end from execution TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=1}, state CANCELED
2026-01-26 15:17:09,651 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [hz.main.seaTunnel.task.thread-7] - Job (1068062844672540673), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MongoDB-CDC]-SplitEnumerator (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=1}] turned from state CANCELING to CANCELED.
2026-01-26 15:17:09,651 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [hz.main.seaTunnel.task.thread-7] - Job (1068062844672540673), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MongoDB-CDC]-SplitEnumerator (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=1}] state process is stopped
2026-01-26 15:17:09,651 INFO [o.a.s.e.s.d.p.SubPlan ] [seatunnel-coordinator-service-9] - Job (1068062844672540673), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MongoDB-CDC]-SplitEnumerator (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=1}] future complete with state CANCELED
2026-01-26 15:17:09,651 INFO [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-9] - start clean pending checkpoint cause Pipeline turn to end state.
2026-01-26 15:17:09,651 INFO [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-9] - Turn checkpoint_state_1068062844672540673_1 state from RUNNING to CANCELED
2026-01-26 15:17:09,651 INFO [o.a.s.e.s.d.p.SubPlan ] [seatunnel-coordinator-service-9] - Job mongodb-cdc (1068062844672540673), Pipeline: [(1/1)] will end with state FAILED
2026-01-26 15:17:09,652 INFO [o.a.s.e.s.d.p.SubPlan ] [seatunnel-coordinator-service-9] - Job mongodb-cdc (1068062844672540673), Pipeline: [(1/1)] turned from state FAILING to FAILED.
2026-01-26 15:17:09,670 INFO [o.a.s.e.s.m.JobMaster ] [seatunnel-coordinator-service-9] - release the pipeline Job mongodb-cdc (1068062844672540673), Pipeline: [(1/1)] resource
2026-01-26 15:17:09,670 INFO [a.s.e.s.s.s.DefaultSlotService] [hz.main.generic-operation.thread-14] - received slot release request, jobID: 1068062844672540673, slot: SlotProfile{worker=[localhost]:5801, slotID=7, ownerJobID=1068062844672540673, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='e7983909-b91d-4837-9f17-84ea965c7e73'}
2026-01-26 15:17:09,670 INFO [o.a.s.e.s.d.p.SubPlan ] [seatunnel-coordinator-service-9] - Job mongodb-cdc (1068062844672540673), Pipeline: [(1/1)] state process is stop
2026-01-26 15:17:09,671 INFO [o.a.s.e.s.d.p.PhysicalPlan ] [seatunnel-coordinator-service-8] - Job mongodb-cdc (1068062844672540673), Pipeline: [(1/1)] future complete with state FAILED
2026-01-26 15:17:09,671 INFO [o.a.s.e.s.d.p.PhysicalPlan ] [seatunnel-coordinator-service-8] - cancel job Job mongodb-cdc (1068062844672540673) because makeJobEndWhenPipelineEnded is true
2026-01-26 15:17:09,671 INFO [o.a.s.e.s.d.p.PhysicalPlan ] [seatunnel-coordinator-service-8] - Job mongodb-cdc (1068062844672540673) turned from state RUNNING to FAILING.
2026-01-26 15:17:09,672 INFO [o.a.s.e.s.d.p.PhysicalPlan ] [seatunnel-coordinator-service-8] - Job mongodb-cdc (1068062844672540673) turned from state FAILING to FAILED.
2026-01-26 15:17:09,673 INFO [o.a.s.e.s.d.p.PhysicalPlan ] [seatunnel-coordinator-service-8] - Job mongodb-cdc (1068062844672540673) state process is stop
2026-01-26 15:17:09,688 INFO [o.a.s.a.e.LoggingEventHandler ] [seatunnel-coordinator-service-8] - log event: JobStateEvent(jobId=1068062844672540673, jobName=mongodb-cdc, jobStatus=FAILED, createdTime=1769411829688)
2026-01-26 15:17:09,690 INFO [o.a.s.e.c.j.ClientJobProxy ] [main] - Job (1068062844672540673) end with state FAILED
2026-01-26 15:17:09,691 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel-248379] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2026-01-26 15:17:09,708 INFO [c.h.i.s.t.TcpServerConnection ] [hz.main.IO.thread-in-1] - [localhost]:5801 [seatunnel-248379] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:35197, qualifier=null, endpoint=[127.0.0.1]:35197, remoteUuid=c83f1160-f24c-4817-bc0a-b699418400fe, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2026-01-26 15:17:09,709 INFO [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel-248379] [5.1] Removed connection to endpoint: [localhost]:5801:8a2c2376-deb3-41e1-97e7-da268789e37a, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:35197->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2026-01-26 15:17:09.688, lastWriteTime=2026-01-26 15:17:05.621, closedTime=2026-01-26 15:17:09.705, connected server version=5.1}
2026-01-26 15:17:09,709 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel-248379] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2026-01-26 15:17:09,712 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel-248379] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2026-01-26 15:17:09,712 INFO [s.c.s.s.c.ClientExecuteCommand] [main] - Closed SeaTunnel client......
2026-01-26 15:17:09,712 INFO [c.h.c.i.ClientEndpointManager ] [hz.main.event-3] - [localhost]:5801 [seatunnel-248379] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:35197, qualifier=null, endpoint=[127.0.0.1]:35197, remoteUuid=c83f1160-f24c-4817-bc0a-b699418400fe, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=c83f1160-f24c-4817-bc0a-b699418400fe, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1769410825408, latest clientAttributes=lastStatisticsCollectionTime=1769411825618,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1769410825316,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=14431338496,os.freePhysicalMemorySize=567451648,os.freeSwapSpaceSize=0,os.maxFileDescriptorCount=1048576,os.openFileDescriptorCount=95,os.processCpuTime=319300000000,os.systemLoadAverage=1.54,os.totalPhysicalMemorySize=1073741824,os.tota
lSwapSpaceSize=0,runtime.availableProcessors=1,runtime.freeMemory=141666840,runtime.maxMemory=518979584,runtime.totalMemory=260308992,runtime.uptime=1011106,runtime.usedMemory=118644472, labels=[]}
2026-01-26 15:17:09,712 INFO [c.h.c.LifecycleService ] [main] - [localhost]:5801 [seatunnel-248379] [5.1] [localhost]:5801 is SHUTTING_DOWN
2026-01-26 15:17:09,719 INFO [c.h.i.p.i.MigrationManager ] [hz.main.cached.thread-8] - [localhost]:5801 [seatunnel-248379] [5.1] Shutdown request of Member [localhost]:5801 - 8a2c2376-deb3-41e1-97e7-da268789e37a [master node] [active master] this is handled
2026-01-26 15:17:09,725 INFO [c.h.i.i.Node ] [main] - [localhost]:5801 [seatunnel-248379] [5.1] Shutting down connection manager...
2026-01-26 15:17:09,729 INFO [c.h.i.i.Node ] [main] - [localhost]:5801 [seatunnel-248379] [5.1] Shutting down node engine...
2026-01-26 15:17:09,812 INFO [o.a.s.e.s.CoordinatorService ] [pool-5-thread-1] - [localhost]:5801 [seatunnel-248379] [5.1] This node become leave active master node, begin clear coordinator service
2026-01-26 15:17:09,812 INFO [.s.s.o.e.j.s.AbstractConnector] [main] - Stopped ServerConnector@1290ed28{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2026-01-26 15:17:09,812 INFO [o.a.s.s.o.e.j.s.session ] [main] - node0 Stopped scavenging
2026-01-26 15:17:09,814 INFO [a.s.s.o.e.j.s.h.ContextHandler] [main] - Stopped o.a.s.s.o.e.j.s.ServletContextHandler@35bfa1bb{/,null,STOPPED}
2026-01-26 15:17:09,816 INFO [.c.c.DefaultClassLoaderService] [main] - close classloader service
2026-01-26 15:17:09,817 INFO [o.a.s.e.s.EventService ] [event-forwarder-0] - Event forward thread interrupted
2026-01-26 15:17:09,824 INFO [c.h.i.i.NodeExtension ] [main] - [localhost]:5801 [seatunnel-248379] [5.1] Destroying node NodeExtension.
2026-01-26 15:17:09,824 INFO [c.h.i.i.Node ] [main] - [localhost]:5801 [seatunnel-248379] [5.1] Hazelcast Shutdown is completed in 106 ms.
2026-01-26 15:17:09,824 INFO [c.h.c.LifecycleService ] [main] - [localhost]:5801 [seatunnel-248379] [5.1] [localhost]:5801 is SHUTDOWN
2026-01-26 15:17:09,824 INFO [s.c.s.s.c.ClientExecuteCommand] [main] - Closed HazelcastInstance ......
2026-01-26 15:17:09,824 INFO [s.c.s.s.c.ClientExecuteCommand] [main] - Closed metrics executor service ......
2026-01-26 15:17:09,825 ERROR [o.a.s.c.s.SeaTunnel ] [main] -
===============================================================================
2026-01-26 15:17:09,825 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Fatal Error,
2026-01-26 15:17:09,825 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues
2026-01-26 15:17:09,825 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Reason:SeaTunnel job executed failed
2026-01-26 15:17:09,827 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:40)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106)
at java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.finishCurrentFetch(SourceReaderBase.java:210)
at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.moveToNextSplit(SourceReaderBase.java:188)
at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169)
at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:93)
at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:119)
at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:159)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:127)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:165)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:132)
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:679)
at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1008)
at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:220)
... 2 more
2026-01-26 15:17:09,827 ERROR [o.a.s.c.s.SeaTunnel ] [main] -
===============================================================================
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:40)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106)
at java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.finishCurrentFetch(SourceReaderBase.java:210)
at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.moveToNextSplit(SourceReaderBase.java:188)
at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169)
at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:93)
at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:119)
at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:159)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:127)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:165)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:132)
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:679)
at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1008)
at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:220)
... 2 more
2026-01-26 15:17:19,642 INFO [r.IncrementalSourceSplitReader] [Source Data Fetcher for BlockingWorker-TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=2}] - Close current fetcher org.apache.seatunnel.connectors.cdc.base.source.reader.external.IncrementalSourceScanFetcher
2026-01-26 15:17:23,315 INFO [o.a.s.c.s.c.s.r.f.SplitFetcher] [Source Data Fetcher for BlockingWorker-TaskGroupLocation{jobId=1068062844672540673, pipelineId=1, taskGroupId=2}] - Split fetcher 0 exited.
2026-01-26 15:17:23,317 INFO [s.c.s.s.c.ClientExecuteCommand] [SeaTunnel-CompletableFuture-Thread-2] - run shutdown hook because get close signal
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [seatunnel-engin] MongoDB CDC 同步数据时报错 [seatunnel]
Posted by "DanielCarter-stack (via GitHub)" <gi...@apache.org>.
DanielCarter-stack commented on issue #10398:
URL: https://github.com/apache/seatunnel/issues/10398#issuecomment-3798641942
<!-- code-pr-reviewer -->
Thanks for reporting. The NPE at `SourceReaderBase.java:210` occurs when `splitStates.remove(finishedSplitId)` returns null, meaning a split ID in `finishedSplits` was never registered in the `splitStates` ConcurrentHashMap.
**Evidence:**
- Fault line: `SourceReaderBase.java:210` in `finishCurrentFetch()` - `splitStates.remove(finishedSplitId).state`
- Registration point: `SourceReaderBase.java:127-137` (`addSplits` method)
- Finished splits generation: `IncrementalSourceSplitReader.java:155-160` uses `currentSplitId`
- MongoDB CDC inherits this via `MongodbIncrementalSource` extends `IncrementalSource`
This suggests a state inconsistency between the split enumerator and reader, likely a race condition or split lifecycle management issue in concurrent scenarios.
**To help diagnose:**
1. Does this error occur at approximately the same data volume (~1M records) each time?
2. Is checkpoint enabled? Do you see checkpoint failures around the same time?
3. Are there any logs about split reassignment/recovery before the crash?
**Temporary workarounds:**
- Adjust `chunk.size` or `batch.size` to reduce split count
- Increase `parallelism` to change split allocation patterns
- Try the latest 2.3.x release if this is a known issue
Please share your job configuration and full logs (especially "Adding split(s) to reader" messages).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [seatunnel-engin] Error occurs when synchronizing data with MongoDB CDC [seatunnel]
Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on issue #10398:
URL: https://github.com/apache/seatunnel/issues/10398#issuecomment-3851318766
> [@yzeng1618](https://github.com/yzeng1618) [@DanielCarter-stack](https://github.com/DanielCarter-stack) Please tell me in which version this bug will be fixed, and how long it is expected to take. Thank you.
The current bug fixes have been merged into the dev branch, and version 2.3.13 will be released soon. Stay tuned!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [seatunnel-engin] MongoDB CDC 同步数据时报错 [seatunnel]
Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on issue #10398:
URL: https://github.com/apache/seatunnel/issues/10398#issuecomment-3798796102
This part needs to ensure in IncrementalSourceSplitReader that the "split finished" event is emitted only once and the splitId is never null. I will fix this issue later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [seatunnel-engin] Error occurs when synchronizing data with MongoDB CDC [seatunnel]
Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 closed issue #10398: [Bug] [seatunnel-engin] Error occurs when synchronizing data with MongoDB CDC
URL: https://github.com/apache/seatunnel/issues/10398
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [seatunnel-engin] MongoDB CDC 同步数据时报错 [seatunnel]
Posted by "letong520 (via GitHub)" <gi...@apache.org>.
letong520 commented on issue #10398:
URL: https://github.com/apache/seatunnel/issues/10398#issuecomment-3798325943
<img width="701" height="233" alt="Image" src="https://github.com/user-attachments/assets/d054cc32-4ab6-45b0-8803-9dc46d9a5353" />
mongodb的数据如图
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [seatunnel-engin] Error occurs when synchronizing data with MongoDB CDC [seatunnel]
Posted by "letong520 (via GitHub)" <gi...@apache.org>.
letong520 commented on issue #10398:
URL: https://github.com/apache/seatunnel/issues/10398#issuecomment-3845951959
@yzeng1618 @DanielCarter-stack
Please tell me in which version this bug will be fixed, and how long it is expected to take. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org