You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "adamfarmer0 (via GitHub)" <gi...@apache.org> on 2025/01/08 19:18:11 UTC
[I] [Bug] [Sink-JDBC] Out of memory issue. [seatunnel]
adamfarmer0 opened a new issue, #8484:
URL: https://github.com/apache/seatunnel/issues/8484
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
I am running seatunnel cluster in a simple docker container,
here is my dockerfile for context:
```
FROM adoptopenjdk:8-jdk-hotspot
ENV SEATUNNEL_HOME=/app/apache-seatunnel-2.3.8
ENV PATH=$PATH:$SEATUNNEL_HOME/bin
RUN mkdir -p /tmp/seatunnel/checkpoint_snapshot
RUN chmod -R 777 /tmp/seatunnel/checkpoint_snapshot
WORKDIR /app
ADD . /app
ENV version="2.3.8"
RUN echo 'export SEATUNNEL_HOME=/app/apache-seatunnel-2.3.8' >> /etc/profile.d/seatunnel.sh && \
echo 'export PATH=$PATH:$SEATUNNEL_HOME/bin' >> /etc/profile.d/seatunnel.sh
RUN chmod +x /etc/profile.d/seatunnel.sh
EXPOSE 5801
CMD seatunnel-cluster.sh
```
There are no changes to seatunnel configurations, these connectors are active, but I am only using jdbc source (mssql) and clickhouse sink jobs.
`connector-cdc-mysql
connector-cdc-mongodb
connector-cdc-sqlserver
connector-cdc-postgres
connector-clickhouse
connector-elasticsearch
connector-email
connector-file-ftp
connector-file-local
connector-file-sftp
connector-http-base
connector-jdbc
connector-kafka
connector-mongodb
connector-rabbitmq
connector-redis
connector-activemq`
I use 5801 port with api to submit jobs to seatunnel. I am not using checkpoint and I use it always with batch mode.
Memory usage graph:

After a day, It uses more than 4gb of ram and I see that after every job, memory increases and drops a bit, but it never goes back to its starting memory. It's memory increases by each job depending on job's size and after a day it starts getting out of memory exception. Before updating I was using version 2.3.3, and in that version after memory exception container got exited and using always restart option in docker it restarted automatically and only one job was lost, after updating to 2.3.8, when I get out of memory exception nothing happens and all my jobs fail and docker container did not exit.
Note that I run about 5000 jobs each day, all are simple but they run every 15 minutes. all submitted through submit-job endpoint.
Some error logs are added to attachments.
[errors.log](https://github.com/user-attachments/files/18351905/errors.log)
### SeaTunnel Version
2.3.8
### SeaTunnel Config
```conf
url: /hazelcast/rest/maps/submit-job?jobName=testjob
request body:
{
"env": {
"execution.parallelism": 4,
"job.mode": "BATCH",
"checkpoint.interval": 10000
},
"source": [
{
"plugin_name": "Jdbc",
"url": "jdbc:sqlserver://*****:1433;database=***;trustServerCertificate=true",
"driver": "com.microsoft.sqlserver.jdbc.SQLServerDriver",
"connection_check_timeout_sec": 100,
"user": "******",
"password": "******",
"query": "select * from SOMETABLENAME"
}
],
"sink": [
{
"plugin_name": "Clickhouse",
"host": "192.168.1.1:8180",
"database": "my_database",
"table": "SOMETABLENAME",
"username": "******",
"password": "*******",
"clickhouse.confg": {
"max_rows_to_read": "100",
"read_overflow_mode": "throw"
}
}
]
}
```
```
### Running Command
```shell
/hazelcast/rest/maps/submit-job?jobName=testjob
```
### Error Exception
```log
[929069645775699969] 2025-01-07 18:00:12,618 INFO [o.a.s.c.s.j.s.ChunkSplitter ] [BlockingWorker-TaskGroupLocation{jobId=929069645775699969, pipelineId=1, taskGroupId=30000}] - Switch to dynamic chunk splitter
java.lang.OutOfMemoryError: Metaspace
Dumping heap to /tmp/seatunnel/dump/zeta-server ...
Unable to create /tmp/seatunnel/dump/zeta-server: No such file or directory
[929069645775831041] 2025-01-07 18:00:21,050 ERROR [.s.e.s.c.CheckpointCoordinator] [hz.main.generic-operation.thread-36] - report error from task
org.apache.seatunnel.common.utils.SeaTunnelException: java.lang.OutOfMemoryError: Metaspace
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.reportCheckpointErrorFromTask(CheckpointCoordinator.java:390) ~[seatunnel-starter.jar:2.3.8]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.reportCheckpointErrorFromTask(CheckpointManager.java:184) ~[seatunnel-starter.jar:2.3.8]
at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointErrorReportOperation.runInternal(CheckpointErrorReportOperation.java:48) ~[seatunnel-starter.jar:2.3.8]
at org.apache.seatunnel.engine.server.task.operation.TracingOperation.run(TracingOperation.java:44) ~[seatunnel-starter.jar:2.3.8]
at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189) ~[seatunnel-starter.jar:2.3.8]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273) ~[seatunnel-starter.jar:2.3.8]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248) ~[seatunnel-starter.jar:2.3.8]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213) ~[seatunnel-starter.jar:2.3.8]
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175) ~[seatunnel-starter.jar:2.3.8]
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139) ~[seatunnel-starter.jar:2.3.8]
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123) ~[seatunnel-starter.jar:2.3.8]
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) ~[seatunnel-starter.jar:2.3.8]
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
java 8
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Sink-JDBC] Out of memory issue. [seatunnel]
Posted by "adamfarmer0 (via GitHub)" <gi...@apache.org>.
adamfarmer0 commented on issue #8484:
URL: https://github.com/apache/seatunnel/issues/8484#issuecomment-2582046562
Thanks for the response, I will close the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Sink-JDBC] Out of memory issue. [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on issue #8484:
URL: https://github.com/apache/seatunnel/issues/8484#issuecomment-2579079185
Please enable classloader-cache-mode in seatunnel.yaml. You can refer https://seatunnel.apache.org/docs/2.3.8/seatunnel-engine/hybrid-cluster-deployment#45-class-loader-cache-mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Sink-JDBC] Out of memory issue. [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on issue #8484:
URL: https://github.com/apache/seatunnel/issues/8484#issuecomment-2579227760
Thanks @MyeoungDev for share this. BTW, the 2.3.8 default classloader-cache-mode is false.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Sink-JDBC] Out of memory issue. [seatunnel]
Posted by "MyeoungDev (via GitHub)" <gi...@apache.org>.
MyeoungDev commented on issue #8484:
URL: https://github.com/apache/seatunnel/issues/8484#issuecomment-2579178798
Hello, I encountered a similar issue with java.lang.OutOfMemoryError: Metaspace.
https://github.com/apache/seatunnel/issues/8288
Currently, the SeaTunnel 2.3.8 official documentation default classloader-cache-mode: true.
However, upon reviewing the actual implementation code, it was found to be set to false.
By setting the classloader-cache-mode option to true, the classLoader will be reused for each submit-job, which should resolve your issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Sink-JDBC] Out of memory issue. [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on issue #8484:
URL: https://github.com/apache/seatunnel/issues/8484#issuecomment-2579243049
> If the documentation hasn’t changed, it seems there might have been a misunderstanding on our end.
I checked you shared the doc link in https://github.com/apache/seatunnel/issues/8288, it's unreleased doc version, that's meaning the version of doc is next version 2.3.9.


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Sink-JDBC] Out of memory issue. [seatunnel]
Posted by "MyeoungDev (via GitHub)" <gi...@apache.org>.
MyeoungDev commented on issue #8484:
URL: https://github.com/apache/seatunnel/issues/8484#issuecomment-2581496966
> > If the documentation hasn’t changed, it seems there might have been a misunderstanding on our end.
>
> I checked you shared the doc link in #8288, it's unreleased doc version, that's meaning the version of doc is next version 2.3.9.  
I am so sorry.
I think I misunderstood by mistake.
I will check again next time.
Thank you for your kind reply, and thank you for always contributing to the development of SeaTunnel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Sink-JDBC] Out of memory issue. [seatunnel]
Posted by "adamfarmer0 (via GitHub)" <gi...@apache.org>.
adamfarmer0 closed issue #8484: [Bug] [Sink-JDBC] Out of memory issue.
URL: https://github.com/apache/seatunnel/issues/8484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Sink-JDBC] Out of memory issue. [seatunnel]
Posted by "MyeoungDev (via GitHub)" <gi...@apache.org>.
MyeoungDev commented on issue #8484:
URL: https://github.com/apache/seatunnel/issues/8484#issuecomment-2579234501
> Thanks @MyeoungDev for share this. BTW, the 2.3.8 default classloader-cache-mode is false. 
Oh, I apologize.
My colleague and I double-checked several times and confirmed that the value was indicated as true in the documentation. That’s why we reported this issue.
If the documentation hasn’t changed, it seems there might have been a misunderstanding on our end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org