You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "yingh0ng (via GitHub)" <gi...@apache.org> on 2025/07/11 06:26:41 UTC
[I] [Bug] [connector-jdbc] mysql to sqlserver is very slow [seatunnel]
yingh0ng opened a new issue, #9562:
URL: https://github.com/apache/seatunnel/issues/9562
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Read a table from mysql, and write to sqlserver is very slow. As a comparison, the datax is more faster than seatunnel.
Are my configs not right or other problems? Please tell me.
### SeaTunnel Version
compile from dev
### SeaTunnel Config
```conf
env {
parallelism = 4
job.mode = "BATCH"
}
source {
Jdbc {
driver = com.mysql.cj.jdbc.Driver
url = "jdbc:mysql://***:3306/bi_test?serverTimezone=Asia/Shanghai&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"
user = "***"
password = "***"
# 分区读取配置(大数据量时启用)
partition_column="id"
partition_column = "id" # 用于分区的列
partition_num = 10 # 分区数量
fetch_size = 10000
query = "select id, 订单ID, 订单日期, 日期时间, 城市, 省自治区, 地区, 类别, 子类别, 产品名称, 销售额, 销量, 利润 from 订单明细1000000"
}
}
sink {
Jdbc {
driver = com.microsoft.sqlserver.jdbc.SQLServerDriver
url = "jdbc:sqlserver://***:1433;databaseName=bi_test;useBulkCopyForBatchInsert=true;bulkCopyBatchSize=5000"
user = SA
password = "***"
# 批量写入优化
batch_size = 5000 # 每次批量写入的记录数
batch_interval_ms = 1000 # 批处理间隔(毫秒)
query = "INSERT INTO 订单明细100w (id,订单ID, 订单日期, 日期时间, 城市, 省自治区, 地区, 类别, 子类别, 产品名称, 销售额, 销量, 利润) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
}
}
```
### Running Command
```shell
./bin/seatunnel.sh --config ./jobs/sqlserver100w.conf -m local
```
### Error Exception
```log
See Screenshots
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
java version: 17.0.15
scala:2.11.12
### Screenshots
### The seatunnel job info
<img width="1831" height="1054" alt="Image" src="https://github.com/user-attachments/assets/ed128367-da42-44a9-adc0-89b245236007" />
### The datax job info running in the same machine
<img width="1875" height="1270" alt="Image" src="https://github.com/user-attachments/assets/54b7d70c-3c79-403b-94b0-206426710257" />
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question] [connector-jdbc] mysql to sqlserver is very slow [seatunnel]
Posted by "yingh0ng (via GitHub)" <gi...@apache.org>.
yingh0ng commented on issue #9562:
URL: https://github.com/apache/seatunnel/issues/9562#issuecomment-3077081082
> You can have a try like this
>
> ```
>
> env {
> job.mode = "BATCH"
> parallelism = 4
> }
>
> source {
> Jdbc {
> url = "jdbc:mysql://your-mysql-host:3306/your_dbserverTimezone=Asia/Shanghai&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"
> driver = "com.mysql.cj.jdbc.Driver"
> user = "your_user"
> password = "your_password"
>
> query = "SELECT * FROM your_table"
> partition_column = "id"
> partition_num = 4
> fetch_size = 5000
> }
> }
>
> sink {
> Jdbc {
> url = "jdbc:sqlserver://your-sqlserver-host:1433;databaseName=your_db;useBulkCopyForBatchInsert=false"
> driver = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
> user = "your_user"
> password = "your_password"
>
> table = "your_table"
> primary_keys = ["id"]
> enable_upsert = false
> batch_size = 5000
> auto_commit = false
> max_retries = 2
> }
> }
> ```
It works, thank you!
So, the problem is the `primary_keys` config?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question] [connector-jdbc] mysql to sqlserver is very slow [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on issue #9562:
URL: https://github.com/apache/seatunnel/issues/9562#issuecomment-3069681858
You can have a try like this
```
env {
job.mode = "BATCH"
parallelism = 4
}
source {
Jdbc {
url = "jdbc:mysql://your-mysql-host:3306/your_dbserverTimezone=Asia/Shanghai&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"
driver = "com.mysql.cj.jdbc.Driver"
user = "your_user"
password = "your_password"
query = "SELECT * FROM your_table"
partition_column = "id"
partition_num = 4
fetch_size = 5000
}
}
sink {
Jdbc {
url = "jdbc:sqlserver://your-sqlserver-host:1433;databaseName=your_db;useBulkCopyForBatchInsert=false"
driver = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
user = "your_user"
password = "your_password"
table = "your_table"
primary_keys = ["id"]
enable_upsert = false
batch_size = 5000
auto_commit = false
max_retries = 2
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question] [connector-jdbc] mysql to sqlserver is very slow [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on issue #9562:
URL: https://github.com/apache/seatunnel/issues/9562#issuecomment-3067587176
Can you paste the runtime logs in detail ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question] [connector-jdbc] mysql to sqlserver is very slow [seatunnel]
Posted by "yingh0ng (via GitHub)" <gi...@apache.org>.
yingh0ng closed issue #9562: [Question] [connector-jdbc] mysql to sqlserver is very slow
URL: https://github.com/apache/seatunnel/issues/9562
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org