You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "jochenshen24 (via GitHub)" <gi...@apache.org> on 2023/06/20 11:44:54 UTC
[GitHub] [seatunnel] jochenshen24 opened a new issue, #4957: [Bug] [JDBC] 达梦数据同步问题
jochenshen24 opened a new issue, #4957:
URL: https://github.com/apache/seatunnel/issues/4957
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
同步达梦数据库到mysql, 报错:Caused by: org.apache.seatunnel.api.common.PrepareFailException: ErrorCode:[API-01], ErrorDescription:[Configuration item validate failed] - PluginName: jdbc, PluginType: source, Message: org.apache.seatunnel.connectors.seatunnel.jdbc.exception.JdbcConnectorException: ErrorCode:[COMMON-06], ErrorDescription:[Illegal argument] - field "id" not contain in query select * from t_1。
达梦数据库开启了大小写敏感参数
select case_sensitive
<img width="90" alt="image" src="https://github.com/apache/seatunnel/assets/105631880/9d4b1215-1217-44d9-882f-45f73309ea52">
数据表SQL:
CREATE TABLE "SHM"."t_1"
(
"id" INT NOT NULL,
"a" VARCHAR2(50) NOT NULL,
"b" DATETIME(6) NOT NULL,
NOT CLUSTER PRIMARY KEY("id")) STORAGE(ON "SHM", CLUSTERBTR) ;
seatunnel配置文件,配置partition_column参数后,执行异常;数据库关闭case_sensitive 大小写敏感后正常。
### SeaTunnel Version
2.3.1
### SeaTunnel Config
```conf
env {
# You can set SeaTunnel environment configuration here
execution.parallelism = 2
job.mode = "BATCH"
#checkpoint.interval = 10000
#execution.checkpoint.interval = 10000
#execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}
source {
# This is a example source plugin **only for test and demonstrate the feature source plugin**
Jdbc {
driver = "dm.jdbc.driver.DmDriver"
url = "jdbc:dm://192.168.0.13:5236?schema=shm"
user = "xxx"
password = "xxx"
partition_column = id
query = "select * from t_1"
}
# If you would like to get more information about how to configure Seatunnel and see full list of source plugins,
# please go to https://seatunnel.apache.org/docs/category/source-v2
}
sink {
Jdbc {
driver = "com.mysql.cj.jdbc.Driver"
url = "jdbc:mysql://xxx:xxx/test?useUnicode=true&characterEncoding=utf8&useSSL=false"
user = "xxx"
password = "xxx"
database = "test"
table = "test"
primary_keys = id
batch_size = 1000
}
# If you would like to get more information about how to configure Seatunnel and see full list of sink plugins,
# please go to https://seatunnel.apache.org/docs/category/sink-v2
}
```
### Running Command
```shell
bin/seatunnel.sh --config config/1.config -m local
```
### Error Exception
```log
2023-06-20 19:41:46,892 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /data/applications/apache-seatunnel-incubating-2.3.1/connectors/seatunnel
2023-06-20 19:41:46,902 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: Jdbc at: file:/data/applications/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-jdbc-2.3.1.jar
2023-06-20 19:41:46,912 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Jdbc'} from path: file:/data/applications/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-jdbc-2.3.1.jar use classloader: org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader
2023-06-20 19:41:47,350 WARN org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource - get row type info exception
dm.jdbc.driver.DMException: 第1 行附近出现错误:
无效的表或视图名[T_1]
at dm.jdbc.driver.DBError.throwException(DBError.java:711) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
at dm.jdbc.a.b.o.x(MSG.java:591) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
at dm.jdbc.a.b.o.C(MSG.java:526) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
at dm.jdbc.a.b.o.B(MSG.java:507) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
at dm.jdbc.a.a.a(DBAccess.java:245) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
at dm.jdbc.a.a.a(DBAccess.java:718) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
at dm.jdbc.driver.DmdbPreparedStatement.prepareSql(DmdbPreparedStatement.java:303) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
at dm.jdbc.driver.DmdbPreparedStatement.allocateHandle(DmdbPreparedStatement.java:229) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
at dm.jdbc.driver.DmdbPreparedStatement.<init>(DmdbPreparedStatement.java:131) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
at dm.jdbc.driver.DmdbPreparedStatement.<init>(DmdbPreparedStatement.java:181) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
at dm.jdbc.driver.DmdbConnection.do_prepareStatement(DmdbConnection.java:722) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
at dm.jdbc.driver.DmdbConnection.do_prepareStatement(DmdbConnection.java:708) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
at dm.jdbc.driver.DmdbConnection.prepareStatement(DmdbConnection.java:1332) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
at org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.JdbcDialect.getResultSetMetaData(JdbcDialect.java:193) ~[connector-jdbc-2.3.1.jar:2.3.1]
at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource.initTableField(JdbcSource.java:152) [connector-jdbc-2.3.1.jar:2.3.1]
at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource.prepare(JdbcSource.java:91) [connector-jdbc-2.3.1.jar:2.3.1]
at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:64) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.core.parse.JobConfigParser.sampleAnalyze(JobConfigParser.java:371) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:132) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:112) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) [seatunnel-starter.jar:2.3.1]
2023-06-20 19:41:47,357 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-260464] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2023-06-20 19:41:47,362 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [localhost]:5801 [seatunnel-260464] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:43114, qualifier=null, endpoint=[127.0.0.1]:43114, remoteUuid=dafc4752-5c7c-46c5-aec7-400e6279e134, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2023-06-20 19:41:47,364 INFO com.hazelcast.client.impl.ClientEndpointManager - [localhost]:5801 [seatunnel-260464] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:43114, qualifier=null, endpoint=[127.0.0.1]:43114, remoteUuid=dafc4752-5c7c-46c5-aec7-400e6279e134, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=dafc4752-5c7c-46c5-aec7-400e6279e134, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1687261306666, latest clientAttributes=lastStatisticsCollectionTime=1687261306709,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1687261306651,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=4900560896,os.freePhysicalMemorySize=298549248,os.freeSwapSpaceSize=8455712768,os.maxFileDescriptorCount=4096,os.openFileDescriptorCount=53,os.processCpuTime=11510000000,os.systemLoadAverage=0.12,os.totalPhysicalMemorySize=8201256960,os.tot
alSwapSpaceSize=8455712768,runtime.availableProcessors=5,runtime.freeMemory=180677216,runtime.maxMemory=477626368,runtime.totalMemory=218103808,runtime.uptime=4546,runtime.usedMemory=37426592, labels=[]}
2023-06-20 19:41:47,365 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-260464] [5.1] Removed connection to endpoint: [localhost]:5801:ab23d2d5-3eaf-4a83-a6b9-5aa336a10a51, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:43114->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2023-06-20 19:41:46.746, lastWriteTime=2023-06-20 19:41:46.744, closedTime=2023-06-20 19:41:47.360, connected server version=5.1}
2023-06-20 19:41:47,365 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-260464] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2023-06-20 19:41:47,370 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-260464] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2023-06-20 19:41:47,370 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
2023-06-20 19:41:47,371 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-260464] [5.1] [localhost]:5801 is SHUTTING_DOWN
2023-06-20 19:41:47,375 INFO com.hazelcast.internal.partition.impl.MigrationManager - [localhost]:5801 [seatunnel-260464] [5.1] Shutdown request of Member [localhost]:5801 - ab23d2d5-3eaf-4a83-a6b9-5aa336a10a51 this is handled
2023-06-20 19:41:47,381 INFO com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-260464] [5.1] Shutting down connection manager...
2023-06-20 19:41:47,384 INFO com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-260464] [5.1] Shutting down node engine...
2023-06-20 19:41:47,396 INFO org.apache.seatunnel.engine.server.SeaTunnelServer - master node check interrupted
2023-06-20 19:41:50,436 INFO com.hazelcast.instance.impl.NodeExtension - [localhost]:5801 [seatunnel-260464] [5.1] Destroying node NodeExtension.
2023-06-20 19:41:50,437 INFO com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-260464] [5.1] Hazelcast Shutdown is completed in 3063 ms.
2023-06-20 19:41:50,438 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-260464] [5.1] [localhost]:5801 is SHUTDOWN
2023-06-20 19:41:50,439 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed HazelcastInstance ......
2023-06-20 19:41:50,439 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
===============================================================================
2023-06-20 19:41:50,439 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Fatal Error,
2023-06-20 19:41:50,440 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Please submit bug report in https://github.com/apache/incubator-seatunnel/issues
2023-06-20 19:41:50,440 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Reason:SeaTunnel job executed failed
2023-06-20 19:41:50,444 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.api.common.PrepareFailException: ErrorCode:[API-01], ErrorDescription:[Configuration item validate failed] - PluginName: jdbc, PluginType: source, Message: org.apache.seatunnel.connectors.seatunnel.jdbc.exception.JdbcConnectorException: ErrorCode:[COMMON-06], ErrorDescription:[Illegal argument] - field id not contain in query select * from t_1
at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource.prepare(JdbcSource.java:96)
at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:64)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.sampleAnalyze(JobConfigParser.java:371)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:132)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:112)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
... 2 more
2023-06-20 19:41:50,444 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
===============================================================================
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.api.common.PrepareFailException: ErrorCode:[API-01], ErrorDescription:[Configuration item validate failed] - PluginName: jdbc, PluginType: source, Message: org.apache.seatunnel.connectors.seatunnel.jdbc.exception.JdbcConnectorException: ErrorCode:[COMMON-06], ErrorDescription:[Illegal argument] - field id not contain in query select * from t_1
at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource.prepare(JdbcSource.java:96)
at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:64)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.sampleAnalyze(JobConfigParser.java:371)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:132)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:112)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
... 2 more
```
### Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [JDBC] 达梦数据同步问题 [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4957:
URL: https://github.com/apache/seatunnel/issues/4957#issuecomment-3100149430
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] laglangyue commented on issue #4957: [Bug] [JDBC] 达梦数据同步问题
Posted by "laglangyue (via GitHub)" <gi...@apache.org>.
laglangyue commented on issue #4957:
URL: https://github.com/apache/seatunnel/issues/4957#issuecomment-1623252371

Are your task configurations and table names the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [JDBC] 达梦数据同步问题 [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4957:
URL: https://github.com/apache/seatunnel/issues/4957#issuecomment-3067449717
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] laglangyue commented on issue #4957: [Bug] [JDBC] 达梦数据同步问题
Posted by "laglangyue (via GitHub)" <gi...@apache.org>.
laglangyue commented on issue #4957:
URL: https://github.com/apache/seatunnel/issues/4957#issuecomment-1623253304
> 
>
> Are your task configurations and table names the same
Include library name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [JDBC] 达梦数据同步问题 [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4957: [Bug] [JDBC] 达梦数据同步问题
URL: https://github.com/apache/seatunnel/issues/4957
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org