You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "jochenshen24 (via GitHub)" <gi...@apache.org> on 2023/06/20 11:44:54 UTC

[GitHub] [seatunnel] jochenshen24 opened a new issue, #4957: [Bug] [JDBC] 达梦数据同步问题

jochenshen24 opened a new issue, #4957:
URL: https://github.com/apache/seatunnel/issues/4957

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   同步达梦数据库到mysql, 报错:Caused by: org.apache.seatunnel.api.common.PrepareFailException: ErrorCode:[API-01], ErrorDescription:[Configuration item validate failed] - PluginName: jdbc, PluginType: source, Message: org.apache.seatunnel.connectors.seatunnel.jdbc.exception.JdbcConnectorException: ErrorCode:[COMMON-06], ErrorDescription:[Illegal argument] - field "id" not contain in query select * from t_1。
   
   达梦数据库开启了大小写敏感参数
   select case_sensitive
   <img width="90" alt="image" src="https://github.com/apache/seatunnel/assets/105631880/9d4b1215-1217-44d9-882f-45f73309ea52">
   
   数据表SQL:
   CREATE TABLE "SHM"."t_1"
   (
   "id" INT NOT NULL,
   "a" VARCHAR2(50) NOT NULL,
   "b" DATETIME(6) NOT NULL,
   NOT CLUSTER PRIMARY KEY("id")) STORAGE(ON "SHM", CLUSTERBTR) ;
   
   seatunnel配置文件,配置partition_column参数后,执行异常;数据库关闭case_sensitive 大小写敏感后正常。
   
   
   ### SeaTunnel Version
   
   2.3.1
   
   ### SeaTunnel Config
   
   ```conf
   env {
     # You can set SeaTunnel environment configuration here
     execution.parallelism = 2
     job.mode = "BATCH"
     #checkpoint.interval = 10000
     #execution.checkpoint.interval = 10000
     #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
   }
   
   source {
     # This is a example source plugin **only for test and demonstrate the feature source plugin**
     Jdbc {
         driver = "dm.jdbc.driver.DmDriver"
         url = "jdbc:dm://192.168.0.13:5236?schema=shm"
         user = "xxx"
         password = "xxx"
   
         partition_column = id
         query = "select * from t_1"
     }
   
     # If you would like to get more information about how to configure Seatunnel and see full list of source plugins,
     # please go to https://seatunnel.apache.org/docs/category/source-v2
   }
   
   sink {
     Jdbc {
       driver = "com.mysql.cj.jdbc.Driver"
       url = "jdbc:mysql://xxx:xxx/test?useUnicode=true&characterEncoding=utf8&useSSL=false"
       user = "xxx"
       password = "xxx"
   
       database = "test"
       table = "test"
       primary_keys = id
       batch_size = 1000
     }
   
     # If you would like to get more information about how to configure Seatunnel and see full list of sink plugins,
     # please go to https://seatunnel.apache.org/docs/category/sink-v2
   }
   ```
   
   
   ### Running Command
   
   ```shell
   bin/seatunnel.sh --config config/1.config -m local
   ```
   
   
   ### Error Exception
   
   ```log
   2023-06-20 19:41:46,892 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /data/applications/apache-seatunnel-incubating-2.3.1/connectors/seatunnel
   2023-06-20 19:41:46,902 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: Jdbc at: file:/data/applications/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-jdbc-2.3.1.jar
   2023-06-20 19:41:46,912 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Jdbc'} from path: file:/data/applications/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-jdbc-2.3.1.jar use classloader: org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader
   2023-06-20 19:41:47,350 WARN  org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource - get row type info exception
   dm.jdbc.driver.DMException: 第1 行附近出现错误:
   无效的表或视图名[T_1]
           at dm.jdbc.driver.DBError.throwException(DBError.java:711) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
           at dm.jdbc.a.b.o.x(MSG.java:591) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
           at dm.jdbc.a.b.o.C(MSG.java:526) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
           at dm.jdbc.a.b.o.B(MSG.java:507) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
           at dm.jdbc.a.a.a(DBAccess.java:245) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
           at dm.jdbc.a.a.a(DBAccess.java:718) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
           at dm.jdbc.driver.DmdbPreparedStatement.prepareSql(DmdbPreparedStatement.java:303) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
           at dm.jdbc.driver.DmdbPreparedStatement.allocateHandle(DmdbPreparedStatement.java:229) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
           at dm.jdbc.driver.DmdbPreparedStatement.<init>(DmdbPreparedStatement.java:131) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
           at dm.jdbc.driver.DmdbPreparedStatement.<init>(DmdbPreparedStatement.java:181) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
           at dm.jdbc.driver.DmdbConnection.do_prepareStatement(DmdbConnection.java:722) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
           at dm.jdbc.driver.DmdbConnection.do_prepareStatement(DmdbConnection.java:708) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
           at dm.jdbc.driver.DmdbConnection.prepareStatement(DmdbConnection.java:1332) ~[DmJdbcDriver18-8.1.2.192.jar:- 8.1.2.192 - Production]
           at org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.JdbcDialect.getResultSetMetaData(JdbcDialect.java:193) ~[connector-jdbc-2.3.1.jar:2.3.1]
           at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource.initTableField(JdbcSource.java:152) [connector-jdbc-2.3.1.jar:2.3.1]
           at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource.prepare(JdbcSource.java:91) [connector-jdbc-2.3.1.jar:2.3.1]
           at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:64) [seatunnel-starter.jar:2.3.1]
           at org.apache.seatunnel.engine.core.parse.JobConfigParser.sampleAnalyze(JobConfigParser.java:371) [seatunnel-starter.jar:2.3.1]
           at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:132) [seatunnel-starter.jar:2.3.1]
           at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:112) [seatunnel-starter.jar:2.3.1]
           at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155) [seatunnel-starter.jar:2.3.1]
           at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147) [seatunnel-starter.jar:2.3.1]
           at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140) [seatunnel-starter.jar:2.3.1]
           at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) [seatunnel-starter.jar:2.3.1]
           at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) [seatunnel-starter.jar:2.3.1]
   2023-06-20 19:41:47,357 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-260464] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
   2023-06-20 19:41:47,362 INFO  com.hazelcast.internal.server.tcp.TcpServerConnection - [localhost]:5801 [seatunnel-260464] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:43114, qualifier=null, endpoint=[127.0.0.1]:43114, remoteUuid=dafc4752-5c7c-46c5-aec7-400e6279e134, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
   2023-06-20 19:41:47,364 INFO  com.hazelcast.client.impl.ClientEndpointManager - [localhost]:5801 [seatunnel-260464] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:43114, qualifier=null, endpoint=[127.0.0.1]:43114, remoteUuid=dafc4752-5c7c-46c5-aec7-400e6279e134, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=dafc4752-5c7c-46c5-aec7-400e6279e134, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1687261306666, latest clientAttributes=lastStatisticsCollectionTime=1687261306709,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1687261306651,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=4900560896,os.freePhysicalMemorySize=298549248,os.freeSwapSpaceSize=8455712768,os.maxFileDescriptorCount=4096,os.openFileDescriptorCount=53,os.processCpuTime=11510000000,os.systemLoadAverage=0.12,os.totalPhysicalMemorySize=8201256960,os.tot
 alSwapSpaceSize=8455712768,runtime.availableProcessors=5,runtime.freeMemory=180677216,runtime.maxMemory=477626368,runtime.totalMemory=218103808,runtime.uptime=4546,runtime.usedMemory=37426592, labels=[]}
   2023-06-20 19:41:47,365 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-260464] [5.1] Removed connection to endpoint: [localhost]:5801:ab23d2d5-3eaf-4a83-a6b9-5aa336a10a51, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:43114->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2023-06-20 19:41:46.746, lastWriteTime=2023-06-20 19:41:46.744, closedTime=2023-06-20 19:41:47.360, connected server version=5.1}
   2023-06-20 19:41:47,365 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-260464] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
   2023-06-20 19:41:47,370 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-260464] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
   2023-06-20 19:41:47,370 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
   2023-06-20 19:41:47,371 INFO  com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-260464] [5.1] [localhost]:5801 is SHUTTING_DOWN
   2023-06-20 19:41:47,375 INFO  com.hazelcast.internal.partition.impl.MigrationManager - [localhost]:5801 [seatunnel-260464] [5.1] Shutdown request of Member [localhost]:5801 - ab23d2d5-3eaf-4a83-a6b9-5aa336a10a51 this is handled
   2023-06-20 19:41:47,381 INFO  com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-260464] [5.1] Shutting down connection manager...
   2023-06-20 19:41:47,384 INFO  com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-260464] [5.1] Shutting down node engine...
   2023-06-20 19:41:47,396 INFO  org.apache.seatunnel.engine.server.SeaTunnelServer - master node check interrupted
   2023-06-20 19:41:50,436 INFO  com.hazelcast.instance.impl.NodeExtension - [localhost]:5801 [seatunnel-260464] [5.1] Destroying node NodeExtension.
   2023-06-20 19:41:50,437 INFO  com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-260464] [5.1] Hazelcast Shutdown is completed in 3063 ms.
   2023-06-20 19:41:50,438 INFO  com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-260464] [5.1] [localhost]:5801 is SHUTDOWN
   2023-06-20 19:41:50,439 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed HazelcastInstance ......
   2023-06-20 19:41:50,439 ERROR org.apache.seatunnel.core.starter.SeaTunnel - 
   
   ===============================================================================
   
   
   2023-06-20 19:41:50,439 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Fatal Error, 
   
   2023-06-20 19:41:50,440 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Please submit bug report in https://github.com/apache/incubator-seatunnel/issues
   
   2023-06-20 19:41:50,440 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Reason:SeaTunnel job executed failed 
   
   2023-06-20 19:41:50,444 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
           at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
           at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
           at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
   Caused by: org.apache.seatunnel.api.common.PrepareFailException: ErrorCode:[API-01], ErrorDescription:[Configuration item validate failed] - PluginName: jdbc, PluginType: source, Message: org.apache.seatunnel.connectors.seatunnel.jdbc.exception.JdbcConnectorException: ErrorCode:[COMMON-06], ErrorDescription:[Illegal argument] - field id not contain in query select * from t_1
           at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource.prepare(JdbcSource.java:96)
           at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:64)
           at org.apache.seatunnel.engine.core.parse.JobConfigParser.sampleAnalyze(JobConfigParser.java:371)
           at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:132)
           at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:112)
           at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
           at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
           at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
           ... 2 more
    
   2023-06-20 19:41:50,444 ERROR org.apache.seatunnel.core.starter.SeaTunnel - 
   ===============================================================================
   
   
   
   Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
           at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
           at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
           at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
   Caused by: org.apache.seatunnel.api.common.PrepareFailException: ErrorCode:[API-01], ErrorDescription:[Configuration item validate failed] - PluginName: jdbc, PluginType: source, Message: org.apache.seatunnel.connectors.seatunnel.jdbc.exception.JdbcConnectorException: ErrorCode:[COMMON-06], ErrorDescription:[Illegal argument] - field id not contain in query select * from t_1
           at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource.prepare(JdbcSource.java:96)
           at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:64)
           at org.apache.seatunnel.engine.core.parse.JobConfigParser.sampleAnalyze(JobConfigParser.java:371)
           at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:132)
           at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:112)
           at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
           at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
           at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
           ... 2 more
   ```
   
   
   ### Flink or Spark Version
   
   _No response_
   
   ### Java or Scala Version
   
   _No response_
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [JDBC] 达梦数据同步问题 [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4957:
URL: https://github.com/apache/seatunnel/issues/4957#issuecomment-3100149430

   This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] laglangyue commented on issue #4957: [Bug] [JDBC] 达梦数据同步问题

Posted by "laglangyue (via GitHub)" <gi...@apache.org>.
laglangyue commented on issue #4957:
URL: https://github.com/apache/seatunnel/issues/4957#issuecomment-1623252371

   ![image](https://github.com/apache/seatunnel/assets/35491928/6ddd44e9-859e-499e-92d0-4427455d7aaa)
   
   Are your task configurations and table names the same


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [JDBC] 达梦数据同步问题 [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4957:
URL: https://github.com/apache/seatunnel/issues/4957#issuecomment-3067449717

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] laglangyue commented on issue #4957: [Bug] [JDBC] 达梦数据同步问题

Posted by "laglangyue (via GitHub)" <gi...@apache.org>.
laglangyue commented on issue #4957:
URL: https://github.com/apache/seatunnel/issues/4957#issuecomment-1623253304

   > ![image](https://user-images.githubusercontent.com/35491928/251398790-6ddd44e9-859e-499e-92d0-4427455d7aaa.png)
   > 
   > Are your task configurations and table names the same
   
   Include library name


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [JDBC] 达梦数据同步问题 [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4957: [Bug] [JDBC] 达梦数据同步问题
URL: https://github.com/apache/seatunnel/issues/4957


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org