You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "cs3163077 (via GitHub)" <gi...@apache.org> on 2023/06/30 07:32:04 UTC
[GitHub] [seatunnel] cs3163077 opened a new issue, #5003: SqlServer-CDC 没有数据
cs3163077 opened a new issue, #5003:
URL: https://github.com/apache/seatunnel/issues/5003
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
SqlServer-CDC 没有数据,也没报错
### SeaTunnel Version
2.3.2
### SeaTunnel Config
```conf
env {
job.mode = "STREAMING"
}
source {
SqlServer-CDC {
result_table_name = "area"
username = "**"
password = "****"
database-names = ["BI"]
table-names = ["BI.dbo.area"]
base-url="jdbc:sqlserver://******:**;databaseName=BI"
}
}
transform {
}
sink {
Console {
}
}
```
### Running Command
```shell
./start-seatunnel-spark-2-connector-v2.sh --config /home/apache-seatunnel-incubating-2.3.1/config/cdc.conf
```
### Error Exception
```log
23/06/30 15:24:25 INFO spark.SparkContext: Added JAR file:/home/apache-seatunnel-incubating-2.3.1/starter/seatunnel-spark-2-starter.jar at spark://hadoop3:37437/jars/seatunnel-spark-2-starter.jar with timestamp 1688109865948
23/06/30 15:24:26 INFO executor.Executor: Starting executor ID driver on host localhost
23/06/30 15:24:26 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41040.
23/06/30 15:24:26 INFO netty.NettyBlockTransferService: Server created on hadoop3:41040
23/06/30 15:24:26 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
23/06/30 15:24:26 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, hadoop3, 41040, None)
23/06/30 15:24:26 INFO storage.BlockManagerMasterEndpoint: Registering block manager hadoop3:41040 with 366.3 MB RAM, BlockManagerId(driver, hadoop3, 41040, None)
23/06/30 15:24:26 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, hadoop3, 41040, None)
23/06/30 15:24:26 INFO storage.BlockManager: external shuffle service port = 7337
23/06/30 15:24:26 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, hadoop3, 41040, None)
23/06/30 15:24:26 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@41b1f51e{/metrics/json,null,AVAILABLE,@Spark}
23/06/30 15:24:27 INFO scheduler.EventLoggingListener: Logging events to hdfs://hadoop1:8020/user/spark/applicationHistory/local-1688109866035
23/06/30 15:24:27 WARN lineage.LineageWriter: Lineage directory /var/log/spark/lineage doesn't exist or is not writable. Lineage for this application will be disabled.
23/06/30 15:24:27 INFO util.Utils: Extension com.cloudera.spark.lineage.NavigatorAppListener not being initialized.
23/06/30 15:24:27 INFO logging.DriverLogger$DfsAsyncWriter: Started driver log file sync to: /user/spark/driverLogs/local-1688109866035_driver.log
23/06/30 15:24:28 INFO discovery.AbstractPluginDiscovery: Load SeaTunnelSource Plugin from /home/apache-seatunnel-incubating-2.3.1/connectors/seatunnel
23/06/30 15:24:28 INFO discovery.AbstractPluginDiscovery: Discovery plugin jar: SqlServer-CDC at: file:/home/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-cdc-sqlserver-2.3.2.jar
23/06/30 15:24:28 INFO discovery.AbstractPluginDiscovery: Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='SqlServer-CDC'} from classpath
23/06/30 15:24:28 WARN sqlserver.SqlServerConnection: The 'server.timezone' option should be specified to avoid incorrect timestamp values in case of different timezones between the database server and this connector's JVM.
23/06/30 15:24:28 INFO utils.TableDiscoveryUtils: Read list of available databases
23/06/30 15:24:29 INFO utils.TableDiscoveryUtils: list of available databases is: [master, tempdb, model, msdb, BI, BIDEMO, SYSD_BI_DEMO, SYSD_BI_DEV_JAVA, *****, SY_BI_COST, SY_BI_1141, TEST1, SYSD_BI_TEST_20211026, SYSD_BI_DEV_JAVA1, test123, testdatabig, test12134, SYBI_Test_TW, zhuhm_BIDW, bigdatatest, zhuhm_BIDW_E, zhuhm_BIDW1, *****, test_yx, S*****Test, SY_BI_CNCEC, BIDW5, *****, S*****, SY_BI_LSC]
23/06/30 15:24:29 INFO utils.TableDiscoveryUtils: Read list of available tables in each database
23/06/30 15:24:29 INFO utils.TableDiscoveryUtils: including 'BI.dbo.area' for further processing
23/06/30 15:24:29 WARN utils.TableDiscoveryUtils: skipping database '*****' due to error reading tables: 由于数据库 '*****' 离线,无法打开该数据库。
23/06/30 15:24:29 WARN utils.TableDiscoveryUtils: skipping database '*****' due to error reading tables: 由于数据库 '*****' 离线,无法打开该数据库。
23/06/30 15:24:29 WARN utils.TableDiscoveryUtils: skipping database '*****' due to error reading tables: 由于数据库 '*****' 离线,无法打开该数据库。
23/06/30 15:24:29 WARN utils.TableDiscoveryUtils: skipping database 'S*****' due to error reading tables: 由于数据库 'S*****' 离线,无法打开该数据库。
23/06/30 15:24:29 INFO jdbc.JdbcConnection: Connection gracefully closed
23/06/30 15:24:29 WARN sqlserver.SqlServerConnection: The 'server.timezone' option should be specified to avoid incorrect timestamp values in case of different timezones between the database server and this connector's JVM.
23/06/30 15:24:29 WARN sqlserver.SqlServerDefaultValueConverter: Cannot parse column default value '(getdate())' to type 'datetime'. Expression evaluation is not supported.
23/06/30 15:24:29 WARN sqlserver.SqlServerDefaultValueConverter: Cannot parse column default value '(getdate())' to type 'datetime'. Expression evaluation is not supported.
23/06/30 15:24:29 WARN sqlserver.SqlServerDefaultValueConverter: Cannot parse column default value '(getdate())' to type 'datetime'. Expression evaluation is not supported.
23/06/30 15:24:30 INFO jdbc.JdbcConnection: Connection gracefully closed
23/06/30 15:24:30 INFO execution.SparkRuntimeEnvironment: register plugins :[file:/home/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-cdc-sqlserver-2.3.2.jar]
23/06/30 15:24:30 INFO discovery.AbstractPluginDiscovery: Load SeaTunnelTransform Plugin from /home/apache-seatunnel-incubating-2.3.1/lib
23/06/30 15:24:30 INFO execution.SparkRuntimeEnvironment: register plugins :[]
23/06/30 15:24:30 INFO discovery.AbstractPluginDiscovery: Load SeaTunnelSink Plugin from /home/apache-seatunnel-incubating-2.3.1/connectors/seatunnel
23/06/30 15:24:30 INFO discovery.AbstractPluginDiscovery: Discovery plugin jar: Console at: file:/home/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-console-2.3.1.jar
23/06/30 15:24:30 INFO discovery.AbstractPluginDiscovery: Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Console'} from classpath
23/06/30 15:24:30 INFO execution.SparkRuntimeEnvironment: register plugins :[file:/home/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-console-2.3.1.jar]
23/06/30 15:24:30 INFO internal.SharedState: loading hive config file: file:/etc/hive/conf.cloudera.hive/hive-site.xml
23/06/30 15:24:30 INFO internal.SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse').
23/06/30 15:24:30 INFO internal.SharedState: Warehouse path is '/user/hive/warehouse'.
23/06/30 15:24:30 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4966bab1{/SQL,null,AVAILABLE,@Spark}
23/06/30 15:24:30 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7f0f84d4{/SQL/json,null,AVAILABLE,@Spark}
23/06/30 15:24:30 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@590d6c76{/SQL/execution,null,AVAILABLE,@Spark}
23/06/30 15:24:30 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@25791d40{/SQL/execution/json,null,AVAILABLE,@Spark}
23/06/30 15:24:30 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5a05dd30{/static/sql,null,AVAILABLE,@Spark}
23/06/30 15:24:31 INFO state.StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
23/06/30 15:24:31 WARN lineage.LineageWriter: Lineage directory /var/log/spark/lineage doesn't exist or is not writable. Lineage for this application will be disabled.
23/06/30 15:24:31 INFO util.Utils: Extension com.cloudera.spark.lineage.NavigatorQueryListener not being initialized.
23/06/30 15:24:34 INFO v2.DataSourceV2Strategy:
Pushing operators to class org.apache.seatunnel.translation.spark.source.SeaTunnelSourceSupport
Pushed Filters:
Post-Scan Filters:
Output: Areaid#0, Areaname#1
23/06/30 15:24:34 INFO codegen.CodeGenerator: Code generated in 199.1582 ms
23/06/30 15:24:34 INFO v2.WriteToDataSourceV2Exec: Start processing data source writer: org.apache.seatunnel.translation.spark.sink.writer.SparkDataSourceWriter@7b8d6c66. The input RDD has 1 partitions.
23/06/30 15:24:34 INFO spark.SparkContext: Starting job: save at SinkExecuteProcessor.java:123
23/06/30 15:24:34 INFO scheduler.DAGScheduler: Got job 0 (save at SinkExecuteProcessor.java:123) with 1 output partitions
23/06/30 15:24:34 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (save at SinkExecuteProcessor.java:123)
23/06/30 15:24:34 INFO scheduler.DAGScheduler: Parents of final stage: List()
23/06/30 15:24:34 INFO scheduler.DAGScheduler: Missing parents: List()
23/06/30 15:24:34 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at save at SinkExecuteProcessor.java:123), which has no missing parents
23/06/30 15:24:34 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 5.9 KB, free 366.3 MB)
23/06/30 15:24:35 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.1 KB, free 366.3 MB)
23/06/30 15:24:35 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on hadoop3:41040 (size: 3.1 KB, free: 366.3 MB)
23/06/30 15:24:35 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1164
23/06/30 15:24:35 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at save at SinkExecuteProcessor.java:123) (first 15 tasks are for partitions Vector(0))
23/06/30 15:24:35 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
23/06/30 15:24:35 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 16026 bytes)
23/06/30 15:24:35 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)
23/06/30 15:24:35 INFO executor.Executor: Fetching spark://hadoop3:37437/jars/connector-cdc-sqlserver-2.3.2.jar with timestamp 1688109865947
23/06/30 15:24:35 INFO client.TransportClientFactory: Successfully created connection to hadoop3/192.168.0.213:37437 after 75 ms (0 ms spent in bootstraps)
23/06/30 15:24:35 INFO util.Utils: Fetching spark://hadoop3:37437/jars/connector-cdc-sqlserver-2.3.2.jar to /tmp/spark-3382b920-11e6-4daf-ad0c-a523df4ead04/userFiles-a4ca48a0-10f3-468f-b5e3-b953bfa97755/fetchFileTemp8739054726675673564.tmp
23/06/30 15:24:35 INFO executor.Executor: Adding file:/tmp/spark-3382b920-11e6-4daf-ad0c-a523df4ead04/userFiles-a4ca48a0-10f3-468f-b5e3-b953bfa97755/connector-cdc-sqlserver-2.3.2.jar to class loader
23/06/30 15:24:35 INFO executor.Executor: Fetching spark://hadoop3:37437/jars/seatunnel-spark-2-starter.jar with timestamp 1688109865948
23/06/30 15:24:35 INFO util.Utils: Fetching spark://hadoop3:37437/jars/seatunnel-spark-2-starter.jar to /tmp/spark-3382b920-11e6-4daf-ad0c-a523df4ead04/userFiles-a4ca48a0-10f3-468f-b5e3-b953bfa97755/fetchFileTemp4417905759660334726.tmp
23/06/30 15:24:36 INFO executor.Executor: Adding file:/tmp/spark-3382b920-11e6-4daf-ad0c-a523df4ead04/userFiles-a4ca48a0-10f3-468f-b5e3-b953bfa97755/seatunnel-spark-2-starter.jar to class loader
23/06/30 15:24:36 INFO executor.Executor: Fetching spark://hadoop3:37437/jars/connector-console-2.3.1.jar with timestamp 1688109865947
23/06/30 15:24:36 INFO util.Utils: Fetching spark://hadoop3:37437/jars/connector-console-2.3.1.jar to /tmp/spark-3382b920-11e6-4daf-ad0c-a523df4ead04/userFiles-a4ca48a0-10f3-468f-b5e3-b953bfa97755/fetchFileTemp8783962742653812302.tmp
23/06/30 15:24:36 INFO executor.Executor: Adding file:/tmp/spark-3382b920-11e6-4daf-ad0c-a523df4ead04/userFiles-a4ca48a0-10f3-468f-b5e3-b953bfa97755/connector-console-2.3.1.jar to class loader
23/06/30 15:24:36 INFO executor.Executor: Fetching spark://hadoop3:37437/jars/seatunnel-transforms-v2.jar with timestamp 1688109865945
23/06/30 15:24:36 INFO util.Utils: Fetching spark://hadoop3:37437/jars/seatunnel-transforms-v2.jar to /tmp/spark-3382b920-11e6-4daf-ad0c-a523df4ead04/userFiles-a4ca48a0-10f3-468f-b5e3-b953bfa97755/fetchFileTemp6075022225571179099.tmp
23/06/30 15:24:36 INFO executor.Executor: Adding file:/tmp/spark-3382b920-11e6-4daf-ad0c-a523df4ead04/userFiles-a4ca48a0-10f3-468f-b5e3-b953bfa97755/seatunnel-transforms-v2.jar to class loader
23/06/30 15:24:36 INFO sink.ConsoleSinkWriter: output rowType: Areaid<STRING>, Areaname<STRING>
23/06/30 15:24:36 WARN sqlserver.SqlServerConnection: The 'server.timezone' option should be specified to avoid incorrect timestamp values in case of different timezones between the database server and this connector's JVM.
23/06/30 15:24:36 INFO utils.TableDiscoveryUtils: Read list of available databases
23/06/30 15:24:36 INFO utils.TableDiscoveryUtils: list of available databases is: [BI]
23/06/30 15:24:36 INFO utils.TableDiscoveryUtils: Read list of available tables in each database
23/06/30 15:24:36 INFO utils.TableDiscoveryUtils: including 'BI.dbo.area' for further processing
23/06/30 15:24:36 WARN utils.TableDiscoveryUtils: skipping database '*****' due to error reading tables: 由于数据库 '*****' 离线,无法打开该数据库。
23/06/30 15:24:36 WARN utils.TableDiscoveryUtils: skipping database '*****' due to error reading tables: 由于数据库 '*****' 离线,无法打开该数据库。
23/06/30 15:24:36 WARN utils.TableDiscoveryUtils: skipping database '*****' due to error reading tables: 由于数据库 '*****' 离线,无法打开该数据库。
23/06/30 15:24:36 WARN utils.TableDiscoveryUtils: skipping database 'S*****' due to error reading tables: 由于数据库 'S*****' 离线,无法打开该数据库。
23/06/30 15:24:36 INFO jdbc.JdbcConnection: Connection gracefully closed
23/06/30 15:24:36 INFO reader.SourceReaderBase: Open Source Reader.
23/06/30 15:24:36 WARN sqlserver.SqlServerConnection: The 'server.timezone' option should be specified to avoid incorrect timestamp values in case of different timezones between the database server and this connector's JVM.
23/06/30 15:24:36 INFO eumerator.SqlServerChunkSplitter: Start splitting table BI.dbo.area into chunks...
23/06/30 15:24:37 WARN sqlserver.SqlServerDefaultValueConverter: Cannot parse column default value '(getdate())' to type 'datetime'. Expression evaluation is not supported.
23/06/30 15:24:37 WARN sqlserver.SqlServerDefaultValueConverter: Cannot parse column default value '(getdate())' to type 'datetime'. Expression evaluation is not supported.
23/06/30 15:24:37 WARN sqlserver.SqlServerDefaultValueConverter: Cannot parse column default value '(getdate())' to type 'datetime'. Expression evaluation is not supported.
23/06/30 15:24:37 INFO eumerator.SqlServerChunkSplitter: Use unevenly-sized chunks for table BI.dbo.area, the chunk size is 8096
23/06/30 15:24:37 INFO eumerator.SqlServerChunkSplitter: Split table BI.dbo.area into 1 chunks, time cost: 971ms.
23/06/30 15:24:37 INFO jdbc.JdbcConnection: Connection gracefully closed
23/06/30 15:24:37 INFO fetcher.SplitFetcher: Starting split fetcher 0
23/06/30 15:24:37 WARN sqlserver.SqlServerConnection: The 'server.timezone' option should be specified to avoid incorrect timestamp values in case of different timezones between the database server and this connector's JVM.
23/06/30 15:24:37 WARN sqlserver.SqlServerConnection: The 'server.timezone' option should be specified to avoid incorrect timestamp values in case of different timezones between the database server and this connector's JVM.
23/06/30 15:24:37 INFO history.DatabaseHistoryMetrics: Finished database history recovery of 0 change(s) in 1 ms
23/06/30 15:24:38 INFO fetcher.SplitFetcher: Finished reading from splits [BI.dbo.area:0]
```
### Flink or Spark Version
spark:2.4.0
### Java or Scala Version
java:8
### Screenshots
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SqlServer-CDC 没有数据 [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5003:
URL: https://github.com/apache/seatunnel/issues/5003#issuecomment-1765466061
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SqlServer-CDC 没有数据 [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #5003: SqlServer-CDC 没有数据
URL: https://github.com/apache/seatunnel/issues/5003
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #5003: SqlServer-CDC 没有数据
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5003:
URL: https://github.com/apache/seatunnel/issues/5003#issuecomment-1664795471
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org