You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "yunfeng79 (via GitHub)" <gi...@apache.org> on 2023/06/19 07:44:07 UTC
[GitHub] [seatunnel] yunfeng79 opened a new issue, #4948: MySQL-CDC SparkTaskExecuteCommand: Run SeaTunnel on spark failed.
yunfeng79 opened a new issue, #4948:
URL: https://github.com/apache/seatunnel/issues/4948
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
apache-seatunnel-incubating-2.3.1
spark-2.4.5
env {
# You can set SeaTunnel environment configuration here
execution.parallelism = 2
job.mode = "BATCH"
checkpoint.interval = 10000
#execution.checkpoint.interval = 10000
#execution.checkpoint.data-uri = "hdfs://dc-01:8020/flink-checkpoints"
}
source {
MySQL-CDC {
base-url = "jdbc:mysql://192.168.1.39:3306/testdb"
parallelism = 1
server-id = 7
user = "dev"
password = "dev"
query = "select id,`name`,age from user1"
table-name = ["testdb.user1"]
result_table_name = "res011"
}
}
sink {
Doris {
fenodes = "192.168.1.231:8030"
username = dev
password = "dev"
table.identifier = "test.user2"
sink.enable-2pc = "true"
sink.label-prefix = "user2_1"
doris.config = {
format = "csv"
column_separator = ","
}
}
}

### SeaTunnel Version
apache-seatunnel-incubating-2.3.1
### SeaTunnel Config
```conf
env {
# You can set SeaTunnel environment configuration here
execution.parallelism = 2
job.mode = "BATCH"
checkpoint.interval = 10000
#execution.checkpoint.interval = 10000
#execution.checkpoint.data-uri = "hdfs://dc-01:8020/flink-checkpoints"
}
source {
MySQL-CDC {
base-url = "jdbc:mysql://192.168.1.39:3306/testdb"
parallelism = 1
server-id = 7
user = "dev"
password = "dev"
query = "select id,`name`,age from user1"
table-name = ["testdb.user1"]
result_table_name = "res011"
}
}
sink {
Doris {
fenodes = "192.168.1.231:8030"
username = dev
password = "dev"
table.identifier = "test.user2"
sink.enable-2pc = "true"
sink.label-prefix = "user2_1"
doris.config = {
format = "csv"
column_separator = ","
}
}
}
```
### Running Command
```shell
./bin/start-seatunnel-spark-2-connector-v2.sh --master local[4] --deploy-mode client --config ./config/v2-cdc-mysqltodoris.conf
```
### Error Exception
```log
23/06/19 15:23:48 INFO AbstractPluginDiscovery: Load SeaTunnelSource Plugin from /home/app/apache-seatunnel-incubating-2.3.1/connectors/seatunnel
23/06/19 15:23:48 INFO AbstractPluginDiscovery: Discovery plugin jar: MySQL-CDC at: file:/home/app/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-cdc-mysql-2.3.1.jar
23/06/19 15:23:48 INFO AbstractPluginDiscovery: Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='MySQL-CDC'} from classpath
23/06/19 15:23:48 ERROR SparkTaskExecuteCommand: Run SeaTunnel on spark failed.
java.lang.NullPointerException
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191)
at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.config.MySqlSourceConfigFactory.create(MySqlSourceConfigFactory.java:61)
at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.MySqlDialect.<init>(MySqlDialect.java:53)
at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.MySqlIncrementalSource.createDataSourceDialect(MySqlIncrementalSource.java:128)
at org.apache.seatunnel.connectors.cdc.base.source.IncrementalSource.prepare(IncrementalSource.java:117)
at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.initializePlugins(SourceExecuteProcessor.java:104)
at org.apache.seatunnel.core.starter.spark.execution.SparkAbstractPluginExecuteProcessor.<init>(SparkAbstractPluginExecuteProcessor.java:49)
at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.<init>(SourceExecuteProcessor.java:51)
at org.apache.seatunnel.core.starter.spark.execution.SparkExecution.<init>(SparkExecution.java:57)
at org.apache.seatunnel.core.starter.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:59)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.spark.SeaTunnelSpark.main(SeaTunnelSpark.java:35)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
23/06/19 15:23:48 ERROR SeaTunnel:
===============================================================================
23/06/19 15:23:48 ERROR SeaTunnel: Fatal Error,
23/06/19 15:23:48 ERROR SeaTunnel: Please submit bug report in https://github.com/apache/incubator-seatunnel/issues
23/06/19 15:23:48 ERROR SeaTunnel: Reason:null
23/06/19 15:23:48 ERROR SeaTunnel: Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException
at org.apache.seatunnel.core.starter.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:63)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.spark.SeaTunnelSpark.main(SeaTunnelSpark.java:35)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
23/06/19 15:23:48 ERROR SeaTunnel:
```
### Flink or Spark Version
spark 2.4.5
### Java or Scala Version
1.8.0_261
### Screenshots
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] closed issue #4948: MySQL-CDC SparkTaskExecuteCommand: Run SeaTunnel on spark failed.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4948: MySQL-CDC SparkTaskExecuteCommand: Run SeaTunnel on spark failed.
URL: https://github.com/apache/seatunnel/issues/4948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #4948: MySQL-CDC SparkTaskExecuteCommand: Run SeaTunnel on spark failed.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4948:
URL: https://github.com/apache/seatunnel/issues/4948#issuecomment-1654793827
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #4948: MySQL-CDC SparkTaskExecuteCommand: Run SeaTunnel on spark failed.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4948:
URL: https://github.com/apache/seatunnel/issues/4948#issuecomment-1642932693
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org