You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "yunfeng79 (via GitHub)" <gi...@apache.org> on 2023/06/19 07:44:07 UTC

[GitHub] [seatunnel] yunfeng79 opened a new issue, #4948: MySQL-CDC SparkTaskExecuteCommand: Run SeaTunnel on spark failed.

yunfeng79 opened a new issue, #4948:
URL: https://github.com/apache/seatunnel/issues/4948

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   apache-seatunnel-incubating-2.3.1
   spark-2.4.5 
   
   
   env {
     # You can set SeaTunnel environment configuration here
     execution.parallelism = 2
     job.mode = "BATCH"
     checkpoint.interval = 10000
     #execution.checkpoint.interval = 10000
     #execution.checkpoint.data-uri = "hdfs://dc-01:8020/flink-checkpoints"
   }
   
   source {
   
     MySQL-CDC { 
       base-url = "jdbc:mysql://192.168.1.39:3306/testdb"
       parallelism = 1 
       server-id = 7
       user = "dev"
       password = "dev"
       query = "select id,`name`,age from user1"
       table-name = ["testdb.user1"]
       result_table_name = "res011"
      }
   
   
   }
   
   
   sink {
     Doris {
       fenodes = "192.168.1.231:8030"
       username = dev
       password = "dev"
       table.identifier = "test.user2"
       sink.enable-2pc = "true"
       sink.label-prefix = "user2_1"
       doris.config = {
             format = "csv"
             column_separator = ","
         }
      }
   }
   
   ![image](https://github.com/apache/seatunnel/assets/21050635/c1007cee-dcd4-44cb-b52e-456623e7734a)
   
   
   
   
   
   ### SeaTunnel Version
   
   apache-seatunnel-incubating-2.3.1
   
   ### SeaTunnel Config
   
   ```conf
   env {
     # You can set SeaTunnel environment configuration here
     execution.parallelism = 2
     job.mode = "BATCH"
     checkpoint.interval = 10000
     #execution.checkpoint.interval = 10000
     #execution.checkpoint.data-uri = "hdfs://dc-01:8020/flink-checkpoints"
   }
   
   source {
   
     MySQL-CDC { 
       base-url = "jdbc:mysql://192.168.1.39:3306/testdb"
       parallelism = 1 
       server-id = 7
       user = "dev"
       password = "dev"
       query = "select id,`name`,age from user1"
       table-name = ["testdb.user1"]
       result_table_name = "res011"
      }
   
   
   }
   
   
   sink {
     Doris {
       fenodes = "192.168.1.231:8030"
       username = dev
       password = "dev"
       table.identifier = "test.user2"
       sink.enable-2pc = "true"
       sink.label-prefix = "user2_1"
       doris.config = {
             format = "csv"
             column_separator = ","
         }
      }
   }
   ```
   
   
   ### Running Command
   
   ```shell
   ./bin/start-seatunnel-spark-2-connector-v2.sh --master local[4] --deploy-mode client --config  ./config/v2-cdc-mysqltodoris.conf
   ```
   
   
   ### Error Exception
   
   ```log
   23/06/19 15:23:48 INFO AbstractPluginDiscovery: Load SeaTunnelSource Plugin from /home/app/apache-seatunnel-incubating-2.3.1/connectors/seatunnel
   23/06/19 15:23:48 INFO AbstractPluginDiscovery: Discovery plugin jar: MySQL-CDC at: file:/home/app/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-cdc-mysql-2.3.1.jar
   23/06/19 15:23:48 INFO AbstractPluginDiscovery: Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='MySQL-CDC'} from classpath
   23/06/19 15:23:48 ERROR SparkTaskExecuteCommand: Run SeaTunnel on spark failed.
   java.lang.NullPointerException
           at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191)
           at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.config.MySqlSourceConfigFactory.create(MySqlSourceConfigFactory.java:61)
           at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.MySqlDialect.<init>(MySqlDialect.java:53)
           at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.MySqlIncrementalSource.createDataSourceDialect(MySqlIncrementalSource.java:128)
           at org.apache.seatunnel.connectors.cdc.base.source.IncrementalSource.prepare(IncrementalSource.java:117)
           at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.initializePlugins(SourceExecuteProcessor.java:104)
           at org.apache.seatunnel.core.starter.spark.execution.SparkAbstractPluginExecuteProcessor.<init>(SparkAbstractPluginExecuteProcessor.java:49)
           at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.<init>(SourceExecuteProcessor.java:51)
           at org.apache.seatunnel.core.starter.spark.execution.SparkExecution.<init>(SparkExecution.java:57)
           at org.apache.seatunnel.core.starter.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:59)
           at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
           at org.apache.seatunnel.core.starter.spark.SeaTunnelSpark.main(SeaTunnelSpark.java:35)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.lang.reflect.Method.invoke(Method.java:498)
           at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
           at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
           at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
           at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
           at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
           at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
           at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
           at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
   23/06/19 15:23:48 ERROR SeaTunnel: 
   
   ===============================================================================
   
   
   23/06/19 15:23:48 ERROR SeaTunnel: Fatal Error, 
   
   23/06/19 15:23:48 ERROR SeaTunnel: Please submit bug report in https://github.com/apache/incubator-seatunnel/issues
   
   23/06/19 15:23:48 ERROR SeaTunnel: Reason:null 
   
   23/06/19 15:23:48 ERROR SeaTunnel: Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException
           at org.apache.seatunnel.core.starter.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:63)
           at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
           at org.apache.seatunnel.core.starter.spark.SeaTunnelSpark.main(SeaTunnelSpark.java:35)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.lang.reflect.Method.invoke(Method.java:498)
           at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
           at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
           at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
           at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
           at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
           at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
           at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
           at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    
   23/06/19 15:23:48 ERROR SeaTunnel:
   ```
   
   
   ### Flink or Spark Version
   
   spark 2.4.5
   
   ### Java or Scala Version
   
   1.8.0_261
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] github-actions[bot] closed issue #4948: MySQL-CDC SparkTaskExecuteCommand: Run SeaTunnel on spark failed.

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4948: MySQL-CDC SparkTaskExecuteCommand: Run SeaTunnel on spark failed.
URL: https://github.com/apache/seatunnel/issues/4948


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] github-actions[bot] commented on issue #4948: MySQL-CDC SparkTaskExecuteCommand: Run SeaTunnel on spark failed.

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4948:
URL: https://github.com/apache/seatunnel/issues/4948#issuecomment-1654793827

   This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] github-actions[bot] commented on issue #4948: MySQL-CDC SparkTaskExecuteCommand: Run SeaTunnel on spark failed.

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4948:
URL: https://github.com/apache/seatunnel/issues/4948#issuecomment-1642932693

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org