You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "ponyliuh (via GitHub)" <gi...@apache.org> on 2023/03/10 10:26:47 UTC
[GitHub] [incubator-seatunnel] ponyliuh opened a new issue, #4326: Loan data from hive to mysql error, because the origin table is empty
ponyliuh opened a new issue, #4326:
URL: https://github.com/apache/incubator-seatunnel/issues/4326
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Loan data from hive to mysql, The derivative operation is not a full table import but an incremental import，When the amount of partition data in my derivative is zero, the derivative reports an error。
### SeaTunnel Version
2.3.0
### SeaTunnel Config
```conf
env {
execution.parallelism = 240
job.mode = "BATCH"
job.name = "SeaTunnel-${hive_tb}-hiveToMysql"
hive.execution.engine=spark
spark.executor.memory=8g
spark.num-executors=10
spark.executor.cores=8
spark.driver.memory=8g
spark.executor.instances=5
}
source {
Hive {
table_name = "$hive_tb"
metastore_uri = "${metastore_uri}"
}
}
transform {
sql {
sql = "replace_tag",
}
}
sink {
jdbc {
url = "jdbc:mysql://${mysql_host}/${mysql_db}"
driver = "com.mysql.cj.jdbc.Driver"
user = "${mysql_user}"
password = "${mysql_pass}"
query = "insert into ${mysql_tb} (${fields}) values(${points})"
}
}
```
### Running Command
```shell
${seatunnel_home}/bin/start-seatunnel-spark-connector-v2.sh \
--master yarn \
--deploy-mode cluster \
--config ${config_template_instance}
```
### Error Exception
```log
error message is like this：
[INFO] 2023-03-10 09:14:43.347 +0800 - -> 23/03/10 09:14:43 ERROR Client: Application diagnostics message: User class threw exception: org.apache.seatunnel.core.starter.exception.CommandExecuteException: Index: 0, Size: 0
at org.apache.seatunnel.core.starter.spark.command.SparkApiTaskExecuteCommand.execute(SparkApiTaskExecuteCommand.java:58)
at org.apache.seatunnel.core.starter.Seatunnel.run(Seatunnel.java:39)
at org.apache.seatunnel.core.starter.spark.SeatunnelSpark.main(SeatunnelSpark.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:684)
Exception in thread "main" org.apache.spark.SparkException: Application application_1638495856834_2481171 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1149)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1526)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
### Flink or Spark Version
spark 2.4.3
### Java or Scala Version
java 1.8
### Screenshots
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lightzhao commented on issue #4326: Loan data from hive to mysql error, because the origin table is empty
Posted by "lightzhao (via GitHub)" <gi...@apache.org>.
lightzhao commented on issue #4326:
URL: https://github.com/apache/incubator-seatunnel/issues/4326#issuecomment-1463605027
I try to fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] closed issue #4326: Loan data from hive to mysql error, because the origin table is empty
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4326: Loan data from hive to mysql error, because the origin table is empty
URL: https://github.com/apache/incubator-seatunnel/issues/4326
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #4326: Loan data from hive to mysql error, because the origin table is empty
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4326:
URL: https://github.com/apache/incubator-seatunnel/issues/4326#issuecomment-1501252128
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #4326: Loan data from hive to mysql error, because the origin table is empty
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4326:
URL: https://github.com/apache/incubator-seatunnel/issues/4326#issuecomment-1512270621
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org