You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "wbbcx (via GitHub)" <gi...@apache.org> on 2024/01/12 07:48:47 UTC
[I] [Bug] [hivesink] can not write date into hive table [seatunnel]
wbbcx opened a new issue, #6194:
URL: https://github.com/apache/seatunnel/issues/6194
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
将MongoDB数据 在spark上跑,同步至hive,发现可以在console打印出来结果,但将sink配置为Hive就不行;怀疑是连不上hive,于是同样的hive配置在spark上跑,hive数据同步至console,日志可以看到数据,就很纳闷?这是能读hive数据,就是写不了hive数据?有遇到相同问题的吗
### SeaTunnel Version
2.3.3
### SeaTunnel Config
```conf
env {
execution.parallelism = 1
job.mode = "BATCH"
spark.sql.catalogImplementation = "hive"
}
source {
MongoDB {
uri = "mongodb://xxxxxxxx:3715"
database = "aaa"
collection = "bbb"
flat.sync-string = true
schema = {
fields {
dtam = string
}
}
result_table_name="mongodb_jv_snapshots1"
}
}
transform {
sql {
query ="select dtam from mongodb_jv_snapshots1"
source_table_name = "mongodb_jv_snapshots1"
result_table_name = "mongodb_jv_snapshots2"
}
}
sink {
Hive {
table_name = "ods.ods_mongodb__test"
metastore_uri = "thrift://xxxx:9083"
hdfs_site_path = "/etc/ecm/hadoop-conf/hdfs-site.xml"
hive_site_path = "/etc/ecm/spark-conf/hive-site.xml"
}
}
```
### Running Command
```shell
./bin/start-seatunnel-spark-3-connector-v2.sh --master local[4] --deploy-mode client --config ./onlinefile/jv_snapshots_hivesink_test.conf
```
### Error Exception
```log
24/01/12 15:24:35 ERROR [main] SparkTaskExecuteCommand: Run SeaTunnel on spark failed.
java.lang.NullPointerException
at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.AbstractWriteStrategy.lambda$buildSchemaWithRowType$0(AbstractWriteStrategy.java:135)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.AbstractWriteStrategy.buildSchemaWithRowType(AbstractWriteStrategy.java:133)
at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.TextWriteStrategy.setSeaTunnelRowTypeInfo(TextWriteStrategy.java:69)
at org.apache.seatunnel.connectors.seatunnel.file.sink.BaseFileSink.setTypeInfo(BaseFileSink.java:71)
at org.apache.seatunnel.core.starter.spark.execution.SinkExecuteProcessor.execute(SinkExecuteProcessor.java:109)
at org.apache.seatunnel.core.starter.spark.execution.SparkExecution.execute(SparkExecution.java:74)
at org.apache.seatunnel.core.starter.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:60)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.spark.SeaTunnelSpark.main(SeaTunnelSpark.java:35)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
24/01/12 15:24:35 ERROR [main] SeaTunnel:
===============================================================================
24/01/12 15:24:35 ERROR [main] SeaTunnel: Fatal Error,
24/01/12 15:24:35 ERROR [main] SeaTunnel: Please submit bug report in https://github.com/apache/seatunnel/issues
24/01/12 15:24:35 ERROR [main] SeaTunnel: Reason:null
24/01/12 15:24:35 ERROR [main] SeaTunnel: Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException
at org.apache.seatunnel.core.starter.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:63)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.spark.SeaTunnelSpark.main(SeaTunnelSpark.java:35)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
### Zeta or Flink or Spark Version
spark版本:3.2.1
### Java or Scala Version
jdk1.8版本
### Screenshots
报错日志参考上面
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [hivesink] can not write date into hive table [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6194: [Bug] [hivesink] can not write date into hive table
URL: https://github.com/apache/seatunnel/issues/6194
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [hivesink] can not write date into hive table [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6194:
URL: https://github.com/apache/seatunnel/issues/6194#issuecomment-1951502844
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [hivesink] can not write date into hive table [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6194:
URL: https://github.com/apache/seatunnel/issues/6194#issuecomment-1937930431
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org