You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "wbbcx (via GitHub)" <gi...@apache.org> on 2024/01/12 07:48:47 UTC

[I] [Bug] [hivesink] can not write date into hive table [seatunnel]

wbbcx opened a new issue, #6194:
URL: https://github.com/apache/seatunnel/issues/6194

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   将MongoDB数据 在spark上跑,同步至hive,发现可以在console打印出来结果,但将sink配置为Hive就不行;怀疑是连不上hive,于是同样的hive配置在spark上跑,hive数据同步至console,日志可以看到数据,就很纳闷?这是能读hive数据,就是写不了hive数据?有遇到相同问题的吗
   
   ### SeaTunnel Version
   
   2.3.3
   
   ### SeaTunnel Config
   
   ```conf
   env {
     execution.parallelism = 1
     job.mode = "BATCH"
     spark.sql.catalogImplementation = "hive"
   }
   source {
     MongoDB {
       uri = "mongodb://xxxxxxxx:3715"
       database = "aaa"
       collection = "bbb"
       flat.sync-string = true
       schema = {
         fields {
           dtam = string
         }
       }
      result_table_name="mongodb_jv_snapshots1"
     }
   }
   transform {
    sql {
         query  ="select dtam from mongodb_jv_snapshots1"
         source_table_name = "mongodb_jv_snapshots1"
         result_table_name = "mongodb_jv_snapshots2"
    }
   }
   
   sink {
      Hive {
         table_name = "ods.ods_mongodb__test"
         metastore_uri = "thrift://xxxx:9083"
         hdfs_site_path = "/etc/ecm/hadoop-conf/hdfs-site.xml"
         hive_site_path = "/etc/ecm/spark-conf/hive-site.xml"
       }
   }
   ```
   
   
   ### Running Command
   
   ```shell
   ./bin/start-seatunnel-spark-3-connector-v2.sh --master local[4] --deploy-mode client --config ./onlinefile/jv_snapshots_hivesink_test.conf
   ```
   
   
   ### Error Exception
   
   ```log
   24/01/12 15:24:35 ERROR [main] SparkTaskExecuteCommand: Run SeaTunnel on spark failed.
   java.lang.NullPointerException
           at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.AbstractWriteStrategy.lambda$buildSchemaWithRowType$0(AbstractWriteStrategy.java:135)
           at java.util.ArrayList.forEach(ArrayList.java:1259)
           at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.AbstractWriteStrategy.buildSchemaWithRowType(AbstractWriteStrategy.java:133)
           at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.TextWriteStrategy.setSeaTunnelRowTypeInfo(TextWriteStrategy.java:69)
           at org.apache.seatunnel.connectors.seatunnel.file.sink.BaseFileSink.setTypeInfo(BaseFileSink.java:71)
           at org.apache.seatunnel.core.starter.spark.execution.SinkExecuteProcessor.execute(SinkExecuteProcessor.java:109)
           at org.apache.seatunnel.core.starter.spark.execution.SparkExecution.execute(SparkExecution.java:74)
           at org.apache.seatunnel.core.starter.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:60)
           at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
           at org.apache.seatunnel.core.starter.spark.SeaTunnelSpark.main(SeaTunnelSpark.java:35)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.lang.reflect.Method.invoke(Method.java:498)
           at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
           at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
           at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
           at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
           at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
           at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
           at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
           at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
   24/01/12 15:24:35 ERROR [main] SeaTunnel: 
   
   ===============================================================================
   
   
   24/01/12 15:24:35 ERROR [main] SeaTunnel: Fatal Error, 
   
   24/01/12 15:24:35 ERROR [main] SeaTunnel: Please submit bug report in https://github.com/apache/seatunnel/issues
   
   24/01/12 15:24:35 ERROR [main] SeaTunnel: Reason:null 
   
   24/01/12 15:24:35 ERROR [main] SeaTunnel: Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException
           at org.apache.seatunnel.core.starter.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:63)
           at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
           at org.apache.seatunnel.core.starter.spark.SeaTunnelSpark.main(SeaTunnelSpark.java:35)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.lang.reflect.Method.invoke(Method.java:498)
           at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
           at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
           at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
           at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
           at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
           at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
           at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
           at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
   ```
   
   
   ### Zeta or Flink or Spark Version
   
   spark版本:3.2.1
   
   ### Java or Scala Version
   
   jdk1.8版本
   
   ### Screenshots
   
   报错日志参考上面
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [hivesink] can not write date into hive table [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6194: [Bug] [hivesink] can not  write date into hive table
URL: https://github.com/apache/seatunnel/issues/6194


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [hivesink] can not write date into hive table [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6194:
URL: https://github.com/apache/seatunnel/issues/6194#issuecomment-1951502844

   This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [hivesink] can not write date into hive table [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6194:
URL: https://github.com/apache/seatunnel/issues/6194#issuecomment-1937930431

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org