You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/09/24 03:55:39 UTC
[GitHub] [incubator-seatunnel] EricJoy2048 opened a new issue, #2872: [Bug] [Connector-V2] Hive Sink Decimal datatype error
EricJoy2048 opened a new issue, #2872:
URL: https://github.com/apache/incubator-seatunnel/issues/2872
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Write hive table with decimal datatype error.
### SeaTunnel Version
dev
### SeaTunnel Config
```conf
env {
# You can set flink configuration here
execution.parallelism = 3
job.name="test_hive_source_to_hive"
}
source {
Hive {
table_name = "test_hive.test_hive_source_orc"
metastore_uri = "thrift://ctyun7:9083"
result_table_name = "tmp_table"
}
}
transform {
sql {
sql = "select test_tinyint, test_smallint,test_int, test_bigint,test_boolean, test_float, test_double,test_string,test_binary,test_timestamp, test_decimal, test_char, test_varchar, test_date , 'p1' as test_par1, 'p2' as test_par2 from tmp_table"
}
}
sink {
# choose stdout output plugin to output data to console
Hive {
table_name = "test_hive.test_hive_sink_text_simple"
metastore_uri = "thrift://ctyun7:9083"
partition_by = ["test_par1", "test_par2"]
sink_columns = ["test_tinyint", "test_smallint", "test_int", "test_bigint", "test_boolean", "test_float", "test_double", "test_string", "test_binary", "test_timestamp", "test_decimal", "test_char", "test_varchar", "test_date", "test_par1", "test_par2"]
}
}
```
### Running Command
```shell
sh bin/start-seatunnel-flink-connector-v2.sh --config config/flink_hiveorc_to_hivetext_simple.conf
```
### Error Exception
```log
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Flink job executed failed
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:366)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
Caused by: org.apache.seatunnel.core.starter.exception.CommandExecuteException: Flink job executed failed
at org.apache.seatunnel.core.starter.flink.command.FlinkApiTaskExecuteCommand.execute(FlinkApiTaskExecuteCommand.java:57)
at org.apache.seatunnel.core.starter.Seatunnel.run(Seatunnel.java:40)
at org.apache.seatunnel.core.starter.flink.SeatunnelFlink.main(SeatunnelFlink.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349)
... 11 more
Caused by: org.apache.seatunnel.core.starter.exception.TaskExecuteException: SeaTunnel transform task: sql execute error
at org.apache.seatunnel.core.starter.flink.execution.TransformExecuteProcessor.execute(TransformExecuteProcessor.java:83)
at org.apache.seatunnel.core.starter.flink.execution.FlinkExecution.execute(FlinkExecution.java:68)
at org.apache.seatunnel.core.starter.flink.command.FlinkApiTaskExecuteCommand.execute(FlinkApiTaskExecuteCommand.java:55)
... 18 more
Caused by: org.apache.flink.table.codegen.CodeGenException: Incompatible types of expression and result type. Expression[GeneratedExpression(result$20,isNull$21,,Decimal(8,2),false)] type is [Decimal(8,2)], result type is [BigDecimal]
at org.apache.flink.table.codegen.CodeGenerator.$anonfun$generateResultExpression$3(CodeGenerator.scala:381)
at org.apache.flink.table.codegen.CodeGenerator.$anonfun$generateResultExpression$3$adapted(CodeGenerator.scala:377)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:58)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:51)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.flink.table.codegen.CodeGenerator.generateResultExpression(CodeGenerator.scala:377)
at org.apache.flink.table.codegen.CodeGenerator.generateConverterResultExpression(CodeGenerator.scala:295)
at org.apache.flink.table.plan.nodes.datastream.StreamScan.generateConversionProcessFunction(StreamScan.scala:118)
at org.apache.flink.table.plan.nodes.datastream.StreamScan.convertToInternalRow(StreamScan.scala:81)
at org.apache.flink.table.plan.nodes.datastream.StreamScan.convertToInternalRow$(StreamScan.scala:38)
at org.apache.flink.table.plan.nodes.datastream.DataStreamScan.convertToInternalRow(DataStreamScan.scala:46)
at org.apache.flink.table.plan.nodes.datastream.DataStreamScan.translateToPlan(DataStreamScan.scala:95)
at org.apache.flink.table.plan.nodes.datastream.DataStreamCalc.translateToPlan(DataStreamCalc.scala:73)
at org.apache.flink.table.plan.nodes.datastream.DataStreamSink.translateInput(DataStreamSink.scala:189)
at org.apache.flink.table.plan.nodes.datastream.DataStreamSink.writeToSink(DataStreamSink.scala:88)
at org.apache.flink.table.plan.nodes.datastream.DataStreamSink.translateToPlan(DataStreamSink.scala:59)
at org.apache.flink.table.planner.StreamPlanner.translateToCRow(StreamPlanner.scala:274)
at org.apache.flink.table.planner.StreamPlanner.$anonfun$translate$1(StreamPlanner.scala:119)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at scala.collection.IterableLike.foreach(IterableLike.scala:70)
at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike.map(TraversableLike.scala:233)
at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.apache.flink.table.planner.StreamPlanner.translate(StreamPlanner.scala:116)
at org.apache.flink.table.api.bridge.java.internal.StreamTableEnvironmentImpl.toDataStream(StreamTableEnvironmentImpl.java:334)
at org.apache.flink.table.api.bridge.java.internal.StreamTableEnvironmentImpl.toRetractStream(StreamTableEnvironmentImpl.java:309)
at org.apache.seatunnel.flink.util.TableUtil.tableToDataStream(TableUtil.java:43)
at org.apache.seatunnel.flink.transform.Sql.processStream(Sql.java:53)
at org.apache.seatunnel.core.starter.flink.execution.TransformExecuteProcessor.execute(TransformExecuteProcessor.java:77)
... 20 more
```
### Flink or Spark Version
SeaTunnel version: dev
Hadoop version: Hadoop 2.10.2
Flink version: 1.12.7
Spark version: 2.4.3, scala version 2.11.12
### Java or Scala Version
JDK 1.8
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2872: [Bug] [Connector-V2] Hive Sink Decimal datatype error
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2872:
URL: https://github.com/apache/incubator-seatunnel/issues/2872#issuecomment-1256849926
link https://github.com/apache/incubator-seatunnel/issues/2816
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 closed issue #2872: [Bug] [Connector-V2] Hive Sink Decimal datatype error
Posted by GitBox <gi...@apache.org>.
EricJoy2048 closed issue #2872: [Bug] [Connector-V2] Hive Sink Decimal datatype error
URL: https://github.com/apache/incubator-seatunnel/issues/2872
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2872: [Bug] [Connector-V2] Hive Sink Decimal datatype error
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2872:
URL: https://github.com/apache/incubator-seatunnel/issues/2872#issuecomment-1256849370
Fix by https://github.com/apache/incubator-seatunnel/pull/2846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector-V2] Hive Sink Decimal datatype error [seatunnel]
Posted by "15810785091 (via GitHub)" <gi...@apache.org>.
15810785091 commented on issue #2872:
URL: https://github.com/apache/seatunnel/issues/2872#issuecomment-2068383008
zeta引擎也有这个问题，版本是2.3.4都是decimal字段导致的错误
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org