You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/06/27 10:08:18 UTC
[GitHub] [incubator-seatunnel] 2013650523 opened a new issue, #2067: [Bug] [seatunnel-spark-new-connector-example] java.lang.String cannot be cast to org.apache.spark.unsafe.types.UTF8String
2013650523 opened a new issue, #2067:
URL: https://github.com/apache/incubator-seatunnel/issues/2067
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
can not run local spark new connector example;
### SeaTunnel Version
api-draft
### SeaTunnel Config
```conf
env {
# You can set spark configuration here
# see available properties defined by spark: https://spark.apache.org/docs/latest/configuration.html#available-properties
#job.mode = BATCH
spark.app.name = "SeaTunnel"
spark.executor.instances = 2
spark.executor.cores = 1
spark.executor.memory = "1g"
spark.master = local
}
source {
# This is a example input plugin **only for test and demonstrate the feature input plugin**
FakeSource {
result_table_name = "fake"
field_name = "name,age,timestamp"
}
# You can also use other input plugins, such as hdfs
# hdfs {
# result_table_name = "accesslog"
# path = "hdfs://hadoop-cluster-01/nginx/accesslog"
# format = "json"
# }
# If you would like to get more information about how to configure seatunnel and see full list of input plugins,
# please go to https://seatunnel.apache.org/docs/spark/configuration/source-plugins/Fake
}
transform {
# split data by specific delimiter
# you can also use other transform plugins, such as sql
sql {
sql = "select name,age from fake"
result_table_name = "sql"
}
# If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
# please go to https://seatunnel.apache.org/docs/spark/configuration/transform-plugins/Split
}
sink {
# choose stdout output plugin to output data to console
Console {}
# you can also you other output plugins, such as sql
# hdfs {
# path = "hdfs://hadoop-cluster-01/nginx/accesslog_processed"
# save_mode = "append"
# }
# If you would like to get more information about how to configure seatunnel and see full list of output plugins,
# please go to https://seatunnel.apache.org/docs/spark/configuration/sink-plugins/Console
}
```
### Running Command
```shell
run
```
### Error Exception
```log
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to org.apache.spark.unsafe.types.UTF8String
at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow$class.getUTF8String(rows.scala:46)
at org.apache.spark.sql.catalyst.expressions.SpecificInternalRow.getUTF8String(SpecificInternalRow.scala:193)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$11$$anon$1.hasNext(WholeStageCodegenExec.scala:619)
at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$$anonfun$run$3.apply(WriteToDataSourceV2Exec.scala:117)
at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$$anonfun$run$3.apply(WriteToDataSourceV2Exec.scala:116)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:146)
at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:67)
at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:66)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
### Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2067: [Bug] [seatunnel-spark-new-connector-example] java.lang.String cannot be cast to org.apache.spark.unsafe.types.UTF8String
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2067:
URL: https://github.com/apache/incubator-seatunnel/issues/2067#issuecomment-1168137485
Hi, this error will fix by #2051 in [InternalRowConverter.java](https://github.com/apache/incubator-seatunnel/pull/2051/files#diff-7de91f33da115924984c707c561b8dbcfcdbb4876ad5cb1a27ac6e07d88e7a6c)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2067: [Bug] [seatunnel-spark-new-connector-example] java.lang.String cannot be cast to org.apache.spark.unsafe.types.UTF8String
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2067:
URL: https://github.com/apache/incubator-seatunnel/issues/2067#issuecomment-1168315452
> tks, I worked your way out of the problem, May I ask the new connector sink Hive, etc., are these configuration files still written as before?
Yep. The developer should try to keep the same as the previous configuration as much as possible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] 2013650523 commented on issue #2067: [Bug] [seatunnel-spark-new-connector-example] java.lang.String cannot be cast to org.apache.spark.unsafe.types.UTF8String
Posted by GitBox <gi...@apache.org>.
2013650523 commented on issue #2067:
URL: https://github.com/apache/incubator-seatunnel/issues/2067#issuecomment-1168313375
tks, I worked your way out of the problem, May I ask the new connector sink Hive, etc., are these configuration files still written as before?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs closed issue #2067: [Bug] [seatunnel-spark-new-connector-example] java.lang.String cannot be cast to org.apache.spark.unsafe.types.UTF8String
Posted by GitBox <gi...@apache.org>.
CalvinKirs closed issue #2067: [Bug] [seatunnel-spark-new-connector-example] java.lang.String cannot be cast to org.apache.spark.unsafe.types.UTF8String
URL: https://github.com/apache/incubator-seatunnel/issues/2067
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #2067: [Bug] [seatunnel-spark-new-connector-example] java.lang.String cannot be cast to org.apache.spark.unsafe.types.UTF8String
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #2067:
URL: https://github.com/apache/incubator-seatunnel/issues/2067#issuecomment-1168816402
close by #2051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org