You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Max Gekk (Jira)" <ji...@apache.org> on 2022/02/24 12:18:00 UTC
[jira] [Commented] (SPARK-38315) Add a config to collect objects as Java 8 types in the Thrift server
[ https://issues.apache.org/jira/browse/SPARK-38315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17497362#comment-17497362 ]
Max Gekk commented on SPARK-38315:
----------------------------------
I am working on this.
> Add a config to collect objects as Java 8 types in the Thrift server
> --------------------------------------------------------------------
>
> Key: SPARK-38315
> URL: https://issues.apache.org/jira/browse/SPARK-38315
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Max Gekk
> Assignee: Max Gekk
> Priority: Major
>
> Add new config that should control collect(), and allow to enable/disable to Java 8 types in the Thrift server. The config should solve the following issue:
> When an user connects to the Thrift Server and a query involve a datasource connect which doesn't handle Java8 types, the user observes the following exception:
> {code:java}
> ERROR SparkExecuteStatementOperation: Error executing query with ac61b10a-486e-463b-8726-3b61da58582e, currentState RUNNING,
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 8) (10.157.1.194 executor 0): java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: java.sql.Timestamp is not a valid external type for schema of timestamp
> if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.sql.catalyst.util.DateTimeUtils$, TimestampType, instantToMicros, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, loan_perf_date), TimestampType), true, false) AS loan_perf_date#1125
> at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:239)
> at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:210)
> at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
> at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
> at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org