You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Lingzhe Sun (Jira)" <ji...@apache.org> on 2022/09/22 07:57:00 UTC
[jira] [Commented] (SPARK-38088) Kryo DataWritingSparkTaskResult registration error
[ https://issues.apache.org/jira/browse/SPARK-38088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608125#comment-17608125 ]
Lingzhe Sun commented on SPARK-38088:
-------------------------------------
[~majsiel] Hi, I'm facing the same situation. Did you manage to get around this?
> Kryo DataWritingSparkTaskResult registration error
> --------------------------------------------------
>
> Key: SPARK-38088
> URL: https://issues.apache.org/jira/browse/SPARK-38088
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.1.2
> Reporter: Michał Wieleba
> Priority: Major
> Attachments: image-2022-02-02-10-09-14-858.png
>
>
> Spark 3.1.2, Scala 2.12
> I'm registering classes with _sparkConf.registerKryoClasses(Array( ..._ method. Inside there are spark structured streaming code. Following settings are added as well:
> sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
> sparkConf.set("spark.kryo.registrationRequired", "true")
> Unfortunately, during execution following error is thrown:
> Caused by: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTaskResult
> Note: To register this class use: kryo.register(org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTaskResult.class);
> As far as I can see in [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala]
> class org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTaskResult is private (private[v2] case class DataWritingSparkTaskResult) therefore not available to register.
>
> !image-2022-02-02-10-09-14-858.png!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org