You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Landon Sterk (Jira)" <ji...@apache.org> on 2021/05/29 00:39:00 UTC
[jira] [Commented] (SPARK-32660) Show Avro related API in
documentation
[ https://issues.apache.org/jira/browse/SPARK-32660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17353640#comment-17353640 ]
Landon Sterk commented on SPARK-32660:
--------------------------------------
[~Gengliang.Wang] hi! My company is migrating from Spark 2.4.x to Spark 3, and we're running into issues because the {{org.apache.spark.sql.avro .AvroSerializer}} class is now package-private. We were previously using this class to convert from DataFrame data into Avro records, which we then use in our Spark applications.
I'm wondering if there's a suggested way to have similar functionality in Spark 3 releases. I certainly understand if this is no longer a feature that is given to developers - the only related API that's marked as a Developer API is the {{org.apache.spark.sql.avro.SchemaConverters}} module, which gives us schema conversion but not data conversion.
We used a custom DF -> Avro data serializer before Spark supported Avro natively, and it's OK if we go back to that; I just wanted to see if the core maintainers had any thoughts about making data conversion a public feature again :)
> Show Avro related API in documentation
> --------------------------------------
>
> Key: SPARK-32660
> URL: https://issues.apache.org/jira/browse/SPARK-32660
> Project: Spark
> Issue Type: Sub-task
> Components: Documentation, SQL
> Affects Versions: 3.0.1, 3.1.0
> Reporter: Gengliang Wang
> Assignee: Gengliang Wang
> Priority: Major
>
> Currently, the Avro related APIs are missing in the documentation https://spark.apache.org/docs/latest/api/scala/org/apache/spark/index.html . This PR is to:
> 1. Mark internal Avro related classes as private
> 2. Show Avro related API in Spark official API documentation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org