You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/03/11 10:40:00 UTC
[jira] [Comment Edited] (SPARK-27124) Expose
org.apache.spark.sql.avro.SchemaConverters as developer API
[ https://issues.apache.org/jira/browse/SPARK-27124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789388#comment-16789388 ]
Hyukjin Kwon edited comment on SPARK-27124 at 3/11/19 10:39 AM:
----------------------------------------------------------------
The way of reaching it will be same as the Python implementation (the POC you worked on). Py4J allows JVM access fully. Of course, it's hacky - I wasn't trying to say this is an official way of using it.
{code}
>>> spark._jvm.org.apache.spark.sql.avro.SchemaConverters.toSqlType(spark._jvm.org.apache.avro.Schema.Parser().parse("""{"type": "int", "name": "fieldA"}""")).toString()
u'SchemaType(IntegerType,false)'
{code}
Usually the signatures are matched between Scala and Python sides. I suspect that you'd open a function that takes JSON-formatted schema in Avro in PySpark side, right?
was (Author: hyukjin.kwon):
The way of reaching it will be same as the Python implementation. Py4J allows JVM access fully. Of course, it's hacky - I wasn't trying to say this is an official way of using it.
{code}
>>> spark._jvm.org.apache.spark.sql.avro.SchemaConverters.toSqlType(spark._jvm.org.apache.avro.Schema.Parser().parse("""{"type": "int", "name": "fieldA"}""")).toString()
u'SchemaType(IntegerType,false)'
{code}
Usually the signatures are matched between Scala and Python sides. I suspect that you'd open a function that takes JSON-formatted schema in Avro in PySpark side, right?
> Expose org.apache.spark.sql.avro.SchemaConverters as developer API
> ------------------------------------------------------------------
>
> Key: SPARK-27124
> URL: https://issues.apache.org/jira/browse/SPARK-27124
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, SQL
> Affects Versions: 3.0.0
> Reporter: Gabor Somogyi
> Priority: Minor
>
> org.apache.spark.sql.avro.SchemaConverters provides extremely useful APIs to convert schema between Spark SQL and avro. This is reachable from scala side but not from pyspark. I suggest to add this as a developer API to ease development for pyspark users.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org