You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ivan Sadikov (Jira)" <ji...@apache.org> on 2022/07/18 00:56:00 UTC
[jira] [Updated] (SPARK-39802) Support Avro recursive schemas in Spark
[ https://issues.apache.org/jira/browse/SPARK-39802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Sadikov updated SPARK-39802:
---------------------------------
Description:
This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718.
It would be great if Spark could support recursive Avro schemas as currently the following exception is thrown:
{code:java}
org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive reference in Avro schema, which can not be processed by Spark: {
"type": "record",
"name": "Struct",
"fields": [
{
"name": "fields",
"type": [
"null",
{
"type": "array",
"items": {
"type": "record",
"name": "fields",
"fields": [
{
"name": "value",
"type": [
"null",
"string"
],
"default": null
}
]
}
}
],
"default": null
}
]
}{code}
> Support Avro recursive schemas in Spark
> ---------------------------------------
>
> Key: SPARK-39802
> URL: https://issues.apache.org/jira/browse/SPARK-39802
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Ivan Sadikov
> Priority: Major
>
> This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718.
> It would be great if Spark could support recursive Avro schemas as currently the following exception is thrown:
> {code:java}
> org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive reference in Avro schema, which can not be processed by Spark: {
> "type": "record",
> "name": "Struct",
> "fields": [
> {
> "name": "fields",
> "type": [
> "null",
> {
> "type": "array",
> "items": {
> "type": "record",
> "name": "fields",
> "fields": [
> {
> "name": "value",
> "type": [
> "null",
> "string"
> ],
> "default": null
> }
> ]
> }
> }
> ],
> "default": null
> }
> ]
> }{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org