[SPARK-29898][SQL] Support Avro Custom Logical Types

[Carlos del Prado Mota <pr...@gmail.com>]

Hi there, 

I recently proposed a change to add support for custom logical types for Avro in Spark. This change provides capabilities to build custom types conversions between StructType and Avro and is fully compatible with the current solution. This is the link for the solution and I would really appreciate if you could check it. I think that @Michael Armbrust is the best candidate to check this, but please redirect to the proper developer if necessary.

https://github.com/apache/spark/pull/26524 <https://github.com/apache/spark/pull/26524>

Many thanks & regards,
Carlos.

Re: [SPARK-29898][SQL] Support Avro Custom Logical Types

[Gengliang Wang <ge...@databricks.com>]

Hi Carlos,

To write Avro files with a schema different from the default mapping, you
can use the option "avroSchema":
 df.write.format("avro").option("avroSchema",
avroSchemaAsJSONStringFormat)...
See
https://spark.apache.org/docs/latest/sql-data-sources-avro.html#supported-types-for-spark-sql---avro-conversion
for more details.
The function `to_avro` also supports customized the output schema with the
last parameter "jsonFormatSchema"

To read Avro file with customized Avro schema, you can also use the option
"avroSchema". To specify a customized Dataframe schema, you can use general
data source method "spark.read.schema(..)..".
If there is missing mapping for the Avro logical types to DataFrame schema(
https://spark.apache.org/docs/latest/sql-data-sources-avro.html#supported-types-for-avro---spark-sql-conversion),
please update it in the `SchemaConverters`.

Hope this helps.

Thank you
Gengliang

On Fri, Nov 22, 2019 at 5:17 AM Carlos del Prado Mota <pr...@gmail.com>
wrote:

> Hi there,
>
> I recently proposed a change to add support for custom logical types for
> Avro in Spark. This change provides capabilities to build custom types
> conversions between StructType and Avro and is fully compatible with the
> current solution. This is the link for the solution and I would really
> appreciate if you could check it. I think that @Michael Armbrust is the
> best candidate to check this, but please redirect to the proper developer
> if necessary.
>
> https://github.com/apache/spark/pull/26524
>
> Many thanks & regards,
> Carlos.
>