[GitHub] [spark] cloud-fan commented on a change in pull request #34503: [SPARK-37225][SQL] Support reading and writing ANSI intervals from/to Avro datasources

[GitBox <gi...@apache.org>]

cloud-fan commented on a change in pull request #34503:
URL: https://github.com/apache/spark/pull/34503#discussion_r744385623



##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala
##########
@@ -51,11 +52,21 @@ object SchemaConverters {
     toSqlTypeHelper(avroSchema, Set.empty)
   }
 
+  // The property specifies Catalyst type of the given field
+  private val CATALYST_TYPE_PROP_NAME = "spark.sql.catalyst.type"
+
   private def toSqlTypeHelper(avroSchema: Schema, existingRecordNames: Set[String]): SchemaType = {
     avroSchema.getType match {
       case INT => avroSchema.getLogicalType match {
         case _: Date => SchemaType(DateType, nullable = false)
-        case _ => SchemaType(IntegerType, nullable = false)
+        case _ =>
+          val catalystTypeAttrValue = avroSchema.getProp(CATALYST_TYPE_PROP_NAME)
+          val catalystType = if (catalystTypeAttrValue == null) {
+            IntegerType
+          } else {
+            CatalystSqlParser.parseDataType(catalystTypeAttrValue)

Review comment:
       Can we add an assert to make sure it's year-month interval? I agree to use a general property `spark.sql.catalyst.type`, which can be useful in the future. But I'm a bit afraid to return an arbitrary type here. It needs to be compatible with a physical avro INT at least.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org