[GitHub] [hudi] alexeykudinkin commented on a change in pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

[GitBox <gi...@apache.org>]

alexeykudinkin commented on a change in pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#discussion_r835784601



##########
File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala
##########
@@ -62,10 +62,10 @@ object AvroConversionUtils {
    * @param rootCatalystType Catalyst [[StructType]] to be transformed into
    * @return converter accepting Avro payload and transforming it into a Catalyst one (in the form of [[InternalRow]])
    */
-  def createAvroToInternalRowConverter(rootAvroType: Schema, rootCatalystType: StructType): GenericRecord => Option[InternalRow] =
-    record => sparkAdapter.createAvroDeserializer(rootAvroType, rootCatalystType)

Review comment:
       @xiarixiaoyao that's a very sneaky issue
   
   Please leave a note in a comment explaining what was the issue and how you've addressed it.
   
   However, i'd suggest to just pull out SparkAdapter from the closure keeping the API of this method intact: we don't want to push casting (to InternalRow/GenericRecord) onto the users
   
   ```
   def createAvroToInternalRowConverter(rootAvroType: Schema, rootCatalystType: StructType): GenericRecord => Option[InternalRow] = {
       val deserilizer = sparkAdapter.createAvroDeserializer(rootAvroType, rootCatalystType)
       record => deserializer.deserialize(record).map(_.asInstanceOf[InternalRow])
   }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org